Data processing method, data processing apparatus, electronic device, storage medium

Abstract

A data processing method, a data processing device, an electronic equipment and a storage medium are provided. The method is applied to the field of data processing. The method comprises: performing forward inference on a network model, collecting an input activation matrix of each linear operator in a plurality of linear operators included in the network model; reading a weight matrix of each linear operator from a storage component; determining a mean square activation value of the linear operator according to the input activation matrix of each linear operator; determining a weight quantization error of the linear operator according to the weight matrix of each linear operator; determining a quantization sensitivity index of the linear operator based on the product of the mean square activation value and the weight quantization error of each linear operator; determining a service constraint parameter; and dividing the plurality of linear operators into a first set and a second set according to the service constraint parameter and the quantization sensitivity index of the plurality of linear operators to obtain an accuracy allocation scheme. The method improves the accuracy of model quantization inference while maintaining low-bit quantization performance.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of electrical data processing, and more specifically to a data processing method, a data processing apparatus, an electronic device, and a non-transitory computer-readable storage medium. Background Technology

[0002] Quantization (or low-bit quantization) refers to compressing the original high-precision floating-point parameters and computational features in a network model into low-bit-width integers such as 4 bits or 8 bits. This reduces model size, computational power, and memory overhead with a small loss of precision, enabling faster inference deployment. For example, large language models, due to their massive number of parameters, face bottlenecks such as high memory usage and high inference latency during deployment. Weight quantization is currently the core method to solve this problem. By compressing model weights from high precision (such as BF16 / FP16) to low precision (such as INT4 / INT8), storage and memory access overhead can be significantly reduced. However, this quantization method is not uniformly safe for all network layers: different layers and different experts have vastly different sensitivities to quantization errors. A uniform quantization strategy often leads to "overquantization" of sensitive layers, resulting in inference artifacts, or "underquantization" of robust layers, wasting precision budget. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a data processing method, comprising: performing forward inference on a network model and acquiring the input activation matrix of each linear operator among a plurality of linear operators included in the network model; reading the weight matrix of each linear operator from a storage unit; determining the mean square activation value of the linear operator based on the input activation matrix of each linear operator, wherein the mean square activation value is used to measure the average energy of the input activation matrix of the linear operator; determining the weight quantization error of the linear operator based on the weight matrix of each linear operator, wherein the weight quantization error is used to measure the inherent error introduced when quantizing the weight matrix of the linear operator; and determining a weight quantization error based on the product of the mean square activation value and the weight quantization error of each linear operator. The quantization sensitivity index of the linear operators; determining business constraint parameters, wherein the business constraint parameters include relevant configuration parameters used when performing binary grouping of the multiple linear operators; based on the business constraint parameters and the quantization sensitivity index of the multiple linear operators, dividing the multiple linear operators into a first set and a second set to obtain a precision allocation scheme for the multiple linear operators, wherein the minimum value of the quantization sensitivity index of the linear operators in the first set is greater than the maximum value of the quantization sensitivity index of the linear operators in the second set, and in the precision allocation scheme, the linear operators in the first set use a higher quantization precision than the linear operators in the second set for quantization inference operations of the network model.

[0004] For example, in the data processing method provided in at least one embodiment of this disclosure, in the quantization inference operation, one or more rows in the input activation matrix of each linear operator share the same scaling factor, and one or more columns in the weight matrix of each linear operator share the same scaling factor.

[0005] For example, in a data processing method provided in at least one embodiment of this disclosure, determining the mean square activation value of the linear operator based on the input activation matrix of each linear operator includes: squaring the multiple elements included in the input activation matrix of the linear operator; and determining the mean of the squares of the multiple elements as the mean square activation value of the linear operator.

[0006] For example, in the data processing method provided in at least one embodiment of this disclosure, determining the weight quantization error of the linear operator based on the weight matrix of each linear operator includes: determining the maximum value among the absolute values ​​of each output channel in the weight matrix of the linear operator; determining the square of the maximum value among the absolute values ​​of each output channel as the intermediate result corresponding to each output channel; and determining the mean of the multiple intermediate results corresponding to the multiple output channels included in the weight matrix of the linear operator as the weight quantization error of the linear operator.

[0007] For example, in the data processing method provided in at least one embodiment of this disclosure, determining the business constraint parameter includes: determining the global distribution statistics of the quantization sensitivity indices of the plurality of linear operators based on the quantization sensitivity indices of the plurality of linear operators, wherein the global distribution statistics include the boundary thresholds corresponding to the preset percentiles of the plurality of linear operators; determining the percentile X, wherein the percentile X represents that X% of the linear operators in the plurality of linear operators are assigned to the second set, and the remaining 1-X% of the linear operators are assigned to the first set, and X is a positive integer less than 100; and determining the boundary threshold corresponding to the percentile X as the business constraint parameter based on the percentile X and the global distribution statistics.

[0008] For example, in a data processing method provided in at least one embodiment of this disclosure, the plurality of linear operators are divided into a first set and a second set according to the business constraint parameters and the quantization sensitivity index of the plurality of linear operators. This includes: sorting the quantization sensitivity index of the plurality of linear operators in descending order to obtain a first sorting result; and according to the first sorting result, assigning the linear operators whose quantization sensitivity index is greater than the threshold corresponding to the percentile X to the first set, and assigning the other linear operators to the second set.

[0009] For example, in a data processing method provided in at least one embodiment of this disclosure, the network model includes multiple hybrid expert model layers, each hybrid expert model layer includes multiple expert networks, and determining business constraint parameters includes: for each hybrid expert model layer: determining a quantization sensitivity index for the hybrid expert model layer, wherein the quantization sensitivity index of the hybrid expert model layer is the average of the quantization sensitivity indices of all linear operators included in the hybrid expert model layer; determining normalized sensitivity indices corresponding to the multiple expert networks included in the hybrid expert model layer based on the quantization sensitivity index corresponding to the hybrid expert model layer; and determining a first threshold corresponding to the hybrid expert model layer based on the normalized sensitivity indices corresponding to the multiple expert networks, wherein the business constraint parameters include the first thresholds corresponding to the multiple hybrid expert model layers.

[0010] For example, in the data processing method provided in at least one embodiment of this disclosure, determining the normalized sensitivity index corresponding to each of the multiple expert networks included in the hybrid expert model layer based on the quantized sensitivity index corresponding to the hybrid expert model layer includes: for each expert network, determining the average value of the quantized sensitivity indices of the multiple linear operators included in the expert network as the quantized sensitivity index corresponding to the expert network; and using the ratio of the quantized sensitivity index corresponding to the expert network to the quantized sensitivity index corresponding to the hybrid expert model layer as the normalized sensitivity index corresponding to the expert network.

[0011] For example, in the data processing method provided in at least one embodiment of this disclosure, the normalized sensitivity index corresponding to the multiple expert networks included in the hybrid expert model layer is determined according to the quantization sensitivity index corresponding to the hybrid expert model layer. This includes: for expert networks whose received number of tokens is less than a preset threshold, a preset label is added to the expert network. In the process of dividing the multiple linear operators into the first set and the second set, the expert network with the preset label does not participate in the sorting, and the linear operators in the expert network with the preset label are directly divided into the first set.

[0012] For example, in the data processing method provided in at least one embodiment of this disclosure, determining the first threshold corresponding to the hybrid expert model layer based on the normalized sensitivity index corresponding to the plurality of expert networks includes: determining a percentile Y, wherein the percentile Y indicates that Y% of the expert networks in the plurality of expert networks are first-type expert networks, and the other expert networks are second-type expert networks, the linear operators in the first-type expert networks are assigned to the second set, the linear operators in the second-type expert networks are assigned to the first set, and Y is a positive integer less than 100; and determining the boundary threshold corresponding to the percentile Y as the first threshold corresponding to the hybrid expert model layer based on the percentile Y and the normalized sensitivity index corresponding to the plurality of expert networks.

[0013] For example, in the data processing method provided in at least one embodiment of this disclosure, the multiple linear operators are divided into a first set and a second set according to the business constraint parameters and the quantized sensitivity index of the multiple linear operators. This includes: for each hybrid expert model layer: sorting the normalized sensitivity index of the multiple expert networks included in the hybrid expert model layer in descending order to obtain a second sorting result; according to the second sorting result, the linear operators included in the expert networks whose normalized sensitivity index is greater than the first threshold corresponding to the hybrid expert model layer are assigned to the first set, and the linear operators included in other expert networks are assigned to the second set.

[0014] For example, in the data processing method provided in at least one embodiment of this disclosure, determining the business constraint parameters includes obtaining a first proportion parameter k, where k is a positive integer; and dividing the multiple linear operators into a first set and a second set according to the business constraint parameters and the quantization sensitivity indices of the multiple linear operators, including: sorting the quantization sensitivity indices of the multiple linear operators in descending order to obtain a first sorting result; and assigning the top k% of the linear operators in the first sorting result to the first set, and assigning the other linear operators to the second set.

[0015] For example, in a data processing method provided in at least one embodiment of this disclosure, the business constraint parameters include a first proportion parameter k and a boundary threshold. Based on the business constraint parameters and the quantization sensitivity index of the plurality of linear operators, the plurality of linear operators are divided into a first set and a second set, including: sorting the quantization sensitivity index of the plurality of linear operators in descending order to obtain a first sorting result; assigning the linear operators in the first sorting result that simultaneously satisfy the condition of being in the top k% and having a quantization sensitivity index greater than the boundary threshold to the first set, and assigning the other linear operators to the second set.

[0016] For example, in a data processing method provided in at least one embodiment of this disclosure, the network model includes multiple hybrid expert model layers, each hybrid expert model layer includes multiple expert networks, the business constraint parameters include a boundary threshold and a first threshold corresponding to each hybrid expert model layer, and the multiple linear operators are divided into a first set and a second set according to the business constraint parameters and the quantization sensitivity index of the multiple linear operators, including: sorting the quantization sensitivity index of the multiple linear operators in descending order to obtain a first sorting result; assigning the linear operators in the first sorting result whose quantization sensitivity index is greater than the boundary threshold to the first set; for each hybrid expert model layer: sorting the normalized sensitivity index of the multiple expert networks included in the hybrid expert model layer in descending order to obtain a second sorting result; assigning at least one linear operator to the first set according to the second sorting result, wherein the at least one linear operator is a linear operator included in the expert network whose normalized sensitivity index is greater than the first threshold corresponding to the hybrid expert model layer; and assigning the other linear operators in the multiple linear operators other than those assigned to the first set to the second set.

[0017] For example, in the data processing method provided in at least one embodiment of this disclosure, the business constraint parameter further includes a received strategy indication, wherein the strategy indication is used to indicate the strategy adopted when determining a preset judgment threshold when performing the binary grouping on the plurality of linear operators.

[0018] For example, at least one embodiment of the data processing method provided in this disclosure further includes: performing the quantization inference operation on the network model according to the precision allocation scheme; in response to the result of the quantization inference operation not meeting the precision requirements, determining new business constraint parameters to obtain a new precision allocation scheme determined based on the new business constraint parameters, and performing the quantization inference operation again based on the new precision allocation scheme; in response to the result of the quantization inference operation meeting the precision requirements, outputting a quantized network model, wherein the model structure of the quantized network model is the same as that of the network model, and the linear operators in the quantized network model adopt the quantization precision specified by the precision allocation scheme.

[0019] For example, in the data processing method provided in at least one embodiment of this disclosure, determining the quantization sensitivity index of the linear operator based on the product of the mean square activation value and the weighted quantization error of each linear operator includes: using the product of the mean square activation value and the weighted quantization error of each linear operator as the quantization sensitivity index of the linear operator; or using the logarithm of the product of the mean square activation value and the weighted quantization error of each linear operator as the quantization sensitivity index of the linear operator.

[0020] For example, in at least one embodiment of the data processing method provided in this disclosure, the data processing method is applied to the fields of natural language processing, text processing, computer vision, speech processing, and multimodal information fusion processing.

[0021] At least one embodiment of this disclosure provides a data processing apparatus, comprising: an acquisition module configured to perform forward inference on a network model and acquire the input activation matrix of each linear operator among a plurality of linear operators included in the network model; a reading module configured to read the weight matrix of each linear operator from a storage unit; a first determining module configured to determine the mean square activation value of the linear operator based on the input activation matrix of each linear operator, wherein the mean square activation value is used to measure the average energy of the input activation matrix of the linear operator; a second determining module configured to determine the weight quantization error of the linear operator based on the weight matrix of each linear operator, wherein the weight quantization error is used to measure the inherent error introduced when quantizing the weight matrix of the linear operator; and a third determining module configured to determine the mean square activation value of each linear operator. The product of the value and the weight quantization error is used as the quantization sensitivity index of the linear operator; the fourth determining module is configured to determine business constraint parameters, wherein the business constraint parameters include relevant configuration parameters used when performing binary grouping of the multiple linear operators; the partitioning module is configured to divide the multiple linear operators into a first set and a second set according to the business constraint parameters and the quantization sensitivity index of the multiple linear operators to obtain a precision allocation scheme for the multiple linear operators, wherein the minimum value of the quantization sensitivity index of the linear operators in the first set is greater than the maximum value of the quantization sensitivity index of the linear operators in the second set, and in the precision allocation scheme, the linear operators in the first set use a higher quantization precision than the linear operators in the second set for the quantization inference operation of the network model.

[0022] At least one embodiment of this disclosure provides an electronic device, including: a memory that non-transitoryly stores computer-executable instructions; and a processor configured to execute the computer-executable instructions, wherein the computer-executable instructions are executed by the processor to implement a data processing method according to any embodiment of this disclosure.

[0023] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the data processing method according to any embodiment of this disclosure.

[0024] The data processing method provided in at least one embodiment of this disclosure determines the quantization sensitivity index of each linear operator based on the product of the mean square activation value and the weight quantization error. The quantization sensitivity index considers not only the influence of the weights but also the potential influence of the input activation tensor, resulting in a precision allocation scheme that better meets the quantization precision requirements. The calculation method of the quantization sensitivity index is easy to implement, requiring only one forward inference, without the need to run a complex quantization process. This achieves a paradigm shift from evaluation to quantization, significantly reducing computational costs and resource consumption. For example, compared to the aforementioned Alpha search method, the computational cost can be reduced by 3-4 orders of magnitude, and GPU time can be reduced by more than 99%. The quantization sensitivity index in this disclosure has a large dynamic range, which can naturally distinguish between high-sensitivity and low-sensitivity objects without manual setting. In the quantization accuracy scheme, linear operators with high quantization sensitivity index, that is, operators that produce larger errors before and after quantization, use higher precision for quantization inference operations, while linear operators with low quantization sensitivity index, that is, operators that produce smaller errors before and after quantization, use lower precision for quantization inference operations. The overall quantization accuracy is close to lossless, and the use of low precision can be maximized to save video memory and bandwidth. While maintaining low-bit quantization performance, the accuracy of model quantization inference is improved, achieving the optimal balance between accuracy and efficiency. Attached Figure Description

[0025] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0026] Figure 1 A schematic flowchart illustrating a data processing method provided in at least one embodiment of this disclosure; Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this disclosure; Figure 3 A schematic block diagram of a data processing apparatus provided in at least one embodiment of this disclosure; Figure 4 A schematic block diagram of an electronic device provided in one embodiment of this disclosure; Figure 5 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of this disclosure; Figure 6 This is a schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0028] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components are omitted.

[0029] Large Language Models (LLMs) are deep learning-based neural network models with a large number of parameters, designed to understand and generate natural language. LLMs typically possess the following characteristics: they have a massive number of parameters, usually ranging from millions to billions, enabling them to learn rich linguistic features and patterns; they are based on deep neural networks, particularly those based on the Transformer architecture, which are large-scale and structurally complex, capable of capturing the complexity and diversity of language. Large language models play a crucial role in Natural Language Processing (NLP), enabling them to generate natural language text, perform machine translation, sentiment analysis, question answering systems, and many other tasks.

[0030] Inference using large language models requires significant amounts of GPU memory and computing power. To improve inference efficiency and reduce the deployment cost of large language models in AI (Artificial Intelligence) applications, quantization inference is typically performed. Quantization inference in neural networks refers to converting floating-point numbers (such as FP32, 32-bit floating-point numbers) weights and activation tensors in the neural network model into low-precision floating-point numbers or integers (such as int8 / uint8, 8-bit integers or unsigned integers), and performing inference calculations under this low-precision representation. This reduces model size and storage requirements, improves inference speed, and reduces power consumption. The process of converting high-precision floating-point numbers to low-precision floating-point numbers or integers is called quantization.

[0031] Currently, commonly used quantization methods for large language models include uniform quantization, smooth quantization (SmoothQuant / AWQ), mixed-precision quantization, and rotational quantization.

[0032] Uniform quantization is the most basic quantization method, mapping floating-point weights to equally spaced integer grid points. Smoothing quantization reduces overall error by migrating quantization difficulty between adjacent linear layers. The core idea is to find a channel-wise smoothing factor *s*, dividing the weights of the preceding linear layer by this factor (reducing the dynamic range) and multiplying the weights of the following linear layer by the factor (compensating for scaling), thus balancing the quantization difficulty between the two layers. Rotation quantization reduces the difficulty of uniform quantization by introducing random orthogonal rotation matrices before and after the linear layers, distributing outliers in the weights and activations across the channels. Mixed-precision quantization allows different layers in the model to use different quantization precisions, such as based on Hessian sensitivity analysis, layer-wise search based on quantization error, or schemes based on activation outlier detection.

[0033] MOE, short for Mixture of Experts, improves overall model performance by integrating the predictions of multiple expert networks. Its core idea is to use a gating network to distribute input data to different expert networks for processing, and then weight and merge the outputs of these networks to generate the final result. Each MOE layer contains dozens or even hundreds of expert networks, each independently possessing three linear operators: gate projection (gate_proj), up projection (up_proj), and down projection (down_proj).

[0034] The calculation process of the network model is performed layer by layer in units of layers. Typically, the output of the previous layer (or previous several layers) is used as the input of the subsequent layer (or subsequent several layers), thus forming data dependency between layers.

[0035] For example, for a network model structure containing an MOE structure, it includes 45 layers, for example, formed by stacking 45 transformer blocks, wherein each of the 42 layers is constituted by a first sublayer and a second sublayer in series, the first sublayer includes a multi-head self-Attention operator, a residual connection operator and a layer normalization operator, the second sublayer includes an MOE layer, a supporting residual connection operator and a layer normalization operator, and each MOE layer is equipped with 288 expert networks; each of the other 3 layers is composed of a first sublayer and a third sublayer in series, and the third sublayer includes a Feed-Forward Network (FFN), a residual connection operator and a layer normalization operator. In this network model, the total number of quantization target tensors can reach about 360000.

[0036] Existing quantization solutions have significant deficiencies in computational efficiency and discrimination capability when targeting large language models, especially large language models containing MOE layers.

[0037] On the one hand, the existing quantization methods as described above either rely on the solution of the Hessian matrix with huge computation amount, or need to repeatedly execute the quantization-evaluation loop. For tens of thousands of quantization objects in the Mixture of Experts (MoE) model, the computational overhead of the above methods is unacceptable. For example, performing smoothing factor (alpha) search on 36000 tensors one by one to find the optimal smoothing factor requires several hours or even several days of Graphics Processing Unit (GPU) time; however, the obtained per-tensor Mean Squared Error (MSE) value only spans a dynamic range of 3.6 times, and the Mean Squared Error (MSE) rankings of different experts in the layer are nearly random, which cannot effectively distinguish quantization objects with high sensitivity and low sensitivity to quantization errors. On the other hand, the dynamic range of the sensitivity index based on Mean Squared Error (MSE) search is too narrow to effectively distinguish quantization objects with high sensitivity and low sensitivity to quantization errors.

[0038] Moreover, the existing quantization methods cannot effectively identify the source of quantization artifacts, and ignore the systematic variation of the mean squared activation value with the model depth. The artifact refers to a systematic output error generated in the quantization inference process, which is manifested as phenomena such as repeated token injection and semantic confusion, for example, inserting the redundant token "on the contrary" in the process of generating an arithmetic formula: 3+4 on the contrary=7. When the quantization error of the layer highly sensitive to quantization error exceeds a certain threshold, the error is amplified layer by layer in subsequent layers, which eventually leads to the shift of the probability distribution of output tokens, resulting in observable abnormal outputs (artifacts).

[0039] In the aforementioned 45-layer network model structure, with a quantization scheme of 4 bits for weights and 8 bits for activations (W4A8), 99% of the output samples exhibit duplicate token artifacts, but existing methods cannot accurately locate the layer from which these artifacts originate. Traditional ablation experiments require testing each layer individually, and a complete ablation of 42 MOE layers requires at least 42 complete quantization-evaluation cycles, which is extremely time-consuming. Furthermore, most existing methods only focus on the characteristics of the weights or activations themselves (e.g., Activation-Aware Weight Quantization (AWQ) focuses on the channel importance of weights), ignoring the systematic increase in mean squared activation values ​​with model depth. In the Transformer architecture, residual connections cause deep activation values ​​to be much larger than shallow ones, amplifying the quantization error of deep layers by several orders of magnitude. This crucial phenomenon has not been captured by existing quantization sensitivity metrics.

[0040] Furthermore, existing quantization methods lack specific handling for the heterogeneity among different experts in hybrid expert models. In hybrid expert models, different experts are routed to tokens of different semantic categories, resulting in significant differences in their activation distribution and weight characteristics: 24.3% of experts receive fewer than 128 tokens, making their statistics unreliable; while some experts process high-frequency semantics, exhibiting extremely high activation power. Existing quantization methods employ a uniform quantization strategy for all experts, failing to distinguish these heterogeneities.

[0041] Furthermore, some current methods attempt to use higher-order statistics of weights as quantization sensitivity indicators, but all have fundamental flaws. For example, kurtosis, percentile, and standard deviation (Std) are used as quantization sensitivity indicators. The common flaw of these statistics is that they attempt to infer quantization difficulty solely from the distribution shape of the weights. However, from the shallow to the deep layers of the network model, input activation gradually increases, and the impact of input activation on accuracy also gradually increases. These statistics completely ignore the error amplification effect of input activation, leading to the systematic under-detection of deep, highly sensitive (sensitive to quantization error) layers.

[0042] This disclosure provides at least one embodiment of a data processing method and apparatus, an electronic device, and a non-transient storage medium. The data processing method includes: performing forward inference on a network model and acquiring the input activation matrix of each linear operator among a plurality of linear operators included in the network model; reading the weight matrix of each linear operator from a storage unit; determining the mean square activation value of the linear operator based on the input activation matrix of each linear operator, wherein the mean square activation value is used to measure the average energy of the input activation matrix of the linear operator; determining the weight quantization error of the linear operator based on the weight matrix of each linear operator, wherein the weight quantization error is used to measure the inherent error introduced when quantizing the weight matrix of the linear operator; and determining the mean square activation value and weight quantization error of each linear operator. The product of the differences is used as the quantization sensitivity index of the linear operator; business constraint parameters are determined, including relevant configuration parameters used when performing binary grouping of multiple linear operators; based on the business constraint parameters, multiple linear operators are divided into a first set and a second set to obtain a precision allocation scheme for multiple linear operators, wherein the minimum value of the quantization sensitivity index of the linear operators in the first set is greater than the maximum value of the quantization sensitivity index of the linear operators in the second set. In the precision allocation scheme, the linear operators in the first set use a higher quantization precision than the linear operators in the second set for quantization inference operations of the network model.

[0043] The data processing method provided in at least one embodiment of this disclosure determines the quantization sensitivity index of each linear operator based on the product of the mean square activation value and the weight quantization error. The quantization sensitivity index considers not only the influence of the weights but also the potential influence of the input activation tensor, resulting in a precision allocation scheme that better meets the quantization precision requirements. The calculation method of the quantization sensitivity index is easy to implement, requiring only one forward inference, without the need to run a complex quantization process. This achieves a paradigm shift from evaluation to quantization, significantly reducing computational costs and resource consumption. For example, compared to the aforementioned Alpha search method, the computational cost can be reduced by 3-4 orders of magnitude, and GPU time can be reduced by more than 99%.

[0044] The quantization sensitivity index in this disclosure has a large dynamic range, which can naturally distinguish between high-sensitivity and low-sensitivity objects without manual setting. In the quantization accuracy scheme, linear operators with high quantization sensitivity index, that is, operators that produce larger errors before and after quantization, use higher precision for quantization inference operations, while linear operators with low quantization sensitivity index, that is, operators that produce smaller errors before and after quantization, use lower precision for quantization inference operations. The overall quantization accuracy is close to lossless, and the use of low precision can be maximized to save video memory and bandwidth. While maintaining low-bit quantization performance, the accuracy of model quantization inference is improved, achieving the optimal balance between accuracy and efficiency.

[0045] The data processing method provided in this disclosure can be applied to the data processing apparatus provided in this disclosure, which can be configured on an electronic device. The electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a mobile phone, tablet computer, laptop computer, or other hardware device.

[0046] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, but this disclosure is not limited to these specific embodiments.

[0047] Figure 1 This is a schematic flowchart illustrating a data processing method provided in at least one embodiment of this disclosure. Figure 1 As shown, at least one embodiment of the present disclosure provides a data processing method including steps S10-S70.

[0048] In step S10, forward inference is performed on the network model, and the input activation matrix of each linear operator in the multiple linear operators included in the network model is collected.

[0049] For example, the network model can be a large language model. This disclosure does not impose specific restrictions on the structure of large language models. For instance, the network model can be a large language model that includes hybrid expert network layers. Of course, this disclosure is not limited to this; other network models that include linear operators and require quantization inference can also be used as the network model of this disclosure.

[0050] For example, operators in a network model typically refer to the basic mathematical operations or processes used in the network layers. These operators are used to construct the various layers and components of the network, enabling data transfer, transformation, and computation. They are the fundamental building blocks of the network model, defining its structure and computational flow, including inputs, outputs, and intermediate computations. In the model, the connections between operators form a directed graph, reflecting the order of computation for different operations. By combining these operators, complex and powerful network models can be constructed to handle various complex tasks and data.

[0051] In network models, linear operators are defined as tensor transformations that satisfy additivity and homogeneity mapping rules and can be represented as weighted sums with constant biases. Typical examples in deep learning frameworks include fully connected operators, various convolution operators and transposed convolution operators, feature projection, mean pooling, and the affine scaling offset part of normalization operators. However, activation functions and max pooling operators are not linear operators.

[0052] Since most of the computing power of the network model is concentrated on the linear operators, and the mathematical structure of the weighted summation of the linear operators can be converted into integer operations without loss, and the quantization error is easy to calibrate and control, the data processing method provided in at least one embodiment of this disclosure calculates the quantization sensitivity index of the linear operators, classifies the linear operators, and selects a portion of them for low-bit quantization.

[0053] For example, it can receive a network model to be quantized and a calibration dataset, use the calibration dataset to perform forward inference on the network model, and collect the input activation matrix of each linear operator during the inference process.

[0054] For example, specific functions, such as hook functions, can be used to automatically capture the input activation matrix of linear operators. Alternatively, custom kernel functions can be used to capture the input activation matrix of linear operators in real time. Of course, those skilled in the art can employ other feasible methods to collect the input activation matrix during forward inference, which will not be listed here.

[0055] In step S20, the weight matrix of each linear operator is read from the storage unit.

[0056] The weight matrix is ​​pre-computed and stored in the storage component of the processing device used for network model inference, for example, in the processing device's memory. For instance, the weight matrix for each linear operator can be read directly from the processing device's memory.

[0057] In step S30, the mean square activation value of the linear operator is determined based on the input activation matrix of each linear operator.

[0058] Mean Squared Activation (MSA) is used to measure the average energy of the input activation matrix of a linear operator.

[0059] The mean squared activation value (MSU) is a measure of signal power, reflecting the magnitude of the activation value processed by the linear operator. A higher MSU indicates a stronger input signal to the linear operator, and a greater amplification of quantization errors at the output. As a factor in quantization sensitivity, the MSU characterizes the "error amplification" effect. Particularly in Transformer models, the MSU increases exponentially with depth, making deeper linear layers more sensitive to quantization errors.

[0060] In step S40, the weight quantization error of the linear operator is determined based on the weight matrix of each linear operator.

[0061] Weight Quantization Error (WQE) measures the inherent error introduced when quantizing the weight matrix of a linear operator. It represents an upper bound of the uniform quantization error. For example, with quantization to INT4 (4-bit integer) and a quantization step size q = row_max / 7, the weight quantization error is proportional to row_max². The weight quantization error characterizes the inherent difficulty of quantizing the weight matrix, where row_max represents the maximum absolute value of a row in the weight matrix. As another factor in quantization sensitivity, the weight quantization error characterizes the "quantization difficulty."

[0062] For example, steps S10, S30 and S20, S40 can be executed in parallel to improve computational efficiency. For instance, during the forward inference process of the network model, the input activation matrix is ​​collected layer by layer and the mean square activation value is calculated; simultaneously, the weight matrices of each linear operator in the network model are read, and the weight quantization error is calculated. In other words, the data acquisition process and subsequent calculation processes can be performed independently and in parallel to save computational time.

[0063] In step S50, the quantization sensitivity index of the linear operator is determined based on the product of the mean square activation value and the weighted quantization error of each linear operator.

[0064] For example, for each linear operator, the mean square activation value (MSA) is obtained in step S30, and the weighted quantization error (WQE) is obtained in step S40. The quantization sensitivity index of the linear operator is S = MSA × WQE. That is, the product of the mean square activation value and the weighted quantization error of the linear operator can be directly used as the quantization sensitivity index of the linear operator.

[0065] For example, in actual measurements, the quantization sensitivity index can span a dynamic range of 88,000,000 times. This disclosure supports logarithmic space calculation to avoid numerical overflow. That is, after obtaining the mean square activation value (MSA) and quantization weight error (WQE) of each linear operator, a logarithmic operation can be performed on the product of the mean square activation value and the weight quantization error of the linear operator. The result of the logarithmic operation is used as the quantization sensitivity index to avoid numerical overflow.

[0066] Regardless of which method of calculating the quantization sensitivity index is used, the quantization sensitivity index is essentially determined solely by the product of the mean square activation value of the linear operator and the weighted quantization error.

[0067] The physical meaning of the quantification sensitivity index can be understood from the following three dimensions: (1) Error amplification perspective: Mean Squared Activation (MSA) is the error amplification factor. The same weighted quantization error will be amplified many times in a linear operator with a high MSA, resulting in a greater disturbance to the final output. In the Transformer residual connection structure, the continuous accumulation of residual flow leads to a much larger MSA in the deeper layers than in the shallower layers, which makes the deeper layers naturally more sensitive to quantization.

[0068] (2) Quantization difficulty perspective: Weight quantization error is an inherent difficulty in quantization. Weight matrices with high weight quantization error have a large in-row dynamic range, which forces the quantization step size to be increased, resulting in finer weight values ​​being approximated by coarse-grained values.

[0069] (3) Comprehensive risk assessment: The quantization sensitivity index combines two independent factors, "how much the error is amplified" and "how large the error itself is", to form a complete assessment of quantization risk. A layer with high weight quantization error but weak input signal (low mean square activation value) may not have serious output perturbation; conversely, a layer with low weight quantization error but extremely strong input signal (high mean square activation value) may also produce significant output perturbation.

[0070] The quantization sensitivity index reflects the sensitivity of a linear operator to the error introduced by quantization. The higher the quantization sensitivity index, the greater the error of the linear operator before and after quantization, indicating high sensitivity to quantization; conversely, the lower the quantization sensitivity index, the smaller the error of the linear operator before and after quantization, indicating low sensitivity to quantization.

[0071] For the quantization sensitivity index provided in at least one embodiment of this disclosure, no independence assumptions need to be made regarding the channel distribution of input activation. The calculation of the quantization sensitivity index is an accurate result guaranteed by the orthogonality of M (term dimension) and N (channel dimension) in linear algebra.

[0072] While low-precision matrix computation is highly efficient, the loss of precision becomes increasingly pronounced as the number of bits decreases. To mitigate this loss, low-precision matrix multiplication introduces a scaling factor to maximize the numerical expressive power of the low-precision tensor. For example, low-precision matrix multiplication includes matrix multiplication for low-precision tensors (such as FP4, FP8, etc.).

[0073] For low-precision matrix multiplication, such as D = A' × B' + C', where A', B', C', and D are all high-precision tensors, the general matrix multiplication operation using scaling factors can be described as follows: D = (α⊙A) × (β⊙B) + σ⊙C or D = (α⊙A) × (β⊙B) + σ⊙C β)⊙(A×B)+σ⊙C Where ⊙ represents element-wise multiplication. denoted as outer product, × denotes matrix multiplication, α is the scaling factor of tensor A', β is the scaling factor of tensor B', σ is the scaling factor of tensor C', γ is the scaling factor of tensor D, and the quantized tensors A, B, C, and D are the low-precision tensors obtained by floating-point quantization of tensors A', B', C', and D', respectively.

[0074] In this disclosure, the computation of the linear operator is defined as Y=XW T , where X represents the input activation matrix of size [M,K], M is the number of labeled lexical units, K is the hidden layer feature dimension, W represents the weight matrix of size [N,K], N is the number of output channels, and matrix multiplication is performed by inner accumulation along the K dimension. M and N are independent orthogonal dimensions.

[0075] Quantization is performed in N dimensions. One or more rows in the input activation matrix of each linear operator share the same scaling factor, and one or more columns in the weight matrix of each linear operator share the same scaling factor.

[0076] Under the above premise, the inventors, through rigorous mathematical verification, derived that the quantization error energy can be uniquely determined by the product of the weighted quantization error and the mean square activation value, and that this factorization is accurate (relying only on the element-wise independence of the weighted quantization error, without making any assumptions about the channel distribution of the input activation matrix).

[0077] Based on the above premise (i.e., one or more rows in the input activation matrix of each linear operator share the same scaling factor, and one or more columns in the weight matrix of each linear operator share the same scaling factor), in some embodiments, step S30 may include: calculating the square value of each of the multiple elements included in the input activation matrix of the linear operator; and determining the mean of the square values ​​of the multiple elements as the mean square activation value of the linear operator.

[0078] The mean square activation value (MSU) measures the input energy density (i.e., the "error amplifier") flowing through a linear operator. In the data processing method provided in at least one embodiment of this disclosure, the quantization sensitivity index uses the MSU as one of the factors. Unlike other mathematical forms such as the sum of squares or root mean square, the MSU can eliminate statistical bias caused by different expert networks due to different routing frequencies (different number of tokens, different M) by taking the average value, thereby improving the accuracy of the quantization sensitivity index.

[0079] Mean Squared Activation Value (MSA) = Mean(X) 2 Here, Mean() represents the mean function, X 2 This represents the element-wise squared value of the input activation matrix X.

[0080] Of course, for higher-dimensional input activation tensors, such as in the case of multiple batches, the mean squared activation values ​​of the input activation matrices from multiple batches can be accumulated as statistics to maintain the stability of the statistics.

[0081] Based on the above premises, in some embodiments, step S40 may include: determining the maximum value among the absolute values ​​of each output channel in the weight matrix of the linear operator; determining the square of the maximum value among the absolute values ​​of each output channel as the intermediate result corresponding to each output channel; and determining the mean of the multiple intermediate results corresponding to the multiple output channels included in the weight matrix of the linear operator as the weight quantization error of the linear operator.

[0082] For example, after reading the weight matrix W for each linear operator, the maximum absolute value of each row in the weight matrix W is first calculated as row_max(Wi), where Wi represents the i-th row of the weight matrix; the intermediate result corresponding to each output channel is represented as row_max(Wi). 2 Weighted quantization error WQE = Mean( ).

[0083] Quantization weight error is a measure of the "absolute sluggishness" (i.e., the source of error) of the weight matrix under the quantization method described above. Unlike the use of the coefficient of variation, the data processing method provided in at least one embodiment of this disclosure uses the mean square value of the channel maxima to represent the quantization weight error, which can more realistically reflect the weight error.

[0084] Then, the precision allocation decision of linear operators can be guided by the quantization sensitivity index of the linear operators, accurately identifying operators that are highly sensitive to quantization errors and placing them in the first set, while other operators are placed in the second set.

[0085] In step S60, the business constraint parameters are determined.

[0086] Business constraint parameters include relevant configuration parameters used when performing binary grouping of multiple linear operators. For example, relevant configuration parameters may include preset decision thresholds for binary grouping.

[0087] For example, business constraint parameters can be directly specified by the user, such as by receiving user input. For example, after obtaining the quantization sensitivity indices of multiple linear operators in step S50, the quantization sensitivity indices of multiple linear operators can be sorted in descending order to obtain a first sorting result, which is provided to the user. The user sets a preset judgment threshold based on the first sorting result, and then performs subsequent binary grouping based on the preset judgment threshold set by the user.

[0088] For example, business constraint parameters can be obtained by analyzing the quantization sensitivity index of multiple linear operators, as detailed in the following text.

[0089] For example, the relevant configuration parameters may also include a strategy indicator for binary grouping, which indicates the strategy to be used when binary grouping multiple linear operators.

[0090] For example, strategies can include a first strategy, a second strategy, a third strategy, and so on.

[0091] The first strategy involves dividing multiple linear operators into a first set and a second set based on a preset decision threshold, using a fixed value. For example, the first strategy could be the absolute threshold strategy described later, and the preset decision threshold could include the boundary threshold used in the absolute threshold strategy.

[0092] The second strategy involves classifying expert networks into either a first-class or second-class expert network based on a preset decision threshold, using the expert networks in the hybrid expert network layer as units. Linear operators of the first-class expert networks in the network model are assigned to a first set, and linear operators of the second-class expert networks in the network model are assigned to a second set. For example, the second strategy could be the layer normalization strategy described later, and the preset decision threshold could include the first threshold used in the layer normalization strategy.

[0093] For example, a third strategy includes quantitatively dividing multiple linear operators into a first set and a second set based on a preset decision threshold. For instance, the third strategy could be the proportional method described later, where the preset decision threshold could include a first proportional parameter used in the proportional method.

[0094] Of course, at least two of the above three strategies can be combined, and this disclosure does not impose any specific restrictions on this.

[0095] For example, in some embodiments, the policy indication can be determined by receiving user input, such as deciding whether to use a first policy, a second policy, a third policy, or a combination of policies based on user input. The predetermined threshold in the policy can be automatically determined with reference to the aforementioned scheme without waiting for user input.

[0096] For example, in other embodiments, the appropriate strategy can be automatically determined by analyzing the characteristics of the network model itself. For instance, for a hybrid expert model, when it is necessary to identify high-sensitivity expert networks within the same layer, a second strategy can be used; for instance, if the network model requires that linear operators with quantization sensitivity exceeding a certain threshold must use high precision, a first strategy can be used; for instance, when the network model has strict memory budget constraints, a third strategy can be used.

[0097] By combining preset judgment thresholds in the business constraint parameters, and following the strategy instructions, and using the quantization sensitivity indices of multiple linear operators, high-sensitivity quantization objects (high quantization sensitivity index, located in the first set) and low-sensitivity quantization objects (low quantization sensitivity index, located in the second set) can be accurately identified. Therefore, based on the quantization sensitivity index, the source of artifacts can be accurately identified and located. Applying a high-precision quantization scheme to these linear operators can completely eliminate artifacts while retaining the advantage of quantization accuracy, thus improving the accuracy of the model inference results.

[0098] In step S70, based on the business constraint parameters and the quantization sensitivity index of multiple linear operators, the multiple linear operators are divided into a first set and a second set to obtain a precision allocation scheme for the multiple linear operators.

[0099] The minimum quantization sensitivity index of the linear operators in the first set is greater than the maximum quantization sensitivity index of the linear operators in the second set. In the precision allocation scheme, the linear operators in the first set use a higher quantization precision than the linear operators in the second set for quantization inference operations of the network model.

[0100] The linear operators in the first set have higher quantization sensitivity, while those in the second set have lower quantization sensitivity. In other words, the linear operators in the first set are more sensitive to quantization errors, while those in the second set are less sensitive.

[0101] The first set is the high-precision group, meaning that the linear operators in the group all use a high-precision quantization scheme. The second set is the low-precision group, meaning that the linear operators in the group all use a lower-precision quantization scheme than the first set.

[0102] For example, quantization schemes may include W8A16 (8-bit weight quantization, 16-bit activation), W4A16 AWQ (4-bit weight AWQ quantization, 16-bit activation), W8A8 FP8 (both weight and activation are 8-bit floating-point quantization), W4A8FP8 (4-bit weight quantization, 8-bit activation quantization), etc. This disclosure does not impose specific limitations on them.

[0103] The resulting precision allocation scheme provides the specific quantization scheme used by each linear operator.

[0104] For example, in the precision allocation scheme, the quantization precision of the weight matrix and input activation matrix of the linear operators in the first set is the original precision, while the quantization precision of the weight matrix and input activation matrix of the linear operators in the second set is lower than the original precision.

[0105] For example, the raw precision is the highest precision of the data before any quantization, such as FP16 or BF16, while the quantization precision of the second set is the lower precision after the data has been quantized, such as INT8, INT4, etc.

[0106] For example, in one example, the linear operators in the first set can use the W8A16 quantization scheme, and the linear operators in the second set can use the W4A8 quantization scheme. In another example, the linear operators in the first set can use the BF16 quantization scheme, and the linear operators in the second set can use the W4A8 quantization scheme. Of course, this disclosure is not limited to these; the linear operators in the first set can use higher precision for quantization inference, and the linear operators in the second set can use lower precision relative to the linear operators in the first set for quantization inference.

[0107] In the data processing method provided in at least one embodiment of this disclosure, the quantization sensitivity index not only considers the weight error but also the error amplification effect brought about by the input activation matrix, more accurately reflecting the sensitivity of the linear operator to the quantization error, thereby obtaining a more reasonable precision allocation scheme and achieving near-lossless quantization precision. The computational cost of the quantization sensitivity index is extremely low, requiring no quantization process, significantly reducing quantization tuning costs, reducing hardware resource consumption, and improving model deployment efficiency. The calculation of the quantization sensitivity index depends only on the input activation matrix and weight matrix of the linear operator, without limiting the network architecture, and is applicable to all network model structures including linear operators, with a wide range of applications. The quantization sensitivity index has a wide dynamic range, naturally distinguishing between high-sensitivity and low-sensitivity objects. Based on the quantization sensitivity index, the source of artifacts can be accurately located, completely eliminating artifacts while retaining the precision advantage of low-precision quantization, and improving the accuracy of network model inference results.

[0108] For example, in some embodiments, an absolute threshold strategy (first strategy) can be used to perform binary grouping of linear operators.

[0109] For example, some network models have explicit business constraints, such as linear operators whose quantization sensitivity exceeds a certain threshold must use a high-precision quantization scheme.

[0110] For example, step S60 may include: determining the global distribution statistics of the quantization sensitivity indices of multiple linear operators based on the quantization sensitivity indices of multiple linear operators, wherein the global distribution statistics include the boundary thresholds corresponding to the preset percentiles of multiple linear operators; determining the percentile X, wherein the percentile X represents that X% of the linear operators are assigned to the second set, and the remaining 1-X% of the linear operators are assigned to the first set, and X is a positive integer less than 100; and determining the boundary threshold corresponding to the percentile X as a business constraint parameter based on the percentile X and the global distribution statistics.

[0111] For example, based on business constraint parameters and the quantization sensitivity index of multiple linear operators, the multiple linear operators can be divided into a first set and a second set. This can include: sorting the quantization sensitivity index of the multiple linear operators in descending order to obtain a first sorting result; and based on the first sorting result, assigning the linear operators whose quantization sensitivity index is greater than the threshold corresponding to the percentile X to the first set, and assigning the other linear operators to the second set.

[0112] For example, firstly, based on the quantization sensitivity indices of multiple linear operators obtained in step S50, the global distribution statistics of the quantization sensitivity indices of multiple linear operators are determined. The global distribution statistics may include some preset percentile thresholds. For example, the global distribution statistics may also include global minimum values, global maximum values, etc.

[0113] For example, preset percentiles can include P99, P95, P90, etc. Since network models typically expect more linear operators to use low precision to improve inference efficiency and reduce memory usage, a preset percentile of P99, for example, serves as a threshold to distinguish between the smaller 99% of all quantization sensitivity values ​​and the largest 1%; a preset percentile of P95, for example, serves as a threshold to distinguish between the smaller 95% of all quantization sensitivity values ​​and the largest 5%; and so on.

[0114] Next, the percentile X is determined. For example, the percentile X could be input from the user based on experience.

[0115] For example, the percentile X can be specified internally by the system. For instance, it can start by trying X=99 or X=95. In the first attempt, using X=99, based on the boundary threshold determined in the aforementioned global distribution statistics, the boundary threshold corresponding to percentile P99 is determined as the business constraint parameter. If the accuracy allocation scheme obtained in the first attempt does not meet the accuracy requirements, a new boundary threshold can be determined in the second attempt using X=95.

[0116] Then, the quantization sensitivity indices of multiple linear operators are sorted in descending order to obtain the first sorting result.

[0117] Based on the first sorting result, linear operators with quantization sensitivity indices greater than the boundary threshold corresponding to P99 are assigned to the first set, and other linear operators are assigned to the second set, thus obtaining the first version of the precision allocation scheme.

[0118] For example, in some embodiments, the data processing method provided in at least one embodiment of this disclosure further includes: performing quantization inference operations on the network model according to a precision allocation scheme; in response to the result of the quantization inference operation not meeting the precision requirements, determining new business constraint parameters to obtain a new precision allocation scheme based on the new business constraint parameters, and performing quantization inference operations again based on the new precision allocation scheme; in response to the result of the quantization inference operation meeting the precision requirements, outputting a quantized network model, wherein the model structure of the quantized network model is the same as that of the network model to be quantized, and the linear operators in the quantized network model adopt the quantization precision specified by the precision allocation scheme. That is, the precision of the linear operators in the second set of the quantized network model has been downgraded, that is, low-bit quantization is used for these linear operators.

[0119] For example, quantization inference operations can be performed based on the first version of the precision allocation scheme to output a quantized network model.

[0120] For example, the results of the quantization inference operation of the first version of the precision allocation scheme can be verified. If the precision requirements are not met, X can be reduced to determine new business constraint parameters. For example, the boundary threshold can be determined by X=95, and multiple linear operators can be binary-grouped according to the new boundary threshold to obtain the second version of the precision allocation scheme. The quantization inference operation can then be performed again based on the second version of the precision allocation scheme.

[0121] If the result of the quantization inference operation of the second precision allocation scheme meets the precision requirements, then the quantization network model corresponding to the second precision allocation scheme is output. If the result of the quantization inference operation of the second precision allocation scheme still does not meet the precision requirements, then a new X can be determined, and a new boundary threshold can be determined based on the new X, and the above process can be continued. This will not be elaborated here.

[0122] For example, accuracy requirements can be judged from multiple perspectives. For instance, inference can be performed using the model's raw accuracy (e.g., BF16) before any quantization, providing a benchmark for accuracy. Accuracy criteria can include flexible accuracy, strict accuracy, and artifact rate. Flexible accuracy describes the accuracy of the final answer, indicating a correct answer. It has higher tolerance for errors in intermediate inference steps and is a more suitable indicator for large-scale question-and-answer scenarios. Strict accuracy imposes strong constraints on accuracy; not only must the final answer be correct, but every step of calculation and derivation must be error-free. This is the most stringent standard and better reflects the degree to which quantization disrupts the integrity of the reasoning logic. Artifact rate reflects the proportion of samples with false calculation biases caused by numerical offsets and parameter perturbations introduced during quantization to compress accuracy.

[0123] For example, meeting accuracy requirements could mean that the accuracy drop relative to the benchmark is less than a predetermined threshold, or that the artifact rate is 0%, etc. Of course, other methods can also be used to determine whether accuracy requirements are met, which will not be elaborated here.

[0124] In actual testing, taking the aforementioned network model structure with 45 layers as an example, based on the absolute threshold strategy and the boundary threshold corresponding to percentile P99, the linear operators in the three deepest layers of the 45 layers can be accurately divided into the first set. These three layers are the source of artifacts. Skipping the low-bit quantization of these three layers can completely eliminate artifacts.

[0125] For example, in some embodiments, the network model includes multiple hybrid expert model layers, each of which includes multiple expert networks. For instance, a layer normalization strategy (the second strategy) can be employed to quantize sensitivity judgments on a per-expert-network basis.

[0126] For example, in some embodiments, step S60 may include: for each hybrid expert model layer: determining a quantization sensitivity index for the hybrid expert model layer, wherein the quantization sensitivity index of the hybrid expert model layer is the average of the quantization sensitivity indices of all linear operators included in the hybrid expert model layer; determining normalized sensitivity indices corresponding to the multiple expert networks included in the hybrid expert model layer based on the quantization sensitivity index corresponding to the hybrid expert model layer; and determining a first threshold corresponding to the hybrid expert model layer based on the normalized sensitivity indices corresponding to the multiple expert networks, wherein the business constraint parameters include the first thresholds corresponding to the multiple hybrid expert model layers.

[0127] For example, in some embodiments, determining the normalized sensitivity index corresponding to each of the multiple expert networks included in the hybrid expert model layer based on the quantization sensitivity index corresponding to the hybrid expert model layer may include: for each expert network, determining the average value of the quantization sensitivity indices of the multiple linear operators included in the expert network as the quantization sensitivity index corresponding to the expert network; and using the ratio of the quantization sensitivity index corresponding to the expert network to the quantization sensitivity index corresponding to the hybrid expert model layer as the normalized sensitivity index corresponding to the expert network.

[0128] For example, determining the first threshold corresponding to the hybrid expert model layer based on the normalized sensitivity index corresponding to multiple expert networks can include: determining the percentile Y, where the percentile Y indicates that Y% of the expert networks are first-class expert networks, and the other expert networks are second-class expert networks, the linear operators in the first-class expert networks are classified into the second set, the linear operators in the second-class expert networks are classified into the first set, and Y is a positive integer less than 100; and determining the boundary threshold corresponding to the percentile Y as the first threshold corresponding to the hybrid expert model layer based on the percentile Y and the normalized sensitivity index corresponding to multiple expert networks.

[0129] For example, based on business constraint parameters and the quantized sensitivity indices of multiple linear operators, the multiple linear operators can be divided into a first set and a second set. This can include: for each hybrid expert model layer: sorting the normalized sensitivity indices of the multiple expert networks included in the hybrid expert model layer in descending order to obtain a second sorting result; based on the second sorting result, assigning the linear operators included in the expert networks whose normalized sensitivity indices are greater than the first threshold corresponding to the hybrid expert model layer to the first set, and assigning the linear operators included in the other expert networks to the second set.

[0130] For example, taking the i-th hybrid expert model layer MOE_i as an example (i is a positive integer and less than the total number of hybrid expert model layers), the average value of the quantization sensitivity index of all linear operators included in the hybrid expert model layer MOE_i is determined as the quantization sensitivity index s_moe(i) of the hybrid expert model layer MOE_i.

[0131] Then, for each expert network in the hybrid expert model layer MOE_i, taking the j-th expert network expert_j as an example (j is a positive integer and less than or equal to the total number of expert networks included in the hybrid expert model layer MOE_i), the average value of the quantization sensitivity index of the multiple linear operators included in the expert network expert_j is determined as the quantization sensitivity index s_expert(j) corresponding to the expert network expert_j.

[0132] Then, the normalized sensitivity index s_norm(j) corresponding to the expert network expert_j is determined to be s_expert(j) / s_moe(i).

[0133] Thus, the normalized sensitivity index corresponding to each expert network in the hybrid expert model layer MOE_i is obtained.

[0134] After obtaining the normalized sensitivity indexes for all expert networks in the hybrid expert model layer MOE_i, the expert networks can be binary-grouped by referring to the aforementioned absolute threshold strategy.

[0135] Specifically, the percentile Y is first determined, which can be input from the user based on experience.

[0136] For example, the percentile Y can be specified internally by the system. For instance, it can start by trying Y=95. In the first attempt, the threshold corresponding to Y=95, P95, is used to distinguish between the smaller 95% of the normalized sensitivity values ​​in the hybrid expert model layer MOE_i and the largest 5% of the values. The threshold corresponding to the percentile Y serves as the first threshold for the hybrid expert model layer.

[0137] Next, the normalized sensitivity indices of the multiple expert networks included in the hybrid expert model layer MOE_i are sorted in descending order to obtain a second sorting result. Based on the second sorting result, the linear operators included in the expert networks whose normalized sensitivity indices are greater than the first threshold corresponding to P95 are assigned to the first set, and the linear operators included in the other expert networks are assigned to the second set. This yields the first version of the precision allocation scheme.

[0138] Subsequently, based on the first version of the precision allocation scheme, quantization inference operations are performed to output a quantized network model. The model structure of the quantized network model is the same as that of the network model to be quantized. The quantized network model has undergone precision downgrading of the linear operators in the second set, that is, low-bit quantization is used for these linear operators.

[0139] For example, the results of the quantization inference operation of the first version of the precision allocation scheme can be further verified. If the results of the quantization inference operation of the first version of the precision allocation scheme do not meet the precision requirements, Y can be reduced to determine new business constraint parameters. For example, Y=92 can be used to determine the boundary threshold, and multiple expert networks can be classified according to the new boundary threshold to obtain the second version of the precision allocation scheme. Then, the quantization inference operation can be performed again based on the second version of the precision allocation scheme.

[0140] If the result of the quantization inference operation of the second precision allocation scheme meets the precision requirements, then the quantized network model obtained by the second precision allocation scheme is output. If the result of the quantization inference operation of the second precision allocation scheme still does not meet the precision requirements, then a new Y can be determined, and a new boundary threshold can be determined based on the new Y, and the above process can continue. This will not be elaborated here.

[0141] The data processing method provided in at least one embodiment of this disclosure can perform quantization sensitivity judgment on an expert network basis, identify high-sensitivity experts (more sensitive to quantization errors) and low-sensitivity experts (less sensitive to quantization errors) in each hybrid expert model layer, and thus adopt different quantization strategies for different expert networks, providing fine-grained quantization guidance for the quantization of hybrid expert models.

[0142] For example, in other embodiments, the normalized sensitivity index corresponding to each of the multiple expert networks included in the hybrid expert model layer is determined according to the quantization sensitivity index corresponding to the hybrid expert model layer. This may include: for expert networks whose received number of tokens is less than a preset threshold, a preset label is added to the expert network. In the process of dividing multiple linear operators into a first set and a second set, the expert network with the preset label does not participate in the sorting, and the linear operators in the expert network with the preset label are directly divided into the first set.

[0143] In actual testing, about 24.3% of the experts in the hybrid expert model layer received fewer than 128 terms, and the statistical measure of their mean square activation values ​​was unreliable.

[0144] Therefore, when determining the normalized sensitivity index corresponding to the expert network, we can judge whether the number of tokens received by the expert network is greater than or equal to the preset threshold. For expert networks that are greater than the preset threshold, these expert networks process high-frequency semantics and have extremely high activation power. The normalized sensitivity index corresponding to the expert network can be determined in the aforementioned way, which will not be elaborated here.

[0145] For example, for an expert network whose value is less than a preset threshold, the median of the mean square activation value of the linear operator in another reliable expert network in the same layer (an expert network whose number of tokens received is greater than the preset threshold) can be used to replace the mean square activation value of the linear operator in the expert network. Then, the normalized sensitivity index corresponding to the expert network can be determined by referring to the aforementioned steps.

[0146] For example, for expert networks that are less than a preset threshold, a preset label can be added, such as "statistics are unreliable". These expert networks do not participate in the ranking process when obtaining the second ranking results, and there is no need to calculate the quantification sensitivity index. The linear operators in these expert networks are directly assigned to the first set, and a conservative high-precision strategy is used.

[0147] For example, in other embodiments, a proportional method (third strategy) can also be used to group linear operators into binary groups.

[0148] For example, in some embodiments, the business constraints of some network models require strict budget constraints on the use of high-precision linear operators, such as requiring that the number of high-precision linear operators used cannot exceed 10% of the number of bus linear operators.

[0149] For example, step S60 may include: obtaining a first scaling parameter k, where k is a positive integer. For example, the first scaling parameter may be input by the user, or the first scaling parameter may be determined according to the business constraints of the network model itself.

[0150] For example, based on business constraint parameters and the quantization sensitivity index of multiple linear operators, the multiple linear operators can be divided into a first set and a second set. This can include: sorting the quantization sensitivity index of the multiple linear operators in descending order to obtain a first sorting result; assigning the top k% of the linear operators in the first sorting result to the first set, and assigning the other linear operators to the second set.

[0151] For example, we can take r = ceil(N) k), ceil represents the floor function, N is the total number of linear operators, and r is a positive integer. Then, the linear operators ranked in the first r positions in the first sorting result are assigned to the first set, and the other linear operators are assigned to the second set.

[0152] In this third strategy, the number of high-precision linear operators used is completely controllable, making it suitable for scenarios with strict memory budgets.

[0153] The three strategies mentioned above (the first strategy, the second strategy, and the third strategy) can also be combined to obtain a more robust accuracy allocation scheme.

[0154] For example, the first and third strategies can be used in combination, which can yield more robust allocation results in actual tests. For example, the business constraint parameters include the first proportional parameter k and the boundary threshold. The method for determining the first proportional parameter k can refer to the relevant description in the aforementioned third strategy, and the method for determining the boundary threshold can refer to the relevant description in the aforementioned first strategy, which will not be repeated here.

[0155] For example, based on business constraint parameters and the quantization sensitivity index of multiple linear operators, the multiple linear operators can be divided into a first set and a second set. This can include: sorting the quantization sensitivity index of the multiple linear operators in descending order to obtain a first sorting result; assigning the linear operators in the first sorting result that simultaneously satisfy the condition of being in the top k% and having a quantization sensitivity index greater than the boundary threshold to the first set, and assigning the other linear operators to the second set.

[0156] In other words, in this embodiment, linear operators assigned to the first set must simultaneously meet two conditions: "quantization sensitivity index exceeds the threshold" and "ranked in the top k%". These linear operators employ a high-precision quantization scheme. Other linear operators that do not simultaneously meet these two conditions are assigned to the second set and employ a low-precision quantization scheme.

[0157] For example, a combination of the first and second strategies can be used. For instance, in a network structure where the network model includes a hybrid expert model layer, and typically also includes a non-hybrid expert model layer, the first strategy can be used for binary classification of linear operators in these non-hybrid expert model layers.

[0158] For example, business constraint parameters include a boundary threshold and a first threshold corresponding to each hybrid expert model layer.

[0159] The method for determining the boundary threshold can refer to the relevant description of the first strategy mentioned above, and the method for determining the first threshold can refer to the relevant description of the second strategy mentioned above, which will not be repeated here.

[0160] For example, based on business constraint parameters and the quantization sensitivity indices of multiple linear operators, dividing multiple linear operators into a first set and a second set may include: sorting the quantization sensitivity indices of multiple linear operators in descending order to obtain a first sorting result; assigning linear operators in the first sorting result whose quantization sensitivity indices are greater than a threshold to the first set; for each hybrid expert model layer: sorting the normalized sensitivity indices of the multiple expert networks included in the hybrid expert model layer in descending order to obtain a second sorting result; assigning at least one linear operator to the first set based on the second sorting result, wherein at least one linear operator is a linear operator included in the expert network whose normalized sensitivity indices are greater than the first threshold corresponding to the hybrid expert model layer; assigning the other linear operators in the multiple linear operators not assigned to the first set to the second set.

[0161] In other words, in this embodiment, the linear operators assigned to the first set include those in the expert network whose normalized sensitivity index is higher than the corresponding first threshold in each hybrid expert model layer. Additionally, they include linear operators whose absolute quantization sensitivity index is greater than the boundary threshold in the global quantization sensitivity index ranking. These operators employ a high-precision quantization scheme, while the other linear operators are assigned to the second set and employ a low-precision quantization scheme.

[0162] The data processing method provided in at least one embodiment of this disclosure is not limited to the specific structure of the network model, but can be applied to fields involving quantitative reasoning of network models, such as natural language processing, text processing, computer vision, speech processing, and multimodal information fusion processing.

[0163] For example, the data processing method provided in at least one embodiment of this disclosure can be adapted to various large language models in the field of natural language processing and the quantitative inference process of traditional text neural networks. It is widely used in specific tasks such as text generation, multilingual machine translation, large models for code generation and code completion, intelligent question answering and dialogue systems, long text summarization and information extraction, sentiment analysis and text sentiment discrimination, keyword extraction, text correction and polishing, intent recognition, named entity recognition, document reading comprehension, logical reasoning and mathematical problem solving, copywriting, knowledge base question answering, and text classification. It supports lightweight deployment of models for high-concurrency online text services in the cloud and can also achieve low-power local inference of large language models on the device side, reducing the overhead of computing power, video memory, and storage resources while ensuring the accuracy of model inference.

[0164] For example, the data processing method provided in at least one embodiment of this disclosure can be adapted to the entire process of structured and unstructured text processing in the field of text processing, and is suitable for various text parsing, processing and optimization tasks. Examples include applications such as optical character recognition (OCR), document text parsing, content extraction, key information filtering, and document classification and archiving, and it is also suitable for refined text processing scenarios such as long text splitting, text polishing, sentence optimization, and privacy text desensitization.

[0165] For example, the data processing method provided in at least one embodiment of this disclosure can be adapted to the quantitative inference of relevant data from various visual networks in the field of computer vision, such as convolutional neural networks and Transformer visual models, covering intelligent processing scenarios across all dimensions of images and videos. For instance, it can be applied to classic visual tasks such as image classification, object detection, image segmentation, image super-resolution reconstruction, image denoising, image enhancement, and feature extraction; it can also support scenarios such as video object tracking, video frame interpolation, video quality restoration, video behavior analysis, and real-time video monitoring.

[0166] For example, the data processing method provided in at least one embodiment of this disclosure can be adapted to the quantitative inference of various network models related to speech signal processing and intelligent speech parsing in the field of speech processing, covering intelligent applications throughout the entire speech process. Application scenarios may include real-time speech recognition, offline speech-to-text conversion, multilingual speech translation, voice wake-up, voiceprint recognition, identity verification, and other speech perception tasks. Simultaneously, it can support speech generation and optimization tasks such as speech synthesis, timbre conversion, speech noise reduction, speech enhancement, audio sentiment analysis, and speech segmentation optimization.

[0167] For example, the data processing method provided in at least one embodiment of this disclosure can be adapted to the quantitative reasoning of fusion reasoning models for multiple types of information such as text, images, voice, and video in the field of multimodal information fusion processing, supporting cross-modal intelligent processing of various scenarios. For example, it can be applied to basic fusion tasks such as image-text matching, image-text generation, image caption generation, video content understanding, cross-modal retrieval, and multimodal content classification, and can also support advanced application scenarios such as intelligent question-answering multimodal interaction, image-text content review, video content summary generation, and cross-modal content conversion.

[0168] Of course, this disclosure is not limited to this. For other application scenarios or fields, as long as quantitative inference of mixed precision of network models is required, the data processing method described in at least one embodiment of this disclosure can be applied, and will not be elaborated here.

[0169] Figure 2 A flowchart illustrating a data processing method provided in an embodiment of this disclosure.

[0170] Figure 2 A complete processing flowchart of a data processing method provided in an embodiment of this disclosure is shown below. Figure 2 This describes the entire processing procedure of the data processing method provided in at least one embodiment of the present disclosure.

[0171] First, input the network model to be quantized and the calibration dataset.

[0172] In Phase 1, statistical data collection is performed. Specifically, the network model is forward-inferenced using a calibration dataset. During forward inference, the input activation matrix X of each linear operator is collected in real time, and the mean squared activation value (MSA) of the linear operator is determined based on the input activation matrix X. The specific process can be found in the descriptions of steps S10 and S30 above, and will not be repeated here.

[0173] Furthermore, the weight matrix of each linear operator is read from the storage component. This weight matrix is ​​pre-stored in the storage component of the processing device for the inference network model, such as in memory, when input into the network model. Based on the weight matrix W of the linear operator, the weight quantization error WQE of the linear operator is determined. The specific process can be referred to the descriptions of steps S20 and S40 above, and will not be repeated here.

[0174] Next, in Stage 2, the quantization sensitivity index is calculated. Specifically, the quantization sensitivity index of each linear operator is determined based on the product of its mean squared activation value and the weighted quantization error. For example, the product of the mean squared activation value and the weighted quantization error can be directly used as the quantization sensitivity index. Alternatively, the logarithm of the product of the mean squared activation value and the weighted quantization error can be used as the quantization sensitivity index.

[0175] Next, in stage 3, mixed-precision decision-making is performed. Specifically, business constraint parameters are first determined. Taking the system as an example, which provides a first strategy, a second strategy, a third strategy, and a combined strategy, the business constraint parameters may include strategy indications and preset judgment thresholds used in the corresponding strategies.

[0176] For example, the first ranking result of the quantization sensitivity index in descending order can be provided to the user. Based on this first ranking result, the user inputs a strategy selection and a preset judgment threshold used in the corresponding strategy. Alternatively, the first ranking result of the quantization sensitivity index in descending order can be provided to the user. Based on this first ranking result, the user inputs a strategy selection, and then the preset judgment threshold used in the selected strategy is automatically determined. Furthermore, in addition to receiving user input, the characteristics of the network model can be analyzed to automatically select a strategy and a preset judgment threshold used in the strategy. For details, please refer to the relevant descriptions in the foregoing embodiments, which will not be repeated here.

[0177] For example, for the first strategy, the global statistics of percentile X and quantization sensitivity index of linear operators can be determined, thereby determining the boundary threshold; then, combined with the first ranking result, linear operators whose quantization sensitivity index is greater than the boundary threshold corresponding to percentile X are assigned to the first set, and other operators are assigned to the second set.

[0178] For example, for the second strategy, the normalized sensitivity index of the expert network can be determined first, and then a first threshold can be determined based on the percentile Y and the normalized sensitivity index of the expert network. After that, the linear operators of the expert network with a normalized sensitivity index greater than the first threshold are assigned to the first set, and the other linear operators are assigned to the second set.

[0179] For example, for the third strategy, we can obtain the first proportional coefficient k, select the k% linear operator with the highest quantization sensitivity index to the first set, and divide the other linear operators into the second set.

[0180] For example, for combined strategies, such as a combination of the first strategy and the second strategy, or a combination of the first strategy and the third strategy, mixed precision decision-making can be performed with reference to the description of the foregoing related embodiments, which will not be repeated here.

[0181] Next, a precision allocation scheme is generated. In this scheme, linear operators in the first set are quantized using a high-precision quantization scheme, while linear operators in the second set are quantized using a low-precision quantization scheme. For example, linear operators in the first set may use W8A8 or BF16 for inference, while linear operators in the second set may use W4A8 for inference.

[0182] Next, quantization inference is performed. For example, in some embodiments, quantization inference can be performed based on a precision allocation scheme to output a quantized network model.

[0183] For example, in other embodiments, such as Figure 2 As shown, we can proceed to the quantitative reasoning process shown in stage 4.

[0184] Specifically, such as Figure 2 As shown, the process first performs quantization inference based on the precision allocation scheme to verify whether the generated scheme meets the precision requirements. If it does, the quantized network model is output; otherwise, new business constraint parameters are determined. For example, a new strategy can be chosen, such as determining new percentiles to obtain a new boundary threshold or first threshold, or determining a new first proportional coefficient k. Then, a new precision allocation scheme is generated based on the new business constraint parameters. If the quantization inference result of the new scheme meets the precision requirements, the quantized network model is output. Otherwise, the above process is repeated until a quantized network model that meets the precision requirements is obtained after multiple attempts.

[0185] Taking the aforementioned 45-layer network model including the MOE structure as an example, the above data processing method was practically verified. The results are shown in Table 1 below: Table 1

[0186] The data in the table above are the accuracy determination parameters of the quantized network model obtained by using the first strategy and the percentile X=99 scheme.

[0187] The accuracy benchmark is based on all operators using BF16 for inference. SmoothQuant W4A8 is a quantization scheme based on SmoothQuant smoothing preprocessing, while W4A8 without SmoothQuant is a smoothing preprocessing scheme without SmoothQuant.

[0188] As shown in Table 1, the technical solution disclosed herein can accurately identify that the quantization sensitivity error of the last three layers in the 45-layer network is significantly higher than that of other layers. The last three hybrid expert network layers are artifact source layers. The linear operators in these three layers are assigned to the first set, and high-precision quantization inference is performed. The artifact rate reaches 0%, which is also completely consistent with the artifact source verified by experiments.

[0189] The leniency accuracy of the disclosed technical solution is only 0.8% lower than the original precision solution of BF16. Compared with the quantization solution without SmoothQuant, the leniency accuracy improves from 79.5% to 90.6% (+11.1%), indicating that the hybrid precision allocation scheme guided by the quantization sensitivity index provided in this disclosure eliminates artifacts while retaining the precision advantage of SmoothQuant quantization.

[0190] Furthermore, the rigorous accuracy of this disclosure even exceeds the accuracy benchmark, indicating that the technical solution provided by this disclosure not only eliminates artifacts but also improves the format matching of certain answers.

[0191] The dynamic range of the quantitative sensitivity index disclosed herein can reach 8.8 × 10⁻⁶. 7 This is mainly due to the large dynamic range of the mean square activation value (MSA). This dynamic range naturally provides the ability to distinguish between high-sensitivity and low-sensitivity layers without the need for additional normalization. The MSA is the dominant factor in the large dynamic range of the quantization sensitivity index, which increases exponentially with depth (due to the cumulative effect of residual flow). The MSA determines the cross-layer differences in the quantization sensitivity index, while the weighted quantization error provides a moderate degree of inter-layer distinction.

[0192] In this disclosure, the quantization sensitivity index not only considers the weight error, but also the error amplification effect brought about by the input activation matrix, which more accurately reflects the sensitivity of the linear operator to the quantization error, thereby obtaining a more reasonable precision allocation scheme. The quantization precision is close to lossless. As shown in Table 1, it is only 0.8% lower than the loose accuracy of the precision benchmark, which is far better than other quantization schemes.

[0193] The quantization sensitivity index disclosed herein has extremely low computational cost. It does not require running a quantization process and can be obtained with only one forward inference, which significantly reduces the cost of quantization tuning, reduces hardware resource consumption, and improves model deployment efficiency.

[0194] The calculation of the quantization sensitivity index in this disclosure relies only on the input activation matrix and weight matrix of the linear operator, has no restrictions on the network architecture, is applicable to all network model structures that include linear operators, and has a wide range of applications.

[0195] The quantization sensitivity index in this disclosure has a wide dynamic range, naturally distinguishing between high-sensitivity and low-sensitivity objects. Based on the quantization sensitivity index, the source of artifacts can be accurately located, completely eliminating artifacts while retaining the accuracy advantage of low-precision quantization, and improving the accuracy of network model inference results.

[0196] Furthermore, this disclosure can perform quantization sensitivity assessment on an expert network basis, identifying high-sensitivity experts (more sensitive to quantization errors) and low-sensitivity experts (less sensitive to quantization errors) within each layer of the hybrid expert model, thereby adopting different quantization strategies for different expert networks and providing fine-grained quantization guidance for the quantization of hybrid expert models.

[0197] At least one embodiment of this disclosure also provides a data processing apparatus. Figure 3 This is a schematic block diagram of a data processing apparatus provided for at least one embodiment of the present disclosure.

[0198] like Figure 3 As shown, the data processing device 200 may include an acquisition module 201, a reading module 202, a first determination module 203, a second determination module 204, a third determination module 205, a fourth determination module 206, and a division module 207. It should be noted that... Figure 3 The components and structure of the data processing apparatus 200 shown are merely exemplary and not limiting. The data processing apparatus 200 may also have other components and structures as needed.

[0199] For example, these modules can be implemented through hardware (e.g., circuit) modules, software modules, or any combination of both. The following embodiments are the same and will not be described again. For example, these units can be implemented through a central processing unit (CPU), a data processor (GPU), a tensor processor (TPU), a field-programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, as well as corresponding computer instructions.

[0200] For example, the data processing device can be a processor that needs to perform inference operations. This could be a general-purpose processor such as a central processing unit (CPU), multi-core processor, graphics processing unit (GPU), general-purpose graphics processing unit (GPGPU), or digital signal processor (DSP), or a dedicated processor such as an AI processor, tensor processor (TPU), or field-programmable gate array (FPGA). Alternatively, the data processing device could be a component specifically designed for optimized computation within GPUs or AI accelerators, such as a deep learning accelerator. Of course, this disclosure is not limited to these examples; the data processing device can be any device that needs to execute data processing methods to perform corresponding inference operations.

[0201] The acquisition module 201 is configured to perform forward inference on the network model and acquire the input activation matrix of each linear operator among the multiple linear operators included in the network model.

[0202] The read module 202 is configured to read the weight matrix of each linear operator from the storage unit.

[0203] The first determining module 203 determines the mean square activation value of the linear operator based on the input activation matrix of each linear operator, wherein the mean square activation value is used to measure the average energy of the input activation matrix of the linear operator.

[0204] The second determining module 204 is configured to determine the weight quantization error of the linear operator based on the weight matrix of each linear operator, wherein the weight quantization error is used to measure the inherent error introduced when quantizing the weight matrix of the linear operator.

[0205] The third determining module 205 is configured to determine the product of the mean square activation value and the weighted quantization error of each linear operator as the quantization sensitivity index of the linear operator.

[0206] The fourth determining module 206 is configured to determine business constraint parameters, wherein the business constraint parameters include relevant configuration parameters used when performing binary grouping of multiple linear operators.

[0207] The partitioning module 207 is configured to divide multiple linear operators into a first set and a second set based on business constraint parameters and the quantization sensitivity index of multiple linear operators, so as to obtain a precision allocation scheme for multiple linear operators. The minimum value of the quantization sensitivity index of the linear operators in the first set is greater than the maximum value of the quantization sensitivity index of the linear operators in the second set. In the precision allocation scheme, the linear operators in the first set use a higher quantization precision than the linear operators in the second set for quantization inference operations of the network model.

[0208] For example, the acquisition module 201, reading module 202, first determination module 203, second determination module 204, third determination module 205, fourth determination module 206, and division module 207 may include code and programs stored in memory; the processor can execute the code and programs to implement some or all of the functions of the acquisition module 201, reading module 202, first determination module 203, second determination module 204, third determination module 205, fourth determination module 206, and division module 207 as described above. For example, the acquisition module 201, reading module 202, first determination module 203, second determination module 204, third determination module 205, fourth determination module 206, and division module 207 may be dedicated hardware devices used to implement some or all of the functions of the acquisition module 201, reading module 202, first determination module 203, second determination module 204, third determination module 205, fourth determination module 206, and division module 207 as described above. For example, the acquisition module 201, the reading module 202, the first determination module 203, the second determination module 204, the third determination module 205, the fourth determination module 206, and the division module 207 can be a single circuit board or a combination of multiple circuit boards, used to implement the functions described above. In the embodiments of this application, the single circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processor; and (3) processor-executable firmware stored in the memory.

[0209] It should be noted that the acquisition module 201 can be used to implement Figure 1 As shown in step S10, the reading module 202 can be used to implement... Figure 1 In step S20 shown, the first determining module 203 can be used to implement... Figure 1 The second determining module 204 can be used to implement step S30 as shown. Figure 1 The third determining module 205, as shown in step S40, can be used to implement... Figure 1 The fourth determining module 206, as shown in step S50, can be used to implement... Figure 1 As shown in step S60, the partitioning module 207 can be used to implement... Figure 1The steps are shown in step S70. For a detailed description of the functions of the acquisition module 201, please refer to the description of step S10 in the embodiments of the data processing method described above; for a detailed description of the functions of the reading module 202, please refer to the description of step S20 in the embodiments of the data processing method described above; for a detailed description of the functions of the first determining module 203, please refer to the description of step S30 in the embodiments of the data processing method described above; for a detailed description of the functions of the second determining module 204, please refer to the description of step S40 in the embodiments of the data processing method described above; for a detailed description of the functions of the third determining module 205, please refer to the description of step S50 in the embodiments of the data processing method described above; for a detailed description of the functions of the fourth determining module 206, please refer to the description of step S60 in the embodiments of the data processing method described above; and for a detailed description of the functions of the dividing module 207, please refer to the description of step S70 in the embodiments of the data processing method described above. Repeated descriptions will not be repeated.

[0210] The data processing apparatus provided in at least one embodiment of this disclosure uses the product of the mean square activation value and the weight quantization error of each linear operator as the quantization sensitivity index of the linear operator. The quantization sensitivity index not only considers the influence of the weights, but also the influence that the input activation tensor may bring. The obtained precision allocation scheme is more in line with the quantization precision requirements. Moreover, the calculation method of the quantization sensitivity index is easy to implement, and it can be obtained with only one forward inference without running a complex quantization process. This realizes the paradigm shift of evaluation before quantization, significantly reducing the computational cost and resource consumption. For example, compared with the aforementioned Alpha search method, the computational cost can be reduced by 3-4 orders of magnitude, and the GPU time can be reduced by more than 99%.

[0211] The quantization sensitivity index in this disclosure has a large dynamic range, which can naturally distinguish between high-sensitivity and low-sensitivity objects without manual setting. In the quantization accuracy scheme, linear operators with high quantization sensitivity index, that is, operators that produce larger errors before and after quantization, use higher precision for quantization inference operations, while linear operators with low quantization sensitivity index, that is, operators that produce smaller errors before and after quantization, use lower precision for quantization inference operations. The overall quantization accuracy is close to lossless, and the use of low precision can be maximized to save video memory and bandwidth. While maintaining low-bit quantization performance, the accuracy of model quantization inference is improved, achieving the optimal balance between accuracy and efficiency.

[0212] The data processing device 200 can achieve similar technical effects to the aforementioned data processing method, and will not be described in detail here.

[0213] It should be noted that, in the embodiments of this disclosure, the data processing device 200 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited; it can be constructed from analog devices, digital chips, or other suitable methods according to circuit principles.

[0214] Figure 4 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. For example... Figure 4 As shown, the electronic device 300 is, for example, suitable for implementing the data processing method provided in the embodiments of this disclosure. It should be noted that... Figure 4 The components of the electronic device 300 shown are merely exemplary and not limiting. The electronic device 300 may have other components as needed for the actual application.

[0215] like Figure 4 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in memory to achieve various functions.

[0216] For example, when the computer-readable instruction processing device 301 executes, it can perform one or more steps of the data processing method according to any of the above embodiments. It should be noted that a detailed description of the processing procedure of the data processing method can be found in the relevant descriptions in the embodiments of the above data processing method.

[0217] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory, etc., whereby computer-readable instructions can be loaded from storage device 308 into RAM 303 to execute. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. Various applications and various data, such as various data used and / or generated by applications, may also be stored in the computer-readable storage medium.

[0218] For example, the processing device 301, ROM 302, and RAM 303 are interconnected via bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0219] Typically, the following devices can be connected to I / O interface 305: input devices 306 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 307 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 308 including, for example, magnetic tapes, hard disks, flash memory, etc.; and communication devices 309. Communication device 309 allows electronic device 300 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 4 An electronic device 300 with various devices is shown, but it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 300 may alternatively implement or possess more or fewer devices. For example, a processing device 301 can control other components in the electronic device 300 to perform desired functions. The processing device 301 may be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), a tensor processor (TPU), or a graphics processing unit (GPU). The central processing unit (CPU) may be an x86, ARM, RISC-V architecture, etc. The GPU may be directly integrated into the SOC, directly integrated onto the motherboard, or built into the northbridge chip of the motherboard.

[0220] For example, the electronic device 300 may further include a peripheral interface (not shown in the figure). This peripheral interface can be various types of interfaces, such as a USB interface, a Lightning interface, etc. The communication device 309 can communicate wirelessly with a network and other devices, such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN). Wireless communication can use any of a variety of communication standards, protocols, and technologies, including but not limited to Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0221] For example, the electronic device can be any device such as a mobile phone, tablet computer, laptop computer, e-book reader, game console, television, digital photo frame, or navigator, or any combination of electronic devices and hardware. The embodiments disclosed herein do not limit this.

[0222] For example, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For instance, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 309, or installed from storage device 308, or installed from ROM 302. When the computer program is executed by processing device 301, the data processing methods defined in the embodiments of this disclosure are performed.

[0223] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In embodiments of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In embodiments of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0224] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0225] Figure 5 This is a schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. For example, such as Figure 5As shown, storage medium 400 can be a non-transitory computer-readable storage medium on which one or more computer-readable instructions 401 can be stored non-transitory. For example, when the computer-readable instructions 401 are executed by a processor, the data processing method described above can be performed.

[0226] For example, the storage medium 400 can be used in the aforementioned electronic device, such as the storage medium 400 may include the memory in the electronic device.

[0227] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0228] For example, the description of storage medium 400 can be found in the description of memory in the embodiments of the electronic device, and the repeated parts will not be repeated.

[0229] Figure 6 This is a schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure. The electronic device provided in this disclosure can be applied to an Internet system.

[0230] use Figure 6 The computer system provided herein can implement the functions of the data processing apparatus and / or electronic device involved in this disclosure. Such computer systems may include personal computers, laptops, tablets, mobile phones, personal digital assistants, smart glasses, smartwatches, smart rings, smart helmets, and any smart portable or wearable device. A specific system in this embodiment uses a functional block diagram to explain a hardware platform including a user interface. Such a computer device can be a general-purpose computer device or a purpose-specific computer device. Both types of computer devices can be used to implement the data processing apparatus and / or electronic device of this embodiment. The computer system may include any components necessary to implement the information required for the data processing described herein. For example, a computer system can be implemented by a computer device through its hardware, software programs, firmware, and combinations thereof. For convenience, Figure 6 Although only one computer device is shown in the figure, the computer functions related to the information required for data processing described in this embodiment can be implemented in a distributed manner by a group of similar platforms, thus distributing the processing load of the computer system.

[0231] like Figure 6As shown, the computer system may include a communication port 250 connected to a network for data communication. For example, the computer system can send and receive information and data through the communication port 250, enabling the computer system to exchange data via wireless or wired communication with other electronic devices. The computer system may also include a processor group 220 (i.e., the processing unit described above) for executing program instructions. The processor group 220 may consist of at least one processor (e.g., a CPU or GPU). The computer system may include an internal communication bus 210. The computer system may include different forms of program storage units and data storage units (i.e., the memories or storage media described above), such as a hard disk 270, read-only memory (ROM) 230, and random access memory (RAM) 240, capable of storing various data files used for computer processing and / or communication, as well as possible program instructions executed by the processor group 220. The computer system may also include an input / output component 260 for implementing input / output data flow between the computer system and other components (e.g., user interface 280, etc.).

[0232] Typically, the following devices can be connected to the input / output component 260: input devices such as touch screens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices such as displays (e.g., LCD, OLED displays, etc.), speakers, vibrators, etc.; storage devices including, for example, magnetic tapes, hard disks, etc.; and communication interfaces.

[0233] Although Figure 6 A computer system with various devices is shown, but it should be understood that the computer system is not required to have all the devices shown, and alternatively, the computer system may have more or fewer devices.

[0234] Those skilled in the art will understand that the contents disclosed herein can be varied and modified in many ways. For example, the various devices or components described above can be implemented in hardware, or in software, firmware, or a combination of some or all of the three.

[0235] Furthermore, while this disclosure makes various references to certain elements of systems according to embodiments of this disclosure, any number of different elements may be used and operated on clients and / or servers. Elements are merely illustrative, and different aspects of the system and method may use different elements.

[0236] This disclosure uses flowcharts to illustrate the steps of a method according to embodiments of this disclosure. It should be understood that the preceding or following steps are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes.

[0237] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. Optionally, all or part of the steps in the above embodiments can also be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiments can be implemented in hardware or as a software functional module. This disclosure is not limited to any particular combination of hardware and software.

[0238] Unless otherwise defined, all terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It should also be understood that terms such as those defined in a common dictionary should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning, unless expressly defined herein.

[0239] The foregoing description is intended to illustrate the present disclosure and should not be construed as limiting it. While several exemplary embodiments of the present disclosure have been described, those skilled in the art will readily understand that many modifications may be made to the exemplary embodiments without departing from the novel teachings and advantages of the present disclosure. Therefore, all such modifications are intended to be included within the scope of the present disclosure as defined by the claims. It should be understood that the foregoing description is intended to illustrate the present disclosure and should not be construed as limiting it to the specific embodiments disclosed, and modifications to the disclosed embodiments and other embodiments are intended to be included within the scope of the appended claims. The present disclosure is defined by the claims and their equivalents.

Claims

1. A data processing method, characterized in that, include: Forward inference is performed on the network model to collect the input activation matrix of each linear operator among the multiple linear operators included in the network model; Read the weight matrix of each linear operator from the storage unit; Based on the input activation matrix of each linear operator, the mean square activation value of the linear operator is determined, wherein the mean square activation value is used to measure the average energy of the input activation matrix of the linear operator; Based on the weight matrix of each linear operator, the weight quantization error of the linear operator is determined, wherein the weight quantization error is used to measure the inherent error introduced when quantizing the weight matrix of the linear operator; The quantization sensitivity index of the linear operator is determined based on the product of the mean square activation value and the weighted quantization error of each linear operator. Determine business constraint parameters, wherein the business constraint parameters include relevant configuration parameters used when performing binary grouping on the plurality of linear operators; Based on the business constraint parameters and the quantization sensitivity index of the multiple linear operators, the multiple linear operators are divided into a first set and a second set to obtain a precision allocation scheme for the multiple linear operators. The minimum value of the quantization sensitivity index of the linear operators in the first set is greater than the maximum value of the quantization sensitivity index of the linear operators in the second set. In the precision allocation scheme, the linear operators in the first set use a higher quantization precision than the linear operators in the second set to perform quantization inference operations on the network model.

2. The data processing method according to claim 1, characterized in that, In the quantization inference operation, one or more rows in the input activation matrix of each linear operator share the same scaling factor, and one or more columns in the weight matrix of each linear operator share the same scaling factor.

3. The data processing method according to claim 2, characterized in that, Based on the input activation matrix of each linear operator, the mean square activation value of the linear operator is determined, including: The square value is calculated for each element of the input activation matrix of the linear operator; The mean square value of the squares of the plurality of elements is determined as the mean square activation value of the linear operator.

4. The data processing method according to claim 2, characterized in that, Based on the weight matrix of each linear operator, the weight quantization error of the linear operator is determined, including: Determine the maximum value among the absolute values ​​of each output channel in the weight matrix of the linear operator; The square of the maximum absolute value of each output channel is determined as the intermediate result for each output channel; The mean value of the intermediate results corresponding to the multiple output channels included in the weight matrix of the linear operator is determined as the weight quantization error of the linear operator.

5. The data processing method according to claim 1, characterized in that, Determine the business constraint parameters, including: Based on the quantization sensitivity indices of the multiple linear operators, a global distribution statistic of the quantization sensitivity indices of the multiple linear operators is determined, wherein the global distribution statistic includes the boundary thresholds corresponding to the preset percentiles of the multiple linear operators; Determine the percentile X, where the percentile X represents X% of the linear operators among the plurality of linear operators being assigned to the second set, and the remaining 1-X% of the linear operators being assigned to the first set, and X is a positive integer less than 100; Based on the percentile X and the global distribution statistics, the boundary threshold corresponding to the percentile X is determined as the business constraint parameter.

6. The data processing method according to claim 5, characterized in that, Based on the business constraint parameters and the quantization sensitivity index of the multiple linear operators, the multiple linear operators are divided into a first set and a second set, including: The quantization sensitivity indices of the multiple linear operators are sorted in descending order to obtain the first sorting result; Based on the first sorting result, the linear operators whose quantization sensitivity index is greater than the threshold corresponding to the percentile X are assigned to the first set, and the other linear operators are assigned to the second set.

7. The data processing method according to claim 1, characterized in that, The network model comprises multiple hybrid expert model layers, and each hybrid expert model layer includes multiple expert networks. Determine the business constraint parameters, including: For each hybrid expert model layer: Determine the quantization sensitivity index of the hybrid expert model layer, wherein the quantization sensitivity index of the hybrid expert model layer is the average value of the quantization sensitivity indices of all linear operators included in the hybrid expert model layer; Based on the quantized sensitivity index corresponding to the hybrid expert model layer, the normalized sensitivity index corresponding to each of the multiple expert networks included in the hybrid expert model layer is determined. Based on the normalized sensitivity indices corresponding to the multiple expert networks, a first threshold is determined for the hybrid expert model layer. The business constraint parameters include the first thresholds corresponding to the multiple hybrid expert model layers, respectively.

8. The data processing method according to claim 7, characterized in that, Based on the quantized sensitivity index corresponding to the hybrid expert model layer, the normalized sensitivity index corresponding to each of the multiple expert networks included in the hybrid expert model layer is determined, including: For each expert network, the average value of the quantization sensitivity index of the multiple linear operators included in the expert network is determined as the quantization sensitivity index corresponding to the expert network. The ratio of the quantized sensitivity index corresponding to the expert network to the quantized sensitivity index corresponding to the hybrid expert model layer is used as the normalized sensitivity index corresponding to the expert network.

9. The data processing method according to claim 7, characterized in that, Based on the quantized sensitivity index corresponding to the hybrid expert model layer, the normalized sensitivity index corresponding to each of the multiple expert networks included in the hybrid expert model layer is determined, including: For expert networks where the number of received tokens is less than a preset threshold: A preset label is added to the expert network, wherein, in the process of dividing the multiple linear operators into the first set and the second set, the expert network with the preset label does not participate in the sorting, and the linear operators in the expert network with the preset label are directly divided into the first set.

10. The data processing method according to claim 7, characterized in that, Based on the normalized sensitivity indices corresponding to the multiple expert networks, the first threshold corresponding to the hybrid expert model layer is determined, including: Determine the percentile Y, wherein the percentile Y indicates that Y% of the expert networks are first-class expert networks, and the other expert networks are second-class expert networks, the linear operators in the first-class expert networks are assigned to the second set, the linear operators in the second-class expert networks are assigned to the first set, and Y is a positive integer less than 100; Based on the percentile Y and the normalized sensitivity index corresponding to the multiple expert networks, the boundary threshold corresponding to the percentile Y is determined as the first threshold corresponding to the hybrid expert model layer.

11. The data processing method according to claim 7, characterized in that, Based on the business constraint parameters and the quantization sensitivity index of the multiple linear operators, the multiple linear operators are divided into a first set and a second set, including: For each hybrid expert model layer: The normalized sensitivity indices of the multiple expert networks included in the hybrid expert model layer are sorted in descending order to obtain a second sorting result; Based on the second sorting result, the linear operators included in the expert networks whose normalized sensitivity index is greater than the first threshold corresponding to the hybrid expert model layer are assigned to the first set, and the linear operators included in the other expert networks are assigned to the second set.

12. The data processing method according to claim 1, characterized in that, Determining the business constraint parameters includes obtaining a first proportional parameter k, where k is a positive integer; Based on the business constraint parameters and the quantization sensitivity index of the multiple linear operators, the multiple linear operators are divided into a first set and a second set, including: The quantization sensitivity indices of the multiple linear operators are sorted in descending order to obtain the first sorting result; The top k% of linear operators in the first sorting result are assigned to the first set, and the other linear operators are assigned to the second set.

13. The data processing method according to claim 1, characterized in that, The business constraint parameters include a first proportional parameter k and a boundary threshold. Based on the business constraint parameters and the quantization sensitivity index of the multiple linear operators, the multiple linear operators are divided into a first set and a second set, including: The quantization sensitivity indices of the multiple linear operators are sorted in descending order to obtain the first sorting result; Linear operators in the first sorting result that simultaneously satisfy the conditions of being in the top k% and having a quantization sensitivity index greater than the threshold are assigned to the first set, while other linear operators are assigned to the second set.

14. The data processing method according to claim 1, characterized in that, The network model comprises multiple hybrid expert model layers, and each hybrid expert model layer includes multiple expert networks. The business constraint parameters include a boundary threshold and a first threshold corresponding to each hybrid expert model layer. Based on the business constraint parameters and the quantization sensitivity index of the multiple linear operators, the multiple linear operators are divided into a first set and a second set, including: The quantization sensitivity indices of the multiple linear operators are sorted in descending order to obtain the first sorting result; Linear operators whose quantization sensitivity index is greater than the boundary threshold in the first sorting result are assigned to the first set; For each hybrid expert model layer: The normalized sensitivity indices of the multiple expert networks included in the hybrid expert model layer are sorted in descending order to obtain a second sorting result; According to the second sorting result, at least one linear operator is assigned to the first set, wherein the at least one linear operator is a linear operator included in the expert network whose normalized sensitivity index is greater than the first threshold corresponding to the hybrid expert model layer among the plurality of expert networks; The linear operators other than those already assigned to the first set are assigned to the second set.

15. The data processing method according to any one of claims 5-14, characterized in that, The business constraint parameters also include a received strategy indication, wherein the strategy indication is used to indicate the strategy used when determining the preset judgment threshold when performing the binary grouping of the plurality of linear operators.

16. The data processing method according to claim 1, characterized in that, Also includes: According to the precision allocation scheme, the quantization inference operation is performed on the network model; In response to the result of the quantization inference operation not meeting the accuracy requirements, new business constraint parameters are determined to obtain a new accuracy allocation scheme based on the new business constraint parameters, and the quantization inference operation is performed again based on the new accuracy allocation scheme. In response to the result of the quantization inference operation meeting the accuracy requirements, a quantization network model is output, wherein the model structure of the quantization network model is the same as that of the network model, and the linear operators in the quantization network model adopt the quantization accuracy specified by the accuracy allocation scheme.

17. The data processing method according to claim 1, characterized in that, The quantization sensitivity index of the linear operator is determined based on the product of the mean square activation value and the weighted quantization error of each linear operator, including: The product of the mean square activation value and the weighted quantization error of each linear operator is used as the quantization sensitivity index of the linear operator; or The logarithm of the product of the mean square activation value and the weighted quantization error of each linear operator is used as the quantization sensitivity index of the linear operator.

18. The data processing method according to any one of claims 1-14 and 16-17, characterized in that, The data processing method described herein is applied in the fields of natural language processing, text processing, computer vision, speech processing, and multimodal information fusion processing.

19. A data processing apparatus, characterized in that, include: The acquisition module is configured to perform forward inference on the network model and acquire the input activation matrix of each linear operator among the multiple linear operators included in the network model; The read module is configured to read the weight matrix of each linear operator from the storage unit; The first determining module is configured to determine the mean square activation value of the linear operator based on the input activation matrix of each linear operator, wherein the mean square activation value is used to measure the average energy of the input activation matrix of the linear operator; The second determining module is configured to determine the weight quantization error of the linear operator based on the weight matrix of each linear operator, wherein the weight quantization error is used to measure the inherent error introduced when quantizing the weight matrix of the linear operator; The third determining module is configured to determine the product of the mean square activation value and the weighted quantization error of each linear operator as the quantization sensitivity index of the linear operator. The fourth determining module is configured to determine business constraint parameters, wherein the business constraint parameters include relevant configuration parameters used when performing binary grouping on the plurality of linear operators; The partitioning module is configured to divide the multiple linear operators into a first set and a second set based on the business constraint parameters and the quantization sensitivity index of the multiple linear operators, so as to obtain a precision allocation scheme for the multiple linear operators. The minimum value of the quantization sensitivity index of the linear operators in the first set is greater than the maximum value of the quantization sensitivity index of the linear operators in the second set. In the precision allocation scheme, the linear operators in the first set use a higher quantization precision than the linear operators in the second set to perform quantization inference operations on the network model.

20. An electronic device, characterized in that, include: Memory stores computer-executable instructions non-transiently; The processor is configured to run computer-executable instructions. The computer-executable instructions are executed by the processor to implement the data processing method according to any one of claims 1-18.

21. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they implement the data processing method according to any one of claims 1-18.

Images