A data processing method, electronic device, storage medium, and program product

Abstract

This invention discloses a data processing method, electronic device, storage medium, and program product, relating to the field of artificial intelligence technology. By performing complexity analysis on the input data to be processed in a pre-trained model, a complexity score is obtained. Based on a comparison between the complexity score and a complexity threshold, the target quantization precision parameter of the target sub-module of the pre-trained model is determined. Thus, during the processing of the input data using the pre-trained model, the input data of the target sub-module is quantized according to the target quantization precision parameter, yielding the quantization result of the target sub-module. This invention realizes a scheme for adaptively adjusting the quantization precision of the sub-modules of the pre-trained model based on the complexity of the input data. Compared to the fixed quantization configuration in traditional static quantization strategies, this method can dynamically match differentiated quantization precision for input samples of different complexities, ensuring inference accuracy while avoiding unnecessary waste of computational resources.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a data processing method, electronic device, storage medium, and program product. Background Technology

[0002] With the development of artificial intelligence technology, large-scale neural network models have emerged in fields such as natural language processing, computer vision, and speech recognition, leading to a dramatic increase in the hardware resources required for model inference. To reduce model deployment costs and improve inference efficiency, quantization techniques have been introduced into neural network inference. For example, high-precision floating-point model parameters can be converted into low-precision fixed-point values, reducing model size and accelerating inference computation. However, traditional model quantization schemes typically involve quantizing model parameters and then deploying them on hardware devices. This static quantization strategy struggles to dynamically adapt to the varying computational precision requirements of different input samples when faced with complex and variable input data in actual inference processes, resulting in decreased model inference accuracy or wasted computational resources.

[0003] Therefore, how to solve the problem that static quantization strategies are difficult to dynamically adapt to the different requirements of different input samples for computational accuracy, resulting in a decrease in model inference accuracy or a waste of computational resources, is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] This invention provides a data processing method, electronic device, storage medium, and program product to at least solve the problem in related technologies that static quantization strategies are difficult to dynamically adapt to the different requirements of different input samples for computational accuracy, resulting in a decrease in model inference accuracy or a waste of computational resources.

[0005] This invention provides a data processing method, comprising: Obtain the input data to be processed from the pre-trained model, perform complexity analysis on the input data to be processed, and obtain a complexity score; Based on the comparison results of the complexity score and the complexity threshold, the target quantization accuracy parameter of the target submodule of the pre-trained model is determined; During the processing of the input data to be processed using the pre-trained model, the input data of the target sub-module is quantized according to the target quantization accuracy parameter to obtain the quantization calculation result of the target sub-module.

[0006] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described data processing methods.

[0007] This invention provides a method for adaptively adjusting the quantization precision of submodules in a pre-trained model based on the complexity of the input data. This is achieved by performing complexity analysis on the input data to be processed in the pre-trained model, obtaining a complexity score, and comparing the complexity score with a complexity threshold to determine the target quantization precision parameter for the target submodule. This allows for the quantization calculation of the input data of the target submodule based on the complexity of the input data. Compared to the fixed quantization configuration in traditional static quantization strategies, this invention can dynamically match differentiated quantization precision for input samples with different complexities, thereby ensuring inference accuracy while avoiding unnecessary waste of computational resources. Attached Figure Description

[0008] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A flowchart of a data processing method provided in an embodiment of the present invention; Figure 2 A system architecture diagram of a data processing method provided in an embodiment of the present invention; Figure 3 A flowchart of complexity analysis provided for an embodiment of the present invention; Figure 4 A flowchart for quantitative decision-making and scheduling is provided as an embodiment of the present invention; Figure 5 A flowchart of hardware scheduling is provided for an embodiment of the present invention. Detailed Implementation

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0011] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0012] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0013] Here, we will first explain some key terms used in the embodiments of the present invention.

[0014] Pre-trained model: In this embodiment of the invention, the pre-trained model refers to a model that has completed regular training and requires no further training or only fine-tuning. This model is used to deploy on an inference computing device to perform inference computing tasks. This embodiment of the invention addresses the problem of how to quantize the model parameters of the pre-trained model to improve inference computing efficiency and save inference computing resources.

[0015] Quantization bit width refers to the number of binary bits used to represent the quantized value. Common quantization bit widths include 8-bit integer (INT8), 4-bit integer (INT4), 3-bit integer (INT3), 2-bit integer (INT2), 8-bit floating-point (FP8), 16-bit half-precision floating-point (FP16), and 16-bit brain-float (BF16). The size of the quantization bit width directly determines the precision and dynamic range of the numerical representation: a higher bit width results in a finer numerical representation and smaller quantization error, but also increases the model size and computational overhead; a lower bit width results in higher model compression and computational efficiency, but may introduce larger quantization errors, affecting the model's inference accuracy.

[0016] Quantization configuration: Represented in "WxA" format, where W is the weight quantization bit width and A is the activation value quantization bit width. Common quantization configurations include symmetric quantization (both weights and activations are quantized) and mixed precision configuration (different bit widths are used for different layers). Common symmetric quantization configurations include 8-bit integer quantization for both weights and activations (W8A8), 4-bit weights and 8-bit activations (W4A8), 4-bit weights and 4-bit activations (W4A4), 8-bit weights and 16-bit activations (W8A16), and 3-bit weights and 8-bit activations (W3A8), etc.

[0017] In this embodiment of the invention, the quantization precision parameter is mainly used to control the quantization bit width of the model parameters, and it can be represented by quantization configuration.

[0018] Traditional methods such as Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT) typically choose a uniform quantization bit width (e.g., all INT8) for the entire model or a fixed module. This ignores the dynamic impact of different input samples on the numerical sensitivity of various parts of the model. For complex inputs, fixed low-precision quantization may lead to significant information loss and accuracy degradation; for simple inputs, fixed high-precision quantization results in unnecessary waste of computational resources.

[0019] To address the problem that static quantization strategies struggle to dynamically adapt to the varying computational accuracy requirements of different input samples, leading to decreased model inference accuracy or wasted computational resources, this invention provides a data processing scheme. This scheme analyzes the complexity of the input data to be processed in a pre-trained model, obtaining a complexity score. Based on a comparison between the complexity score and a complexity threshold, the target quantization accuracy parameter for the target sub-module of the pre-trained model is determined. During the processing of the input data using the pre-trained model, the input data of the target sub-module is quantized according to the target quantization accuracy parameter, yielding the quantization result of the target sub-module. This achieves a scheme that adaptively adjusts the quantization accuracy of sub-modules in the pre-trained model based on the complexity of the input data. Compared to the fixed quantization configuration in traditional static quantization strategies, this invention can dynamically match differentiated quantization accuracy for input samples of varying complexity, thereby ensuring inference accuracy while avoiding unnecessary waste of computational resources.

[0020] The embodiments of the present invention provide a data processing method. The method is described in detail below in conjunction with the execution flow of the data processing method.

[0021] Figure 1 A flowchart of a data processing method provided in an embodiment of the present invention; Figure 2 This is a system architecture diagram of a data processing method provided in an embodiment of the present invention.

[0022] like Figure 1 As shown, the data processing method provided in this embodiment of the invention may include: S101: obtaining the input data to be processed of the pre-trained model, performing complexity analysis on the input data to be processed, and obtaining a complexity score.

[0023] S102: Based on the comparison results of the complexity score and the complexity threshold, determine the target quantization accuracy parameter of the target submodule of the pre-trained model.

[0024] S103: During the process of using the pre-trained model to process the input data to be processed, the input data of the target sub-module is quantized according to the target quantization accuracy parameter to obtain the quantization calculation result of the target sub-module.

[0025] To overcome the shortcomings of traditional static quantization strategies, this invention provides a dynamic quantization strategy. By sensing the complexity of the input data, the quantization accuracy parameters of sub-modules in the pre-trained model are dynamically adjusted, thereby ensuring inference accuracy while avoiding unnecessary waste of computational resources.

[0026] In this embodiment of the invention, in order to distinguish between the input data of the pre-trained model and the input data of the intermediate layer or sub-module of the pre-trained model, the input data of the pre-trained model is defined as "input data to be processed".

[0027] like Figure 2 As shown, the data processing method provided in this embodiment of the invention can be implemented collaboratively based on all or part of the components in the real-time input complexity analysis and perception module, the adaptive mixed precision quantization decision and scheduling module, the hardware collaborative quantization calculation acceleration device, and the inference calibration and accuracy assurance module.

[0028] In specific implementation, for S101, before performing forward propagation computation on the input data to be processed using the pre-trained model, the complexity of the input data to be processed is analyzed to obtain a complexity score.

[0029] S101 can be implemented based on a real-time analysis and perception module for input complexity. For example... Figure 2 As shown, the real-time input complexity analysis and perception module runs at the very front of the inference pipeline, performing complexity analysis on each input sample (text or image input data to be processed) or batch.

[0030] For S102, the quantization precision parameters of the target submodule of the pre-trained model are selected based on the complexity score and a preset complexity threshold. First, the complexity score is compared with the preset complexity threshold; the result of this comparison serves as the decision-making basis for selecting the quantization precision parameters. This decision-making process can be implemented based on the correspondence between complexity and quantization precision parameters obtained through pre-testing.

[0031] The quantization precision parameter can be either the quantization bit width or the quantization configuration.

[0032] In some optional embodiments of the present invention, the target submodule may refer to dividing the pre-trained model into multiple quantizable basic units, such as different network layers or different modules. These basic units are denoted as submodules, and each submodule is regarded as a target submodule. In other optional embodiments of the present invention, the quantization accuracy parameters of some model parameters (e.g., normalization layers) in the pre-trained model can be obtained and configured through offline analysis. The remaining submodules, such as activation function layers, which are not suitable for static quantization processing, are regarded as target submodules, thereby improving the efficiency of dynamic quantization decision-making.

[0033] S102 can be executed based on the adaptive mixed-precision quantization decision and scheduling module. For example... Figure 2 As shown, the adaptive mixed-precision quantization decision and scheduling module receives the complexity score output by the input complexity real-time analysis and perception module, and decides the quantization precision parameters of each target sub-module.

[0034] For S103, based on the target quantization precision parameters of the target submodule, the corresponding hardware computing module for quantization precision is invoked to perform forward propagation calculations on the target submodule. The output result is the quantization calculation result of the current target submodule. Thus, after completing one forward propagation calculation of the pre-trained model, the inference result for the input data to be processed is obtained.

[0035] S103 can be executed by a hardware-coordinated quantization computation acceleration device under the control of the adaptive mixed-precision quantization decision and scheduling module, and finally obtain the inference result as the model output.

[0036] The data processing method provided in this embodiment of the invention may further include a supervision mechanism for the reasoning process, i.e., such as... Figure 2 The inference calibration and accuracy assurance module shown can analyze intermediate calculation results and final inference results during the inference process, detect anomalies, and trigger operations such as adjusting configuration parameters.

[0037] In this embodiment of the invention, the complexity analysis step and decision-making step (i.e., S101 and S102) of the current input data to be processed can be executed in parallel with the quantization calculation step (i.e., S103) of the previous input data to be processed, thereby hiding the decision-making overhead.

[0038] The data processing method provided in this invention analyzes the complexity of the input data to be processed in a pre-trained model to obtain a complexity score. Based on the comparison between the complexity score and the complexity threshold, the target quantization precision parameter of the target sub-module of the pre-trained model is determined. Then, during the processing of the input data using the pre-trained model, the input data of the target sub-module is quantized according to the target quantization precision parameter to obtain the quantization result of the target sub-module. This achieves a scheme for adaptively adjusting the quantization precision of the sub-modules of the pre-trained model based on the complexity of the input data. Compared with the fixed quantization configuration in traditional static quantization strategies, this invention can dynamically match differentiated quantization precision for input samples with different complexities, thereby ensuring inference accuracy while avoiding unnecessary waste of computational resources.

[0039] Figure 3 A flowchart of a complexity analysis provided for an embodiment of the present invention.

[0040] Based on the above embodiments, in this embodiment of the invention, performing complexity analysis on the input data to be processed in S101 to obtain a complexity score may include: calculating multiple types of complexity statistical scores for the input data to be processed; obtaining a complexity score based on the multiple types of complexity statistical scores; wherein, the types of complexity statistical scores include at least two of the following: data content complexity score, data structure complexity score, and model feedback score.

[0041] To obtain a more comprehensive complexity score, this embodiment of the invention analyzes the complexity of the input data to be processed from multiple dimensions, namely at least two of the following: data content dimension, data structure dimension, and model feedback dimension. Specifically, the data content dimension provides the assessment of "how complex is the input data itself?" (objective content complexity). The data structure dimension provides the assessment of "how much computational resources are required to process this input data?" (objective load complexity). The model feedback dimension provides the assessment of "how the pre-trained model perceives the complexity of this input data?" (subjective response complexity).

[0042] In a better implementation, the complexity statistics scores of the above three dimensions of the input data to be processed can be calculated separately, and the complexity score can be obtained by combining them. By analyzing the multi-dimensional complexity scores of the input data to be processed, the most appropriate complexity score of the input data to be processed on the pre-trained model can be obtained.

[0043] When multiple complexity statistics are selected, the process of calculating different complexity scores can be executed in parallel. For example, it can be started simultaneously on different computing units (such as central processing unit cores and accelerator cores), and each dimension shares a copy of the input data to be processed, without the need for repeated copying.

[0044] In some alternative embodiments of the present invention, performing complexity analysis on the input data to be processed in S101 to obtain a complexity score may also include: calculating one of the following as the complexity score: the data content complexity score, the data structure complexity score, and the model feedback score of the input data to be processed.

[0045] In this embodiment of the invention, the calculation steps of the data content complexity score may include: calculating multiple types of content complexity scores of the input data to be processed; obtaining the data content complexity score based on the multiple types of content complexity scores; wherein the type of the content complexity score belongs to at least one of information content feature score, numerical distribution range feature score, and outlier data feature score.

[0046] Among them, the feature types of information content feature score (also known as uncertainty score) can include at least one of information entropy, variance, etc., to describe how much information, randomness, or unpredictability is contained in the input data to be processed, that is, how complex or chaotic the input data to be processed is.

[0047] The feature types of numerical distribution range feature scores can include at least one of the following: numerical dynamic range, standard deviation, range, etc., used to describe the span and dispersion of the values ​​of the input data to be processed, that is, how wide the distribution of the data values ​​of the input data to be processed is.

[0048] Outlier data feature scoring can include at least one of the following: outlier proportion, kurtosis, sparsity, etc., used to describe the proportion of extreme values ​​that deviate from the norm in the data, that is, whether there are "anomalies" in the data that require special handling.

[0049] Taking the scoring of information content features using normalized entropy, the scoring of numerical distribution range features using dynamic range, and the scoring of outlier data features using outlier ratio as examples.

[0050] The steps for calculating the normalized information entropy of the input data to be processed may include: calculating the information entropy of the probability distribution of the input sequence (such as text or image patches) and normalizing it by dividing by the maximum possible entropy (log N), to obtain: , ; in, Represents the normalized information entropy. Represents discrete random variables. Represents information entropy. Represents the maximum possible entropy. Indicates the total number of possible values. The unit that determines information entropy. Represents random variables Take the first The probability value of each value.

[0051] The judgment logic is as follows: High entropy (close to 1) → uniform distribution of tokens / pixels, no obvious dominant pattern → semantic ambiguity or a lot of noise → high complexity; Low entropy (close to 0) → existence of strong dominant patterns (such as repeated words, smooth regions) → simple structure, easy to predict → low complexity.

[0052] Example: A scientific paper containing a large number of technical terms that are evenly distributed has an Hnorm of 0.85, which is rated as high complexity; while "The weather is nice today" has an Hnorm of 0.2, which is rated as low complexity.

[0053] The steps for calculating the numerical dynamic range of the input data to be processed may include: calculating the difference (or standard deviation) between the maximum and minimum values ​​of the input embedding vector (such as token embedding) or image pixel gradient, and then normalizing it to [0,1]. Judgment logic: Large dynamic range → significant feature differences and rich details → model requires high-precision representation → high complexity; Small dynamic range → feature convergence and flatness → tolerable low-precision quantization → low complexity.

[0054] The calculation steps for the outlier ratio of the input data to be processed may include: the proportion of elements in the input features that exceed the mean ± 2 standard deviations. Judgment logic: High outlier ratio → presence of abnormal values ​​or extreme semantics (such as rare words, sharp edges) → quantization is prone to distortion → high complexity; Low outlier ratio → concentrated data distribution → strong robustness → low complexity. Summary of statistical criteria: The higher any statistical indicator, the more incompressible and unpredictable the input, i.e., the higher the complexity score should be.

[0055] In this embodiment of the invention, the calculation steps of the data structure complexity score may include: calculating multiple types of structure complexity scores of the input data to be processed; obtaining the data structure complexity score based on the multiple types of structure complexity scores; wherein the type of the structure complexity score belongs to at least one of data scale feature score, spatial structure complexity score, and temporal structure complexity score.

[0056] Among them, the feature types of data scale feature scoring can include at least one of sequence length, image resolution, point cloud density, etc., which are used to describe how long the text is or how big the image is, and determine how many basic operations (such as multiplication and addition operations) the model needs to perform. These features can be obtained directly from the original data or shape of the input data to be processed.

[0057] The feature type for spatial structure complexity scoring can include at least one of spatial structure complexity, image texture richness, semantic ambiguity, and signal-to-noise ratio. It describes the difficulty for the model to process the input data. For example, for images of the same resolution, denser textures make processing more difficult. This feature needs to be obtained after analyzing the input data.

[0058] The feature types of temporal structure complexity scores can include at least one of inter-frame difference, temporal entropy, dynamic range, etc., which are used to describe the strength of the regularity of data points changing over time and the richness of dynamic patterns in time series data (such as video frame sequences, audio signals, text, sensor data). This determines whether the model needs more iterations, longer context windows or more attention resources when processing the data to be processed.

[0059] For example, the feature types for data structure complexity scoring include the sequence length, resolution, and spatial structure complexity of the data to be processed.

[0060] The logic for determining the sequence length or resolution of the input data to be processed can include: long sequence / high resolution → large computational load, deep context dependency → higher computational precision is required to ensure that the accumulated error is not amplified → high complexity; short sequence / low resolution → low information density, short computational path → low precision can be safely used → low complexity.

[0061] The processing of the sequence length or resolution of the input data to be processed can include: mapping the original length L to the [0,1] interval, for example, through an activation (sigmoid) function: ; in, Indicates the structural complexity score. This is an empirical threshold (e.g., 512 lexical units). Control the slope.

[0062] The steps for calculating the spatial structure complexity or temporal complexity of the input data to be processed may include: for text, the syntactic tree depth, the number of dependencies, or the n-gram diversity index can be used; for images, the standard deviation of local gradient magnitude, edge density, or the proportion of high-frequency energy in the frequency domain can be calculated.

[0063] The logic for judging spatial structural complexity or temporal complexity is as follows: High structural complexity → deep semantic hierarchy / rich texture → model attention mechanism is easily affected by quantization perturbation → high complexity; Low structural complexity → single sentence / smooth background → insensitive to quantization → low complexity.

[0064] In summary, the structural dimension criteria are as follows: the denser, deeper, and more frequent the structure, the higher the complexity score.

[0065] In this embodiment of the invention, the calculation steps for the model feedback score may include: obtaining a simplified model corresponding to the pre-trained model; performing forward propagation calculation on the input data to be processed using the simplified model, and extracting the intermediate calculation results of the simplified model; determining the model feedback score based on the data distribution characteristics of the intermediate calculation results of the simplified model; wherein, the model parameters of the simplified model come from the pre-trained model, and the number of parameters of the simplified model is smaller than that of the pre-trained model.

[0066] In other words, a simplified version of the pre-trained model (also known as a probe network or simplified network) can be designed to analyze the complexity of the input data to be processed and obtain the model feedback score for the input data. This simplified model can be a simplified copy of the first few layers of the pre-trained model. By using the simplified model to perform forward propagation calculations on the input data to be processed, the sparsity, distribution variance, and other characteristics of intermediate calculation results of the intermediate activation values ​​can be observed as indirect indicators of complexity.

[0067] In some optional embodiments of the present invention, the steps of obtaining a simplified model may include: (1) architecture pruning: copying the original model parameters of some sub-modules of the pre-trained model, for example, copying the first 1-2 layers of the pre-trained model (such as the vectorization (Embedding) layer and the first Transformer block), but performing channel pruning on the linear layers therein, reducing the hidden layer dimension to 1 / 4 or lower of the original model (e.g., from 4096 dimensions to 1024 dimensions).

[0068] (2) Functional imitation: The training objective of this simplified model is not to complete the final task, but to learn and imitate the intermediate activation patterns (such as the sparsity of activation values ​​and the variance of the distribution) of the first few layers of the original pre-trained model when encountering inputs of different complexities. This is a lightweight knowledge distillation process that enables it to perceive input features with very few parameters.

[0069] (3) The simplified model is trained offline, and the process is independent of the quantization process of the pre-trained model.

[0070] During the offline training phase of the simplified model, a quantized calibration dataset (such as a portion of the wikitext-2 dataset) or a subset sampled from the training set can be used as input. The training objective of the simplified model can be to minimize the mean squared error (MSE) between the simplified model output and these soft labels, using the statistical features of the intermediate activation values ​​of the first few layers of the pre-trained model (such as the mean and variance of each channel). The control parameters for the offline training phase of the simplified model can be: using a small batch size (e.g., 64) and a low learning rate (e.g., 0.0002), training for approximately 50 epochs to prevent overfitting and ensure stable convergence.

[0071] Thus, a simplified model after training was obtained. The simplified model was then used to perform forward propagation calculations on the input data to be processed, and the data distribution characteristics of the intermediate calculation results were observed to obtain the model feedback score.

[0072] The core advantage of this simplified model lies in its negligible impact on the overall system with extremely low overhead. In practical applications, due to the significant structural simplification, the number of parameters in the simplified model can be only 0.01% to 0.1% of the total number of parameters in the original large model. Furthermore, the additional inference latency added by the simplified model during forward propagation is typically less than 0.1 milliseconds. Moreover, because the simplified model occupies very few activation values, its memory overhead is also minimal. Therefore, in a complete flow of the data processing method provided in this embodiment of the invention (including complexity analysis of the input, decision-making, and quantization calculation), the computational overhead of the simplified model can account for less than 0.5% of the total inference time.

[0073] As can be seen, the simplified model approximates the first reaction of the pre-trained model to input complexity at a very low cost. It does not run the entire pre-trained model, but only runs a simplified version of the first few layers of the original pre-trained model, extracting the statistical features of the intermediate activations (such as variance, sparsity, distribution shape, etc.) as a proxy indicator of input complexity.

[0074] In summary, the complexity score is obtained based on multiple types of complexity statistical scores, which may include: normalizing the multiple types of complexity statistical scores respectively; and weighted summing of the normalized complexity statistical scores to obtain the complexity score.

[0075] Multiple complexity statistics scores are mapped to the [0,1] interval to eliminate the influence of dimensions, and finally a scalar complexity score CS (Complexity_Score∈[0,1]) is obtained.

[0076] In practice, the complexity statistical score is first normalized, and each original feature value is then normalized. Mapping to the [0,1] interval eliminates the influence of dimensions, resulting in a normalized complexity statistical score. : ; in, and It can be predetermined based on statistical empirical values ​​from a large number of samples or calibration datasets.

[0077] Assign a weight to each normalized complexity statistical score The initial comprehensive score is obtained by weighted summation. : .

[0078] The logical basis for weight allocation can include: task relevance: different tasks have different sensitivities to features. For example, in text summarization tasks, the weight of sequence length should be significantly higher than that of gradient magnitude; the opposite is true in image classification tasks.

[0079] Empirical data-driven: In the offline stage, regression analysis can be used to fit the correlation between each feature value and the inference time or attention entropy of the model at full precision (FP16) to determine the initial weights.

[0080] Interpretability principle: Generally, global and fundamental metrics such as sequence length / resolution and information entropy will receive higher weights (e.g., 0.3 each), while detailed metrics such as outlier ratio will have lower weights (e.g., 0.1).

[0081] Initial overall score It may exceed the range of [0,1], or require nonlinear calibration to better align with human perception of "complexity" (i.e., the existence of marginal effects): ; in, , , These are calibration parameters, which can be adjusted empirically. This is the compression / stretch factor, typically ranging from 0.5 to 2.0. The larger, The more sensitive the β is to small changes, the less sensitive it is to small changes. For amplitude scaling, typical values ​​are 0.5~1.0, controlling the output range, in conjunction with... use, For translation, the typical value is 0~0.5, which adjusts the output center position. The function can Mapped to (-1,1) and smoothed at both ends, the min / max operation ensures that the final output is within [0,1].

[0082] Taking text as the input data to be processed as an example, assuming only three features are used: information entropy (E), sequence length (L), and norm variance (V), with empirical weights of 0.4, 0.4, and 0.2 respectively, and the pre-trained model is a Mixture of Experts (MoE) model. The complexity score is obtained based on multi-class complexity statistical scoring, which can include the following process.

[0083] Input: A text segment, which is then divided into a sequence of words.

[0084] Feature extraction is performed on each dimension of the complexity statistical scoring: E=0.85 (high information entropy, rich content information); L=512 tokens (long text); V=0.60 (Word importance fluctuates greatly).

[0085] Normalize the features of each dimension (assuming they are based on empirical maximum values): ; ; .

[0086] Weighted fusion of normalized features: .

[0087] Calibration output (assuming simple linearity): CS=min(1.0,0.66)=0.66.

[0088] Final conclusion: Complexity score CS=0.66. This indicates a moderately complex input. The adaptive mixed-precision quantization decision and scheduling module will use this score to potentially select a relatively conservative W4A8 configuration for the MoE routing expert.

[0089] This invention provides a complete complexity analysis process, which can be described as follows: Figure 3 As shown, the real-time input complexity analysis and perception module receives the input data to be processed (which can be text, images, voice, etc.), and uses the statistical feature analysis unit, structural feature unit, and model feedback analysis unit to extract features from the input data to be processed, perform feature fusion and normalization processing, and calculate the data content complexity score, data structure complexity score, and model feedback score of the input data to be processed. By performing weight allocation, normalization processing, and multi-dimensional synthesis on the scores of each dimension, the complexity score is calculated and output to the adaptive mixed precision quantization decision and scheduling module.

[0090] In real-world business scenarios, input data often exhibits temporal locality and category locality: Temporal locality: Requests arriving within a short period (e.g., within seconds) may have highly similar input characteristics (e.g., text length, image resolution). Category locality: Inputs from the same task type (e.g., medical report analysis, financial contract review) tend to have a relatively stable complexity distribution.

[0091] The data processing method provided in this embodiment of the invention may further include: associating and storing the complexity score of the input data to be processed with the summary information of the input data to be processed, so that when the next input data to be processed is received, if the summary information matches, the complexity analysis step is skipped and the corresponding complexity score is directly called as the complexity score of the next input data to be processed.

[0092] In other words, in practical applications, it is not necessary to perform an independent complexity analysis on each input data to obtain a complexity score. Instead, the complexity score of the current input data can be obtained based on the complexity scores of other stored inputs. Furthermore, it is not necessary to perform an independent complexity analysis and quantization decision on each input data to obtain a quantization strategy for the current input data based on the quantization strategies of other stored inputs.

[0093] In practical implementation, a caching mechanism for complexity scores or quantification precision parameters can be established. Taking complexity scores as an example, a complexity score cache table can be created, with the key being the input summary (such as the hash value of text or the thumbnail fingerprint of an image). When a new input arrives, the cache is checked first; if a match is found, the cached complexity score is used directly. Simultaneously, an invalidation policy can be set for this cache, such as setting a time-to-live (TTL) period (e.g., 30 seconds) or employing a Least Recently Used (LRU) eviction mechanism to ensure timely updates to the complexity score cache table.

[0094] Specifically, this can be achieved by setting an adaptive frequency controller. This controller maintains a complexity score cache and calculates a summary (such as a fast fingerprint, like SimHash for text or Perceptual Hash for images) for each input data to be processed. When new input data arrives, the complexity score cache is queried first. If a match is found and the cache is not expired, the cached complexity score is used directly. If a match is missed or the cache expires, a simplified model is invoked for real-time analysis, and the results are stored in the cache. The adaptive frequency controller can also be used to dynamically adjust the cache expiration time and analysis probability based on the current system load, cache hit rate, and recent quantization rollback count. For example, when the system load exceeds a threshold, the cache expiration time is extended to 5 seconds; when the number of rollbacks increases, the expiration time is forcibly shortened.

[0095] Based on the above caching method, the complexity score of the input data to be processed can be associated with its summary information and stored in the cache. When the next input data to be processed is obtained, it is determined whether its summary information has been cached and is not expired. If it has been cached and is not expired, the complexity score is used as the complexity score of the next input data to be processed, and its complexity analysis is skipped. If it has not been cached or is expired, a simplified model is used to perform complexity analysis on the next input data to be processed, obtain its complexity score, and associate the obtained complexity score with the summary information of the next input data to be processed and store it in the cache. This allows complexity scores to be reused for similar inputs, further reducing the number of computations required for complexity analysis.

[0096] Therefore, it is not necessary to perform complexity analysis or quantitative decision-making on all input data to be processed, which significantly improves reasoning efficiency.

[0097] Furthermore, a classification mechanism for the input data to be processed can be established. Specifically, a lightweight input classifier (such as a logistic regression model or a small tree model) can be pre-trained. When the pre-trained model performs inference tasks, this input classifier is used to quickly classify the input data (such as "short text / long text", "high entropy / low entropy"). Each class is pre-bound with a quantization configuration template, which contains a set of quantization accuracy parameters. The overhead of this input classifier can be less than that of a simplified model, but its granularity is coarser.

[0098] In addition, when the system supports batch inference, the entire batch of inputs can be subjected to an aggregate complexity analysis, and the worst-case or average value can be used as the quantization strategy for the batch, without having to perform complexity analysis and quantization decisions for each input data to be processed separately.

[0099] Alternatively, a periodic sampling approach can be adopted, where a complexity analysis and quantization decision are performed every N input data to be processed, and the quantization strategy for other input data to be processed within the period is dynamically adjusted according to the resource usage of the system.

[0100] In this embodiment of the invention, a dynamic regulator can be designed to adjust the analysis frequency in real time based on factors such as the current system load, the stability of historical inputs, and the quantization backoff frequency. ; In other words, when the system load is high, the analysis frequency is reduced (p decreases), and cache reuse is prioritized. When rollbacks occur frequently (indicating cache inaccuracy), the analysis frequency is increased.

[0101] In this embodiment of the invention, the aforementioned complexity score caching strategy can be coordinated only with model feedback score calculation. That is, data content complexity and data structure complexity are scored for each input data to be processed, but the cached model feedback score can be used instead when designing the model feedback score. In this case, using a simplified model, subtle changes in the input data to be processed can be perceived with high precision. This is suitable for scenarios with frequent input changes and high accuracy requirements. The complexity score output by the simplified model can be cached for reuse in frequency control. Frequency control can significantly reduce overhead, making it suitable for high-throughput scenarios where the input is relatively stable and the load is high. Frequency control determines when to use the simplified model, avoiding excessive calls.

[0102] Alternatively, the complexity score of the input data to be processed can be completely replaced by the cached complexity score instead of the analysis.

[0103] The data processing method provided in this invention achieves dynamic quantization decision-making with minimal overhead. It integrates a lightweight proxy model with a caching mechanism, combines a simplified model with a frequency controller, and constitutes a complete low-overhead adaptive decision-making subsystem, making input-driven quantization possible. In practical applications, through the above mechanism, this invention further reduces the average overhead of complexity analysis to less than 0.1% of the original inference time while ensuring decision accuracy, achieving a balance between high performance and high accuracy. Furthermore, this invention can extend the complexity evaluation mechanism to multimodal inputs (such as combined text and speech inputs), enabling unified cross-modal complexity evaluation and quantization scheduling. In model-parallel or pipeline-parallel environments, each computing node can dynamically adjust its quantization strategy based on local input complexity and collaborate with other nodes to achieve optimal global efficiency.

[0104] It should be noted that the combination of the simplified model and the caching mechanism has a synergistic effect in this embodiment of the invention. On the one hand, the number of parameters in the simplified model is only 0.01% to 0.1% of the original large model, and the additional inference latency introduced is very small, as mentioned above, it can be as low as less than 0.1 milliseconds, thus enabling high-precision perception of input complexity with low overhead. On the other hand, the caching mechanism avoids repeated analysis of similar inputs by reusing the calculated complexity score. After the two are combined, the overall overhead of complexity analysis can be reduced to less than 0.1% of the original inference time, thereby achieving low-overhead reuse of complexity scores while maintaining perception accuracy.

[0105] Figure 4 This is a flowchart of a quantitative decision-making and scheduling method provided in an embodiment of the present invention.

[0106] like Figure 4 As shown, the adaptive mixed-precision quantization decision and scheduling module receives a complexity score and determines the quantization bit width for each layer or sub-module, which may include the following steps.

[0107] Step 1: Start execution. Input the complexity score CS of the task to be processed. The value of the complexity score CS is in the range of [0,1].

[0108] Step 2: Based on the input complexity score CS, query the sensitivity prior knowledge base to obtain the sensitivity prior information corresponding to each module.

[0109] Step 3: Determine the global tone for the complexity score CS to establish the global quantization strategy. Specifically, first, determine if CS < 0.3: if yes, the global tone is set to aggressive quantization; if no, further determine if CS ≥ 0.7. In the further determination, if no (i.e., 0.3 ≤ CS < 0.7), the global tone is set to balanced quantization; if yes, the global tone is set to conservative quantization.

[0110] Step 4: After determining the overall tone, we enter the module-level fine-tuning stage, and perform sensitivity judgment and precision configuration for three types of modules: routing expert, MLA (multi-head potential attention) module and shared expert. Specifically, it includes: (1) judging whether the sensitivity of the routing expert is greater than the corresponding threshold: if yes, the routing expert is configured as W8A8 dynamic quantization; if no, the routing expert is configured as W4A8 dynamic quantization.

[0111] (2) Determine whether the sensitivity of the MLA module is greater than the corresponding threshold: if yes, configure the MLA module as W8A8 static quantization; if no, configure the MLA module as W4A8 static quantization.

[0112] (3) Determine whether the sensitivity of the shared expert is greater than the corresponding threshold: if yes, configure the shared expert as W8A8 static quantization; if no, configure the shared expert as W4A8 static quantization.

[0113] Step 5: Combining the global tone determined in Step 3 with the precision configuration of each module determined in Step 4, generate the corresponding quantization configuration scheme. The quantization configuration scheme includes generating an aggressive quantization configuration scheme, a balanced quantization configuration scheme, and a conservative quantization configuration scheme, which correspond to the three global tones of aggressive quantization, balanced quantization, and conservative quantization.

[0114] Step six: Output the quantization configuration to the hardware acceleration device, which then performs subsequent inference operations based on the quantization configuration.

[0115] Step 7: Through the feedback loop, the global threshold or module sensitivity threshold is dynamically updated based on the confidence level of the inference result, and the updated result is fed back to the sensitivity prior knowledge base, thereby iteratively optimizing the subsequent query and judgment process and forming a closed-loop adjustment.

[0116] like Figure 4 As shown by the dashed line, the feedback link connects back from the output stage to the query sensitivity prior knowledge base stage to achieve dynamic adaptive updating of the threshold.

[0117] Based on the above embodiments, in this embodiment of the invention, determining the target quantization accuracy parameter of the target sub-module of the pre-trained model according to the complexity score and the complexity threshold in S102 may include: determining the complexity evaluation interval (also called the complexity interval) where the complexity score is located according to the relationship between the complexity score and the complexity threshold; and determining the target quantization accuracy parameter of the target sub-module in the complexity evaluation interval according to the pre-established sensitivity prior knowledge base.

[0118] To achieve quantitative decision-making, embodiments of the present invention pre-divide multiple complexity evaluation intervals using complexity thresholds, and preset corresponding quantization precision parameters for each complexity evaluation interval, thereby enabling rapid decision-making.

[0119] In practical implementation, this can be achieved by establishing a sensitivity prior knowledge base offline in advance. By analyzing a large amount of representative calibration data, the quantization error-accuracy loss sensitivity curve of the target sub-module of the pre-trained model can be established. The minimum acceptable quantization accuracy (e.g., INT4) and recommended accuracy (e.g., INT8) for each target sub-module can be determined.

[0120] It should be noted that the minimum acceptable quantization precision refers to the lowest quantization bit width that the target submodule can theoretically withstand, provided that the preset global precision loss tolerance (such as Δacc≤1%) is met.

[0121] The minimum quantization precision can be obtained based on the global precision loss tolerance and module sensitivity curves.

[0122] The global accuracy loss tolerance can be provided by the user or obtained based on the business scenario. The types of global accuracy loss tolerance can include: accuracy baseline, such as "the accuracy after quantization shall not be lower than 98% of the original model", which can be converted into a global accuracy loss tolerance (e.g., 2%).

[0123] Performance targets, such as "inference latency must be less than 50ms", translate into constraints on computational load and indirectly affect the choice of precision.

[0124] Hardware limitations, such as "can only run on INT8 computing units", directly limit the range of available precision (only W8A8 or lower can be selected).

[0125] Security levels, such as "at least W8A8 must be retained in financial scenarios," directly specify the lower limit of precision for certain modules.

[0126] For example, users can input "the accuracy of the model on MMLU after quantization shall not decrease by more than 1%" or "the AUC of financial approval tasks shall not decrease by more than 0.5%", expressing the user's bottom line for the quantization effect.

[0127] These user constraints are input into the optimization problem as additional limitations, further tightening the "acceptable" range. For example, if the user requires "all attention layers must be ≥ W8A8", then during the solution process, the candidate accuracy set for layer A (if it is an attention layer) is reduced from [W4A8, W8A8, W16A16] to [W8A8, W16A16]. If the user requires "total computation must be ≤ a certain value", the system may be forced to accept the W4A8 configuration for some modules, even if their individual errors are slightly higher, as long as the total error remains within the tolerable range.

[0128] The module sensitivity curves were obtained through offline analysis. For each target submodule, its error was tested at different quantization accuracies to obtain the relationship between quantization accuracy and output error. This allowed for the deduction of the minimum tolerance level for each module in order to keep the global accuracy loss within acceptable limits. In other words, users can provide global accuracy constraints, and offline analysis can be used to obtain the accuracy-error data for each target submodule.

[0129] In this embodiment of the invention, the steps of constructing the sensitivity prior knowledge base may include: determining multiple target sub-modules from a pre-trained model; acquiring a dataset related to the target task domain of the input data to be processed; performing forward propagation calculations based on the dataset using the original model parameters of the pre-trained model to obtain baseline accuracy parameters; while keeping the original model parameters of other target sub-modules in the pre-trained model, performing forward propagation calculations for the target sub-modules in the pre-trained model under different quantization accuracy parameter configurations to obtain the loss evaluation parameters of the target sub-modules under the quantization accuracy parameter configurations compared to the baseline accuracy parameters; determining the quantization accuracy parameters corresponding to the complexity evaluation intervals based on the correspondence between the quantization accuracy parameters of the target sub-modules and the loss evaluation parameters of the target sub-modules; and constructing the sensitivity prior knowledge base using the quantization accuracy parameters corresponding to the complexity evaluation intervals of multiple target sub-modules.

[0130] In practice, the pre-trained model is first divided into sub-modules. Specifically, the raw model parameters of the pre-trained model (e.g., FP16 format) can be loaded, and its computational graph structure can be parsed. The pre-trained model is divided into a series of independent quantifiable units, i.e., target sub-modules, which correspond to the natural structure of the model. For example: Transformer model: each encoder / decoder layer, embedding layer, and output layer; MoE model: each expert, shared expert, and router. Further finer-grained sub-modules include attention heads in each multi-head attention mechanism and specific matrix multiplications in each MLP. A unique ID is assigned to each target sub-module, and its position and parameter size in the computational graph are recorded.

[0131] Then, a moderately sized but representative dataset (e.g., 500-2000 samples) is collected from the target domain of the input data to be processed (e.g., financial text, medical reports). This dataset should cover different input scenarios from simple to complex. It is processed using the exact same preprocessing workflow as the online inference process.

[0132] Design and execute stratified sensitivity experiments. This is the most crucial step, the purpose of which is to quantify the sensitivity of each unit to quantization error. Specifically, this may include: (1) obtaining baseline performance: using the original model with FP16 accuracy to run on the full validation set (a larger, labeled dataset), recording its overall accuracy (e.g., precision) as well as the prediction results and confidence scores for each sample, as the baseline accuracy.

[0133] (2) Quantization simulation and evaluation for each target submodule: The "single-variable control" principle is adopted. For each quantizable unit to be evaluated, all other units are kept at FP16 precision. Only the weights and activation values ​​of the unit are simulated and quantized according to a set of predetermined quantization configurations (e.g., W8A8 (representing 8-bit weights and 8-bit activations) static, W4A8 (representing 4-bit weights and 8-bit activations) static, W8A8 dynamic, and W4A8 dynamic). Forward propagation is performed on a representative calibration dataset, and the change of the model's final output relative to the FP16 baseline output after quantization of this layer is recorded.

[0134] (3) The evaluation index may include at least one of the following: task accuracy loss: the percentage decrease in the overall accuracy of the model (e.g., accuracy); output distribution difference: using measures such as KL divergence and cosine similarity to compare the difference in the final output logits or feature vectors of the model before and after quantization; intermediate representation distortion: comparing the statistical characteristics (e.g., mean square error, signal-to-noise ratio) of the output activation value distribution of the layer before and after quantization; constructing a “quantization intensity-error” curve: for each unit, connecting the error indices calculated under different quantization configurations to form the sensitivity curve of the unit.

[0135] (4) Data extraction and knowledge base construction. Key parameters are extracted from the sensitivity experiment and stored in a structured manner.

[0136] The following quantization precision can be extracted as the quantization precision corresponding to each complexity evaluation interval:

[0137] Minimum available accuracy: Find the most aggressive quantization configuration (such as W4A8 static) that satisfies the accuracy loss < preset tolerance threshold (e.g., 1%). This is the safety bottom line for this cell.

[0138] Recommended Balanced Precision: A configuration that strikes a good balance between efficiency (low bit width) and precision loss (small). Typically, a configuration where the precision loss is half the tolerance threshold (e.g., 0.5%) can be selected.

[0139] Recommended bit width for high precision: When the input is complex, it is recommended to allocate a bit width (such as INT8 or FP16) to this cell.

[0140] The three types of quantization accuracy parameters corresponding to the three complexity evaluation intervals mentioned above may overlap. For some target sub-modules, the minimum usable accuracy may be equal to the recommended balanced accuracy (for example, if the error of W4A8 has reached the inflection point, the three may be the same value).

[0141] In practice, for each quantizable unit, error curves are established offline under different quantization accuracies. Based on a preset global accuracy loss tolerance, the minimum viable precision (MVP) of the unit is determined by solving a global optimization problem; that is, the most aggressive configuration that guarantees the global constraints.

[0142] Furthermore, by analyzing the "precision-bit width" marginal benefit curve of the unit itself, the inflection point of the marginal benefit decline is identified, and the configuration corresponding to the inflection point is defined as the recommended balanced precision (RBP).

[0143] Finally, a configuration with negligible error (e.g., less than 0.1%) is defined as the High-Precision Recommendation (HPR) bit width for this unit, used to handle highly complex inputs.

[0144] The three types of quantization accuracy parameters mentioned above together constitute the quantization strategy space of this unit: MVP is a rigid lower bound, HPR is an upper bound protection, and RBP is the default option for daily operation. The dynamic decision engine can choose between RBP and HPR based on the real-time input complexity score, but it will never be lower than MVP.

[0145] The sensitivity coefficient is calculated using the formula shown below, which quantifies the above curve characteristics into a scalar sensitivity coefficient. : ; scalar sensitivity coefficient The larger the value, the more sensitive the cell is to quantization. This formula represents the loss of INT4 compared to INT8, which can be replaced by the loss between other two quantization precision parameters in actual calculations.

[0146] The knowledge base is stored in a structured manner. A look-up table or JSON configuration file is created to store the following example information for each unit ID: {"unit_id":"layer_5_attention_output", / / The output of the attention mechanism in layer 5; "sensitivity_score": 0.85, / / Sensitivity score is 0.85; "min_safe_precision":"W8A8", / / Minimum safe precision; "recommended_precision":"W8A8", / / Recommended precision; "aggressive_precision":"W4A8", / / Suggested precision in aggressive mode; "quant_type_suggestion":"dynamic", / / It is recommended to use dynamic / static quantization; "calibration_params":{ / / Optional: Pre-calculated calibration parameters (such as scaling factor range); "weight_scale_range":[0.01,0.5], / / Suggested range of weighting factors; "activation_scale_range":[0.05,1.2]}; / / Suggested range of scaling factors for activation values; }

[0147] In other words, the sensitivity prior knowledge base records the suggested quantization precision and specific quantization precision parameters used for each target submodule in different complexity ranges.

[0148] (5) Knowledge base verification and iteration can be carried out in the following ways.

[0149] Cross-validation: Use another set of unseen calibration data to verify the effectiveness of the knowledge base's guidance.

[0150] Overall simulation: Based on the knowledge base, several preset global strategies (such as aggressive, balanced, and conservative) are simulated, and their accuracy and efficiency are tested on the entire validation set to ensure that the knowledge base can support effective global decision-making.

[0151] Therefore, the sensitivity prior knowledge base constructed in this embodiment of the invention can adapt to the current input data to be processed and the system resource occupancy status, and support the rapid and effective provision of target quantization accuracy parameters in the dynamic quantization decision-making of the pre-trained model.

[0152] Based on the constructed sensitivity prior knowledge base, a dynamic policy decision tree can be further designed to select the quantified policy from the sensitivity prior knowledge base.

[0153] The operations performed by a dynamic policy decision tree can include global complexity mapping and module-level fine-tuning.

[0154] (1) Global Complexity Mapping: Based on the complexity score CS of the input, determine the global quantization tone for this inference. For example: CS < 0.3: tends to adopt a more aggressive quantization globally (such as W4A8 dynamic). To further reduce the error, implement two-level quantization for the weights: first perform W8A8 per-channel quantization, and then perform W4A8 per-group quantization for each group (e.g., 32) of W8A8 weights. Activation still uses per-token dynamic quantization.

[0155] This process does not involve requantizing the already quantized INT8 weights. Instead, it uses the scale factor of INT8 per-channel quantization as guiding information to map the original FP16 weights to the INT4 space in a one-time, step-by-step manner. Its core objective is to achieve a high compression ratio in INT4 while preserving the weight distribution characteristics by leveraging the finer-grained scale information of the INT8 channels, thereby reducing errors.

[0156] 0.3≤CS<0.7: Use balanced quantization with mixed precision. Use W8A8 for sensitive modules (such as attention output layer and first layer MLP) and W4A8 for the rest.

[0157] CS≥0.7: Use conservative quantization globally (such as W8A8), and even keep FP16 for core modules.

[0158] (2) Module-level fine-tuning: Fine-tuning is performed under the global framework, combined with the module sensitivity knowledge base. For example... Figure 4 As shown, taking a hybrid expert model as an example: IF Complexity_Score > 0.7 THEN / / Judgment: When the complexity score of the task / input is higher than 0.7, enter the high complexity branch and adopt a more conservative (higher precision) quantization strategy overall; MLA.quant_scheme = W8A8 (static, per-channel / per-tensor) / / For the Multi-Head Latent Attention (MLA) module: weights and activations are both 8 bits, static quantization is used (scaling parameters are fixed in advance), and scaling is performed at the per-channel or per-tensor granularity; Shared_Expert.quant_scheme = W8A8 (dynamic, per-token) / / Shared Expert / / For Shared Expert: Same W8A8, but uses dynamic quantization (real-time scaling during inference) and per-token granularity to adapt to the numerical distribution of different tokens; Routed_Expert.quant_scheme = W8A8 / / Routing experts are also upgraded to INT8 / / For routing experts: even under high complexity, they are upgraded to INT8 (W8A8), and no longer pushed lower, in order to ensure performance; ELSE / / Otherwise: If the complexity score is not higher than 0.7, enter the low complexity branch, where more aggressive compression can be performed in exchange for speed and size; MLA.quant_scheme = W8A8 (static) / / For the MLA module: still maintain W8A8 static quantization, the accuracy remains unchanged (this layer is more sensitive, so it is not adjusted). Shared_Expert.quant_scheme = W8A8 (dynamic) / / For shared experts: maintain W8A8 dynamic quantization with unchanged precision; Routed_Expert.quant_scheme = W4A8 (Dynamic, weight INT8→INT4 two-level quantization) / / For routing experts: the weight is further compressed to 4 bits, the activation is kept at 8 bits, dynamic quantization is adopted, and the precision loss caused by INT4 is reduced by the two-level quantization method of INT8→INT4. END IF / / End of conditional statement.

[0159] In other words, for the quantization strategy of Multi-head Latent Attention (MLA), when the complexity score CS > 0.7, MLA uses W8A8 quantization configuration (static, per-channel / per-tensor) for the shared expert, while using W8A8 quantization configuration (dynamic, per-token) for the routed expert (i.e., the routed expert is also upgraded to INT8). Otherwise, MLA uses static W8A8 configuration, the shared expert uses dynamic W8A8 configuration, and the routed expert uses dynamic W4A8 configuration (two-stage quantization from INT8 to INT4 for weights).

[0160] The weights are quantized in two stages to further reduce errors: first, W8A8 per-channel quantization is performed, and then W4A8 per-group quantization is performed on each group (e.g., 32 weights) of W8A8 weights. Activation still uses per-token dynamic quantization.

[0161] Furthermore, when the target quantization precision parameter indicates that the weights of the target submodule are quantized using the target bit width, the above two-level quantization can be implemented as follows: First, the weights of the target submodule are quantized channel by channel according to the first bit width to obtain the scaling factor under the first bit width, where the first bit width is greater than the target bit width. Then, using the scaling factor under the first bit width as guiding information, the weights of the target submodule are quantized group by group according to the target bit width to obtain the quantized weights under the target bit width. Taking the first bit width as INT8 and the target bit width as INT4 as an example, W8A8 channel-by-channel quantization is first performed to obtain the scaling factor, and then W4A8 group-by-group quantization is performed on each group of weights using this scaling factor as guidance, thereby reducing the quantization error while reducing the weight bit width.

[0162] The data processing method provided in this embodiment of the invention may further include: adjusting a complexity threshold in response to a threshold adjustment signal; wherein the type of the threshold adjustment signal includes at least one of the following: the resource utilization status of the device, the status of the execution result of the pre-trained model, and the time period for executing the input data to be processed.

[0163] like Figure 2 As shown, a feedback loop can be set up inside the adaptive mixed-precision quantization decision and scheduling module. When the confidence of the inference results of multiple consecutive high-complexity inputs is lower than the threshold (i.e. the accuracy is not up to standard), the complexity threshold in the decision tree can be automatically fine-tuned to achieve fine-tuning for different scenarios and realize repeated threshold adjustment.

[0164] In practical implementation, in the initial stage of the system, a complexity threshold obtained from offline testing can be used, such as 0.3 or 0.7. This threshold is determined by analyzing the relationship between the complexity score CS and the model performance index on a representative dataset through offline calibration experiments.

[0165] After the pre-trained model is deployed, the complexity threshold can be dynamically adjusted during the execution of inference computation tasks. The adjustment can be based on at least one of the following: system load, accuracy of data processing results, and time period.

[0166] The complexity threshold adjustment step based on system load may include: comparing the system load parameters with a preset load threshold to determine the range in which the system load parameters fall, and determining the corresponding complexity threshold adjustment method. For example: If system load > high load threshold: Lower bound threshold = 0.3 - 0.05 = 0.25 # For a more aggressive approach, use W4A8 for more inputs; Upper threshold = 0.7 - 0.05 = 0.65; el if system load < low load threshold: Lower threshold = 0.3 + 0.05 = 0.35 # More conservative, use high precision for more inputs; The upper threshold is 0.7 + 0.05 = 0.75.

[0167] The complexity threshold adjustment step based on the accuracy of the data processing results may include: maintaining a sliding window, monitoring the quantization backoff frequency of the most recent N input data to be processed, comparing the quantization backoff frequency with a tolerance threshold, determining the interval in which the quantization backoff frequency falls, and determining the corresponding complexity threshold adjustment method. For example: If the backoff rate is greater than the tolerance threshold: Lower threshold += 0.02# Increase the threshold to enter conservative quantization earlier; Upper threshold += 0.02; el if backoff rate < tolerance threshold / 2 and system load is high: Lower threshold -= 0.02 # Lowering the threshold is more aggressive; Upper threshold -= 0.02.

[0168] The complexity threshold adjustment steps based on time periods can include: determining the complexity threshold for different time periods according to the workload of different time periods. For example: assuming that 08:00-20:00 is the core trading period, the complexity threshold can be a more conservative threshold [0.25, 0.65] (entering mixed precision earlier); 20:00-08:00 is a low-load period, the complexity threshold can be a more aggressive threshold [0.35, 0.75] (more often using W4A8); weekends are mainly for analysis tasks, and a more balanced threshold [0.30, 0.70] can be used.

[0169] The complexity threshold can be dynamically adjusted by combining multiple criteria, such as system load and the accuracy of data processing results. The complexity threshold can be updated using the following formula: ; ; Where T_low_base and T_high_base represent the baseline thresholds determined offline, α represents the load sensitivity coefficient (adjustable), and β represents the accuracy sensitivity coefficient (adjustable). When the load is high, (1 - load / rated) is negative, the threshold is lowered, and the decision is more aggressive. When the backoff rate is high, the threshold is raised, and the decision is more conservative.

[0170] It should be noted that the combination of the sensitivity prior knowledge base and online threshold adjustment enables the embodiments of the present invention to possess long-term adaptive capabilities. The sensitivity prior knowledge base establishes quantization accuracy parameters for each target sub-module in different complexity evaluation intervals through offline analysis, providing a priori basis for quantization decisions. Based on this, the system can also adjust the complexity threshold online during operation according to threshold adjustment signals such as the resource utilization status of the device, the execution result status of the pre-trained model, and the time period of the input data to be processed. Therefore, the quantization decision of the embodiments of the present invention is not a one-time offline configuration, but can be continuously adjusted according to changes in system load, backoff rate, and time period, enabling the system to maintain a balance between accuracy and performance during long-term operation.

[0171] Based on the embodiments of the present invention, in addition to the three complexity intervals (corresponding to three types of quantization precision parameters) described above, fewer or more complexity intervals can be set, with each complexity interval corresponding to a set of quantization configurations. In practical applications, it is recommended to select 3 to 5 complexity intervals, which is the optimal balance between complexity and performance.

[0172] Through the complexity analysis and quantization decision-making scheme provided in this invention, the system can continuously learn the mapping relationship between complexity and quantization strategy from inference data by combining an online learning mechanism, achieving long-term adaptive optimization. A lightweight complexity assessment and decision-making front-end is deployed on edge devices, while high-load quantization computation is performed in the cloud, realizing edge-cloud collaborative adaptive quantization inference. Furthermore, for dynamic model structures (such as hypernetworks and conditional computation models), the system can maintain the adaptability of the quantization strategy even when the structure of the pre-trained model changes. In addition, it can be combined with other model compression techniques such as pruning and knowledge distillation to form a multi-level adaptive compression system.

[0173] Based on the above embodiments, this invention further describes the hardware scheduling scheme for quantization computation.

[0174] Figure 5 A flowchart of hardware scheduling is provided for an embodiment of the present invention.

[0175] like Figure 5 As shown, the hardware acceleration device provided in this embodiment of the invention mainly consists of a quantization strategy configuration unit, a kernel selector, a weight unpacker, a variable precision fusion kernel, a prequantization model, a hardware-specific instruction set, a pipelined parallel controller, and an output activation value unit. The connection relationships and functions between the various parts are as follows.

[0176] The quantization strategy configuration unit is used to receive the quantization strategy configuration from the decision module and output the quantization strategy configuration to the kernel selector and the pre-quantization model respectively.

[0177] The kernel selector is connected to the quantization strategy configuration unit and is used to select the kernel with the corresponding precision according to the received quantization strategy configuration and schedule it to the corresponding computing unit. On the one hand, the kernel selector outputs the selected kernel to the weight unpacker, and on the other hand, it outputs the scheduling result to the hardware dedicated instruction set.

[0178] The weight unpacker is connected to the kernel selector and is used to unpack the weights as needed, load the corresponding scaling factors and zeros, and then output the processing results to the variable precision fusion kernel.

[0179] The variable precision fusion kernel is connected to the weight unpacker to perform operations such as weight unpacking, low-precision matrix multiplication, INT4 / INT8 dequantization, activation function operation, and output.

[0180] The pre-quantized model is connected to the quantization strategy configuration unit to store or characterize the model in block sparse and quantized format and provide the corresponding data to the hardware-specific instruction set.

[0181] The dedicated hardware instruction set is connected to the kernel selector and the pre-quantized model respectively. It includes functional components such as Tensor core, INT4 computing unit and mixed precision support, which are used to perform low-level mixed precision calculations according to the scheduled kernel and pre-quantized model, and output the calculation results to the pipeline parallel controller.

[0182] The pipelined parallel controller is connected to a dedicated hardware instruction set to prepare for the next task while performing the current computation, thereby realizing pipelined parallel processing of computation and scheduling, and outputting the processing results to the output activation value unit.

[0183] The output activation value unit is connected to the pipeline parallel controller and is used to output activation values, which are sent to the next layer for subsequent calculations or as the final output.

[0184] Through the above structure, the hardware acceleration device provided in this embodiment of the invention can dynamically select and schedule the kernel of corresponding precision according to the quantization strategy given by the decision module, combine the variable precision fusion kernel and the hardware dedicated instruction set to achieve efficient inference of mixed precision, and improve the overall computing throughput and hardware utilization by means of the pipelined parallel controller.

[0185] In this embodiment of the invention, step S103, which involves quantizing the input data of the target submodule according to the target quantization precision parameter to obtain the quantization calculation result of the target submodule, may include: selecting a corresponding target quantization calculation core from multiple quantization calculation cores pre-deployed in the device according to the target quantization precision parameter; and using the quantization calculation instruction set corresponding to the pre-deployed target quantization calculation core to call the target quantization calculation core to perform quantization calculation on the input data of the target submodule to obtain the quantization calculation result of the target submodule; wherein different quantization precisions correspond to different quantization calculation cores.

[0186] In this embodiment of the invention, the quantized model parameters can be packaged and stored for easy retrieval. For example... Figure 2 As shown, the hardware-coordinated quantization computation acceleration device packages and stores quantization weights using a storage format that combines block sparsity and quantization. For example, for W4A8 weights, multiple 4-bit weights are packaged into an INT8 / INT32 word, and the associated scaling factor and zero point are recorded. The storage format is optimized and arranged according to the memory access habits of the target hardware (computing unit) when loading the model, ensuring continuous alignment of memory accesses.

[0187] Simultaneously, a kernel fusion and dedicated instruction calling mechanism can be constructed to integrate operations such as quantization unpacking, matrix multiplication, and dequantization / activation functions into an instruction set (e.g., fused into a single computing core CUDA Kernel or corresponding Neural Processing Unit (NPU) instructions). By pre-developing and deploying a series of variable-precision kernels, this kernel can dynamically select the computation path (e.g., calling INT4 TensorCore or INT8 TensorCore) based on the precision descriptor passed in at runtime, without recompiling or loading different kernels.

[0188] Meanwhile, as in the above embodiments, the quantization calculation of the current input and the complexity analysis and strategy decision-making process of the next input are pipelined in parallel, hiding the decision-making overhead and realizing pipeline optimization.

[0189] The data processing method provided in this invention uses a deep hardware and software collaborative acceleration mechanism to design a variable precision fusion kernel and intelligent packaging format, so that the quantization strategy can be directly mapped to the optimal hardware execution path, maximizing the utilization of hardware performance.

[0190] It should be noted that the dynamic invocation of hardware quantization strategies in this embodiment of the invention is not merely an algorithmic selection of quantization precision, but rather an end-to-end processing approach that integrates software and hardware. In this approach, the quantization precision parameters determined by the adaptive mixed-precision quantization decision and scheduling module are further mapped to the specific execution method of the hardware-coordinated quantization computation acceleration device. This includes selecting the corresponding variable-precision fusion kernel from multiple quantization computation cores, packaging and unpacking quantization weights according to an intelligent packaging format, and pipelined parallel processing of the current input quantization computation with the complexity analysis and strategy decision for the next input. Thus, quantization decisions can be directly translated into hardware execution methods, forming a continuous processing flow from input complexity perception to hardware quantization computation, thereby making fuller use of hardware performance.

[0191] Based on the above embodiments, an accuracy guarantee mechanism can be added during the inference calculation process.

[0192] In some optional embodiments of the present invention, the input data to be processed or a portion of the input data to be processed can be data augmented before being input into the dynamically quantized pre-trained model. For example, for low-precision (such as INT4) inference paths, a very small, random noise perturbation or scaling transformation can be applied to the input features before computation to simulate a regularization effect and enhance the robustness of the pre-trained model to quantization noise.

[0193] Specifically, the quantization process essentially involves superimposing a deterministic but nonlinear noise onto the original continuous value: ; in It depends on the quantization error of x itself. For low-bit quantization (such as INT4), this error may be large for some x values ​​and exhibit non-uniform, non-Gaussian characteristics.

[0194] Apply a small random perturbation to the input during inference. , equivalent to: Input becomes ; Quantization output becomes ; In other words, for a small neighborhood of the original input x, the quantization error The expected value may be lower than the error at the precise point x.

[0195] The principle of lightweight data augmentation during inference: quantization error function The input space is non-smooth, with some unfavorable locations that lead to larger errors. By applying a small random perturbation to the input during inference, which is essentially an implicit sampling and averaging of the neighborhood of x, the error is smoothed. This function avoids falling into error spikes. This operation is equivalent to input-level Monte Carlo error cancellation, and its effectiveness depends on the perturbation magnitude and quantization step size. This operation does not change the model parameters of the pre-trained model, but rather optimizes for specific locations in the current input data, thus further improving the accuracy of quantization inference even when the model parameters are fixed.

[0196] The data processing method provided in this embodiment of the invention may further include: collecting the data distribution status of intermediate calculation results during the processing of the input data to be processed; and updating the scaling factor of the corresponding target sub-module according to the data distribution status.

[0197] In practice, this can be achieved by maintaining an online calibration cache. For dynamic quantization (per-token), the scaling factor is periodically updated using the KL divergence minimization method based on the recently observed distribution of activation values ​​to prevent cumulative errors caused by distribution drift.

[0198] The data processing method provided in this embodiment of the invention may further include: monitoring for anomalies in intermediate calculation results and final output results during the processing of input data; when an abnormal calculation result is detected, discarding the abnormal calculation result and replacing the corresponding sub-module of the pre-trained model with higher precision quantization parameters before re-performing the calculation.

[0199] In practice, the numerical range of the intermediate layer output (such as whether overflow occurs) and the confidence level of the final output can be monitored in real time. Once an anomaly is detected, a rollback mechanism is immediately triggered: the current low-precision calculation result is discarded, and the layer or sample is recalculated using the high-precision (such as FP16) weights of the same layer backup.

[0200] Building upon the aforementioned anomaly rollback, this embodiment of the invention can also employ a tiered caching rollback approach, setting up a three-level cache for high-precision backups of each layer, including L1 video memory, L2 RAM, and L3 disk. Upon locating an anomaly layer, different rollback methods can be adopted based on the cache level where the high-precision backup of that layer resides. If the high-precision backup of the anomaly layer exists in L1 video memory, it is directly used for recalculation, resulting in lower overhead. If the high-precision backup exists in L2 RAM, it is loaded into a temporary area of ​​video memory and recalculated. After completion, a strategy can be used to determine whether to promote the backup to L1 video memory or retain it in L2 RAM. If the high-precision backup exists only in L3 disk, the layer weight is asynchronously loaded into video memory via a memory-mapped file. Simultaneously, the system can choose to pause the current request or switch to a more conservative global quantization configuration to continue providing services, maintaining overall responsiveness. Furthermore, the system can maintain a layer anomaly frequency table. For layers that frequently trigger anomaly rollback, their high-precision backups are promoted to a higher cache level, for example, from L3 disk to L2 RAM or even L1 video memory. For backups that have not been used for a long time, reduce their cache level or remove them from the cache to save resources.

[0201] Therefore, the data processing method provided by the embodiments of the present invention provides a new paradigm of input-driven adaptive quantization. By establishing a dynamic mapping relationship between the complexity of input data and the quantization accuracy of the model, it solves the inherent contradiction that static quantization cannot take into account diverse input scenarios.

[0202] For cutting-edge complex model architectures such as hybrid expert models, a refined hybrid quantization method based on a module-sensitive knowledge base is proposed. This method achieves differentiated and optimized accuracy allocation for different functional modules within the same model, breaking through the performance bottleneck of unified quantization on heterogeneous architectures.

[0203] We designed and implemented an end-to-end collaborative system that covers algorithmic decision-making, hardware execution, and security assurance. This system not only significantly improves the efficiency and accuracy of quantitative inference but also ensures the overall reliability of the system through a built-in intelligent rollback mechanism, achieving a balance between performance and robustness.

[0204] Based on the above embodiments, this invention provides an embodiment suitable for practical application, scenario: deploying a pre-trained model for real-time approval of medical insurance claims. The hardware uses a server equipped with an accelerator.

[0205] The data processing method provided in this embodiment of the invention may include: Initialization: Load the pre-trained model into the system, run the calibration dataset offline, and build the sensitivity prior knowledge base for each target sub-module. Based on the hardware characteristics of the server, compile and generate a variable-precision fusion kernel, and load the default parameters of the decision tree.

[0206] During the inference computation task performed using the pre-trained model, each time a piece of input data to be processed is received, such as a claim form text (sequence length 256), after preprocessing, its complexity score is calculated by the input complexity real-time analysis and perception module. If its information entropy is high (describes the disease in a complex way) and the simplified model displays a wide activation distribution, the overall output complexity score CS=0.75.

[0207] Quantization Strategy Decision: The adaptive mixed-precision quantization decision and scheduling module, based on the relationship between complexity score and complexity threshold (i.e., 0.75 is greater than 0.7), queries the sensitivity prior knowledge base and obtains the following results: MLA is highly sensitive and uses W8A8 static quantization. MoE shared experts are moderately sensitive and use W8A8 dynamic quantization. MoE routing experts are low sensitive, with the basic template being W4A8. In other words, a conservative quantization inference path is adopted globally, followed by module-level fine-tuning, generating the conservative quantization decision as follows: MLA uses W8A8 static quantization, while shared experts and routing experts both use W8A8 dynamic quantization (per-token).

[0208] Hardware execution: The adaptive mixed-precision quantization decision and scheduling module distributes inference computation tasks and precision descriptors. The hardware-coordinated quantization computation acceleration device calls the W8A8 version of the fusion kernel for computation. Weights are unpacked in real time from the efficient package format and matrix multiplied with INT8 activation values ​​on the tensor core.

[0209] Accuracy Monitoring and Output: During inference computation, the calibration and accuracy assurance module monitors the overflow of output from each submodule. The pre-trained model ultimately outputs a "reject claim" conclusion with a high confidence level (0.92). The result is returned. Simultaneously, the activation value distribution of this inference is recorded for subsequent dynamic range calibration and fine-tuning.

[0210] Through experimental comparison, if the input data to be processed is a simple routine physical examination claim (Complexity_Score=0.2), the adaptive mixed-precision quantization decision and scheduling module switches the routing expert to W4A8 mode. At this time, the system calls the more efficient INT4 / INT8 mixed-precision kernel, which improves the calculation speed by about 1.4 times and reduces the memory usage, while the precision assurance module ensures the reliability of the results through lightweight enhancements.

[0211] As can be seen, the data processing method provided by the embodiments of the present invention has achieved the following effective improvements in practical applications: Significantly reduced video memory usage: For example, in a test, for a 7 billion parameter model, using the original model parameters with FP16 requires 28GB of video memory, while using the static quantization strategy with W8A8 requires 14GB of video memory. Using the dynamic quantization strategy provided by the embodiments of the present invention, the average video memory usage is only 3.6GB (which changes dynamically according to the input complexity), making it possible to deploy a model with hundreds of billions of parameters on a single card.

[0212] Inference latency is significantly reduced: In financial approval scenarios, the average inference latency is reduced from approximately 50ms for the original FP16 model parameters and 30ms for W8A8 using a static quantization strategy, to 28ms using the dynamic quantization strategy provided in this embodiment of the invention, meeting real-time requirements. For simple inputs, the latency can be further reduced to less than 20ms.

[0213] Accuracy is strictly guaranteed: using standard evaluation datasets and the pre-trained model with dynamic quantization provided in this embodiment of the invention, the accuracy loss is always controlled within an acceptable range, and in real business applications, the accuracy rate remains above 97%, which is no different from the baseline accuracy of the original model parameters of FP16.

[0214] Improved system energy efficiency: By avoiding the waste of "simple input high-precision calculation", the overall system energy efficiency (throughput / power consumption) is improved by about 25% compared to the static W8A8 quantization.

[0215] Enhanced deployment flexibility: A single system and a single model copy can adaptively handle full-spectrum input tasks ranging from simple to complex, eliminating the need to maintain multiple quantized versions of the model for different scenarios.

[0216] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0217] Embodiments of the present invention also provide a data processing apparatus, comprising: a real-time input complexity analysis and perception module, used to acquire the input data to be processed of a pre-trained model, perform complexity analysis on the input data to be processed, and obtain a complexity score; an adaptive mixed-precision quantization decision and scheduling module, used to determine the target quantization precision parameter of the target sub-module of the pre-trained model based on the comparison result of the complexity score and the complexity threshold; and a hardware collaborative quantization calculation acceleration device, used to perform quantization calculation on the input data of the target sub-module based on the target quantization precision parameter during the processing of the input data to be processed using the pre-trained model, and obtain the quantization calculation result of the target sub-module.

[0218] In this embodiment of the invention, the input complexity real-time analysis and perception module performs complexity analysis on the input data to be processed and obtains a complexity score, which may include: calculating multiple types of complexity statistical scores for the input data to be processed; obtaining a complexity score based on the multiple types of complexity statistical scores; wherein, the types of complexity statistical scores include at least two of the following: data content complexity score, data structure complexity score, and model feedback score.

[0219] In this embodiment of the invention, the calculation steps of the data content complexity score may include: calculating multiple types of content complexity scores of the input data to be processed; obtaining the data content complexity score based on the multiple types of content complexity scores; wherein the type of the content complexity score belongs to at least one of information content feature score, numerical distribution range feature score, and outlier data feature score.

[0220] In this embodiment of the invention, the calculation steps of the data structure complexity score may include: calculating multiple types of structure complexity scores of the input data to be processed; obtaining the data structure complexity score based on the multiple types of structure complexity scores; wherein the type of the structure complexity score belongs to at least one of data scale feature score, spatial structure complexity score, and temporal structure complexity score.

[0221] In this embodiment of the invention, the calculation steps for the model feedback score may include: obtaining a simplified model corresponding to the pre-trained model; performing forward propagation calculation on the input data to be processed using the simplified model, and extracting the intermediate calculation results of the simplified model; determining the model feedback score based on the data distribution characteristics of the intermediate calculation results of the simplified model; wherein, the model parameters of the simplified model come from the pre-trained model, and the number of parameters of the simplified model is smaller than that of the pre-trained model.

[0222] In this embodiment of the invention, the input complexity real-time analysis and perception module obtains a complexity score based on multiple types of complexity statistical scores, which may include: normalizing the multiple types of complexity statistical scores respectively; and weighted summing of the normalized complexity statistical scores to obtain the complexity score.

[0223] In this embodiment of the invention, the adaptive mixed-precision quantization decision and scheduling module determines the target quantization precision parameters of the target sub-module of the pre-trained model based on the complexity score and the complexity threshold. This may include: determining the complexity evaluation interval in which the complexity score is located based on the relationship between the complexity score and the complexity threshold; and determining the target quantization precision parameters of the target sub-module in the complexity evaluation interval based on a pre-established sensitivity prior knowledge base.

[0224] In this embodiment of the invention, the steps of constructing the sensitivity prior knowledge base may include: determining multiple target sub-modules from a pre-trained model; acquiring a dataset related to the target task domain of the input data to be processed; performing forward propagation calculations based on the dataset using the original model parameters of the pre-trained model to obtain baseline accuracy parameters; while keeping the original model parameters of other target sub-modules in the pre-trained model, performing forward propagation calculations for the target sub-modules in the pre-trained model under different quantization accuracy parameter configurations to obtain the loss evaluation parameters of the target sub-modules under the quantization accuracy parameter configurations compared to the baseline accuracy parameters; determining the quantization accuracy parameters corresponding to the complexity evaluation intervals based on the correspondence between the quantization accuracy parameters of the target sub-modules and the loss evaluation parameters of the target sub-modules; and constructing the sensitivity prior knowledge base using the quantization accuracy parameters corresponding to the complexity evaluation intervals of multiple target sub-modules.

[0225] In this embodiment of the invention, the adaptive mixed-precision quantization decision and scheduling module can also be used to adjust the complexity threshold in response to a threshold adjustment signal; wherein, the type of the threshold adjustment signal includes at least one of the following: the resource utilization status of the device, the status of the execution result of the pre-trained model, and the time period for executing the input data to be processed.

[0226] In this embodiment of the invention, the hardware-coordinated quantization calculation acceleration device performs quantization calculation on the input data of the target sub-module according to the target quantization precision parameter to obtain the quantization calculation result of the target sub-module. This may include: selecting a corresponding target quantization calculation core from multiple quantization calculation cores pre-deployed in the device according to the target quantization precision parameter; using the quantization calculation instruction set corresponding to the pre-deployed target quantization calculation core, calling the target quantization calculation core to perform quantization calculation on the input data of the target sub-module to obtain the quantization calculation result of the target sub-module; wherein different quantization precisions correspond to different quantization calculation cores.

[0227] The data processing apparatus provided in this embodiment of the invention may further include: an inference-time calibration and accuracy assurance module, used to collect the data distribution status of intermediate calculation results during the processing of the input data to be processed; and to update the scaling factor of the corresponding target sub-module according to the data distribution status.

[0228] The data processing apparatus provided in this embodiment of the invention may further include: an inference-time calibration and accuracy assurance module, used to monitor the intermediate calculation results and the final output results during the processing of the input data to be processed; when an abnormal calculation result is detected, the abnormal calculation result is discarded, and the corresponding sub-module of the pre-trained model is replaced with a higher precision quantization parameter before re-execution of the calculation.

[0229] For a description of the features in the embodiment corresponding to the data processing device, please refer to the relevant description in the embodiment corresponding to the data processing method, which will not be repeated here.

[0230] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data processing method embodiments.

[0231] Embodiments of the present invention also provide a non-volatile storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above data processing method embodiments when running.

[0232] In one exemplary embodiment, the aforementioned non-volatile storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0233] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0234] Embodiments of the present invention also provide another computer program product, including a non-volatile storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0235] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0236] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0237] The data processing method, electronic device, storage medium, and program product provided by this invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A data processing method, characterized by, include: Obtain the input data to be processed from the pre-trained model, perform complexity analysis on the input data to be processed, and obtain a complexity score; Based on the comparison results of the complexity score and the complexity threshold, the target quantization accuracy parameter of the target submodule of the pre-trained model is determined; During the processing of the input data to be processed using the pre-trained model, the input data of the target sub-module is quantized according to the target quantization accuracy parameter to obtain the quantization calculation result of the target sub-module; The complexity analysis of the input data to be processed, resulting in a complexity score, includes: The multi-class complexity statistical scores of the input data to be processed are calculated; The complexity score is obtained based on the complexity statistics scores of the multiple categories; The complexity statistical score includes at least two of the following: data content complexity score, data structure complexity score, and model feedback score.

2. The data processing method according to claim 1, characterized in that, The calculation steps for the data content complexity score include: The multi-category content complexity scores of the input data to be processed are calculated; The data content complexity score is obtained based on the multiple types of content complexity scores. The content complexity score is of at least one of the following types: information content feature score, numerical distribution range feature score, and outlier data feature score.

3. The data processing method according to claim 1, characterized in that, The calculation steps for the data structure complexity score include: The multi-class structural complexity scores of the input data to be processed are calculated; The data structure complexity score is obtained based on the various types of structural complexity scores. The structural complexity score is of at least one of the following types: data size feature score, spatial structural complexity score, and temporal structural complexity score.

4. The data processing method according to claim 1, characterized in that, The calculation steps for the model feedback score include: Obtain the simplified model corresponding to the pre-trained model; The simplified model is used to perform forward propagation calculations on the input data to be processed, and the intermediate calculation results of the simplified model are extracted. The model feedback score is determined based on the data distribution characteristics of the intermediate calculation results of the simplified model. The simplified model's parameters are derived from the pre-trained model, and the simplified model has fewer parameters than the pre-trained model.

5. The data processing method according to claim 4, characterized in that, Also includes: The complexity score of the input data to be processed is associated with the summary information of the input data to be processed and stored in the cache; Upon obtaining the next input data to be processed, determine whether the summary information of the next input data to be processed has hit the cache and has not expired; If the cache is hit and not invalidated, the complexity score of the hit is used as the complexity score of the next input data to be processed, and the complexity analysis of the next input data to be processed is skipped. In the event of a cache miss or an invalid cache, the simplified model is used to perform a complexity analysis on the next input data to be processed, to obtain a complexity score for the next input data to be processed, and the obtained complexity score is associated with the summary information of the next input data to be processed and stored in the cache.

6. The data processing method according to claim 1, characterized in that, The complexity score is obtained based on multiple types of complexity statistical scores, including: The complexity statistical scores of the various categories are normalized respectively; The complexity score is obtained by weighted summation of the complexity statistics scores after multi-class normalization.

7. The data processing method according to claim 1, characterized in that, Based on the comparison results of the complexity score and the complexity threshold, the target quantization accuracy parameters of the target submodule of the pre-trained model are determined, including: Based on the relationship between the complexity score and the complexity threshold, the complexity evaluation interval in which the complexity score falls is determined; Based on a pre-established sensitivity prior knowledge base, the target quantization accuracy parameter of the target submodule within the complexity evaluation range is determined.

8. The data processing method according to claim 7, characterized in that, The steps for constructing the sensitivity prior knowledge base include: Multiple target sub-modules are determined from the pre-trained model; Obtain a dataset from the target task domain of the input data to be processed; Using the original model parameters of the pre-trained model, perform forward propagation calculations based on the dataset to obtain baseline accuracy parameters; While keeping the original model parameters for the other target sub-modules in the pre-trained model, forward propagation calculations are performed on the target sub-modules in the pre-trained model under different quantization precision parameter configurations to obtain the loss evaluation parameters of the target sub-modules under the quantization precision parameter configurations compared to the baseline precision parameters. Based on the correspondence between the quantization accuracy parameters of the target submodule and the loss evaluation parameters of the target submodule, determine the quantization accuracy parameters corresponding to the complexity evaluation interval; The quantization precision parameters corresponding to the complexity evaluation intervals of the multiple target sub-modules are used to construct the sensitivity prior knowledge base.

9. The data processing method according to claim 1, characterized in that, Also includes: In response to the threshold adjustment signal, the complexity threshold is adjusted; The type of the threshold adjustment signal includes at least one of the following: the resource utilization status of the device, the status of the execution result of the pre-trained model, and the time period for executing the input data to be processed.

10. The data processing method according to claim 1, characterized in that, Based on the target quantization accuracy parameter, the input data of the target sub-module is quantized to obtain the quantization calculation result of the target sub-module, including: Based on the target quantization accuracy parameters, select the corresponding target quantization calculation core from among multiple quantization calculation cores pre-deployed in the device; Using the pre-deployed quantization calculation instruction set corresponding to the target quantization calculation core, the target quantization calculation core is invoked to perform quantization calculation on the input data of the target sub-module, thereby obtaining the quantization calculation result of the target sub-module; Different quantization accuracies correspond to different quantization calculation cores.

11. The data processing method according to claim 1, characterized in that, Also includes: The data distribution status of intermediate calculation results during the processing of the input data to be processed; The scaling factor of the corresponding target submodule is updated according to the data distribution status.

12. The data processing method according to claim 1, characterized in that, Also includes: Anomalies are monitored in the intermediate calculation results and final output results during the processing of the input data to be processed. When abnormal calculation results are detected, the abnormal calculation results are discarded, and the corresponding sub-module of the pre-trained model is replaced with higher precision quantization parameters before re-execution.

13. The data processing method according to claim 1, characterized in that, When the target quantization precision parameter indicates that the weights of the target submodule are quantized using the target bit width, the weights of the target submodule are quantized, including: The weights of the target sub-modules are quantized channel by channel according to the first bit width to obtain the scaling factor under the first bit width, wherein the first bit width is greater than the target bit width; Using the scaling factor under the first bit width as guiding information, the weights of the target sub-modules are quantized group by group according to the target bit width to obtain the quantized weights under the target bit width.

14. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data processing method as described in any one of claims 1 to 13.

Images