Downstream task processing model deployment method and device, and storage medium

Abstract

The application provides a deployment method and device of a downstream task processing model and a storage medium. The method comprises the following steps: inputting a task data set into an initial model, performing forward propagation and reverse propagation under the condition of frozen model parameters, and obtaining gradient information of each weight parameter relative to a task loss of a downstream task; constructing comprehensive sensitivity of the weight parameter according to the gradient information, a quantization error estimation and amplitude information; calculating an importance score of an output channel according to the comprehensive sensitivity of all weight parameters corresponding to the output channel; quantizing the weight parameters corresponding to the output channel in the initial model according to a target quantization bit width and an optimal quantization ratio to obtain a task processing model, and deploying the task processing model on a target edge device. The application can make the runtime computing power requirement of the quantized task processing model less than an upper limit value, thereby stably deploying on an edge device with limited memory capacity and maintaining inference accuracy on a specific downstream task.

Description

Technical Field

[0001] This application relates to the field of computer technology, and specifically to a method, apparatus and storage medium for deploying a downstream task processing model. Background Technology

[0002] In recent years, while large language models based on the Transformer architecture possess powerful language understanding and generation capabilities, their massive number of parameters leads to a surge in storage and computational overhead, making them difficult to deploy directly on edge devices with limited memory and computing power. Post-training quantization techniques, by mapping floating-point weights to low-bit integers, can significantly compress the model size without retraining, becoming an important approach for the edge deployment of large language models.

[0003] However, existing model quantization methods often use general corpora as calibration data and layer output reconstruction errors as optimization targets. While these methods can maintain the model's generalization ability, they frequently exhibit significant performance degradation in specific downstream tasks such as legal document analysis, industrial equipment fault prediction, and code generation. More critically, existing model quantization methods lead to a lack of effective matching between the runtime computing power requirements of the quantized model and the actual computing power capacity of edge devices. This results in deployment failures, runtime crashes, or exacerbated performance degradation in specific downstream tasks. Summary of the Invention

[0004] In view of this, this application proposes a deployment method, apparatus and storage medium for a downstream task processing model to solve the problem that existing model quantization methods lead to a mismatch between the runtime computing power requirements of the quantized model and the device's carrying capacity, and significant performance degradation in specific downstream tasks.

[0005] The first aspect of this application proposes a method for deploying a downstream task processing model, the method comprising: Obtain the weight parameters of the initial model to be deployed to the target edge device, the task dataset corresponding to the downstream task, and the computing power limit of the target edge device; the weight parameters are distributed in the weight matrix of the linear layer in the initial model; The task dataset is input into the initial model, and forward and backward propagation are performed with the model parameters frozen to obtain the gradient information of each weight parameter relative to the task loss of the downstream task; the gradient information is used to characterize the importance of the corresponding weight parameter to the downstream task. For any weight parameter, a comprehensive sensitivity of the weight parameter is constructed based on its gradient information, quantization error estimation, and amplitude information. The comprehensive sensitivity is used to evaluate the sensitivity contribution of the weight parameter to the task loss of the downstream task. The amplitude information is used to characterize the structural stability of the weight parameter itself. The quantization error estimation is used to characterize the inherent vulnerability of the weight parameter to quantization noise. For any output channel in the weight matrix, an importance score is calculated based on the comprehensive sensitivity of all weight parameters corresponding to the output channel. The importance score is used to characterize the sensitivity contribution of the output channel to the task loss of the downstream task. The row index of the weight matrix corresponds to the output channel, and the column index corresponds to the input channel. The target quantization bit width is determined based on the upper limit of computing power, and the optimal quantization ratio is calculated based on the importance score of the output channel; the target quantization bit width is the uniform bit width supported by the target edge device. Based on the target quantization bit width and the optimal quantization ratio, all weight parameters corresponding to the output channel in the initial model are quantized to obtain a task processing model for the downstream task, and the task processing model is deployed on the target edge device; the runtime computing power requirement of the task processing model is less than the upper limit of computing power.

[0006] This application embodiment obtains the upper limit of the computing power of the target edge device and determines the target quantization bit width accordingly, ensuring that the runtime computing power requirement of the quantized task processing model is less than the upper limit, thereby ensuring that the model can be stably deployed and run on the target edge device. By performing forward and backward propagation to obtain gradient information under the condition of freezing model parameters, there is no need to retrain or fine-tune the initial model, which significantly reduces data costs and computational overhead, making it suitable for scenarios such as large language models where retraining costs are extremely high. By fusing gradient information, quantization error estimation, and amplitude information to construct a comprehensive sensitivity, it can simultaneously consider the task relevance of weight parameters to specific downstream tasks, their inherent vulnerability to quantization noise, and their own structural stability, thereby accurately identifying truly sensitive weights worthy of protection and avoiding misjudgments caused by relying solely on gradients or amplitudes. By aggregating the comprehensive sensitivity per parameter into an importance score per output channel and searching for the optimal quantization ratio based on the importance score, channel-level differentiated quantization protection is achieved: key task channels receive a finer quantization ratio, while edge channels are allowed coarser quantization, thereby significantly improving the performance of the quantized task processing model on specific downstream tasks under hardware-friendly constraints of uniform bit width.

[0007] An embodiment of the second aspect of this application provides a deployment apparatus for a downstream task processing model, comprising: The data acquisition module is used to acquire the weight parameters of the initial model to be deployed to the target edge device, the task dataset corresponding to the downstream task, and the computing power limit of the target edge device; the weight parameters are distributed in the weight matrix of the linear layer in the initial model; The gradient information acquisition module is used to input the task dataset into the initial model, perform forward propagation and backward propagation under the condition of freezing the model parameters, and acquire the gradient information of each weight parameter relative to the task loss of the downstream task; the gradient information is used to characterize the importance of the corresponding weight parameter to the downstream task. A comprehensive sensitivity construction module is used to construct a comprehensive sensitivity of any weight parameter based on its gradient information, quantization error estimation, and amplitude information. The comprehensive sensitivity is used to evaluate the sensitivity contribution of the weight parameter to the task loss of the downstream task. The amplitude information is used to characterize the structural stability of the weight parameter itself. The quantization error estimation is used to characterize the inherent vulnerability of the weight parameter to quantization noise. The importance score calculation module is used to calculate the importance score of any output channel in the weight matrix based on the comprehensive sensitivity of all weight parameters corresponding to the output channel. The importance score is used to characterize the sensitivity contribution of the output channel to the task loss of the downstream task. The row index of the weight matrix corresponds to the output channel, and the column index corresponds to the input channel. The optimal quantization ratio calculation module is used to determine the target quantization bit width based on the upper limit of computing power, and to calculate the optimal quantization ratio based on the importance score of the output channel; the target quantization bit width is the uniform bit width supported by the target edge device; The model deployment module is used to quantize all weight parameters corresponding to the output channel in the initial model according to the target quantization bit width and the optimal quantization ratio to obtain a task processing model for the downstream task, and deploy the task processing model on the target edge device; the runtime computing power requirement of the task processing model is less than the computing power upper limit.

[0008] An embodiment of the third aspect of this application provides a computer device including a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the deployment method of the downstream task processing model described in the first aspect above.

[0009] An embodiment of the fourth aspect of this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the deployment method of the downstream task processing model described in the first aspect above.

[0010] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0011] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a deployment method for a downstream task processing model provided in an embodiment of this application is shown. Figure 2 This invention provides a schematic diagram of the structure of a deployment device for a downstream task processing model according to an embodiment of the present application. Figure 3 This invention provides a schematic diagram of the structure of a computer device according to an embodiment of the present application. Figure 4 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0012] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0013] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0014] The following describes the technical scenarios involved in the embodiments of this application: In recent years, while large language models based on the Transformer architecture possess powerful language understanding and generation capabilities, their massive parameter count leads to a surge in storage and computational overhead, making direct deployment on edge devices with limited memory and computing power difficult. Specifically, edge devices typically have strict upper limits on computing power, including memory capacity limits and processor bit width support limits (such as only supporting uniform bit width operations like INT3 or INT4), making it impossible for the original large language models to run directly on such devices. Post-training quantization techniques, by mapping floating-point weights to low-bit integers, can significantly compress the model size and reduce runtime computational requirements without retraining the model, becoming an important approach for the edge deployment of large language models.

[0015] In a unified affine quantization framework for linear layers, the quantized weight parameters are typically obtained through scaling, rounding, zero-addition, and clipping operations. Specifically, for the weight matrix... Its quantized weight parameters The channel-level quantizer is determined by the following formula:

[0016] in, For quantizer, For scaling vectors, Zero point This refers to the bit width. Theoretically, the ideal quantization objective should be to find the optimal quantization parameters that minimize the expected task loss of the quantized model on a specific downstream task dataset, i.e.: However, because the above quantization process includes rounding ( ) and cutting ( For discrete operations such as [missing information], directly optimizing the discrete quantization parameters with the global task loss function as the target is computationally infeasible. More importantly, existing model quantization methods lead to a lack of effective matching between the runtime computing power requirements of the quantized model and the actual computing power capacity of edge devices, resulting in deployment failures, runtime crashes, or exacerbated performance degradation on specific downstream tasks.

[0017] To address the infeasibility of directly optimizing discrete quantization parameters, this application introduces a Taylor series approximation to estimate the impact of quantization noise on task loss. Specifically, let... The change in task loss is represented by the perturbation introduced by quantization. It can be approximated by a second-order Taylor expansion at the original weights as follows:

[0018] in, For gradient, The matrix is ​​the Hessian matrix. For large language models with billions of parameters, calculating the complete Hessian matrix is ​​neither storage-efficient nor computationally feasible. Therefore, this application focuses on the first-order term in the above expansion, utilizing gradient information to capture the directional sensitivity of the loss landscape, thereby achieving effective quantification of the task sensitivity of each weight parameter within an acceptable computational cost.

[0019] Furthermore, this application recognizes that relying solely on gradients ignores the magnitude of quantization errors, while relying solely on quantization errors ignores task relevance. Therefore, this application constructs a composite sensitivity that integrates gradient information, quantization error estimation, and magnitude information to simultaneously consider three dimensions: task relevance, quantization vulnerability, and structural stability. This enables precise positioning of task-sensitive weights and adaptively determines the target quantization ratio based on the upper limit of the target edge device's computing power. This ensures that the runtime computing power requirement of the quantized task processing model is less than the upper limit, thus mitigating performance degradation of specific downstream tasks to the greatest extent possible.

[0020] Example 1: The model edge deployment method provided in this application is applicable to various edge devices with inference capabilities. Its implementation does not depend on specific hardware configurations, memory capacity, or downstream task types. The deployment process is illustrated below with a specific example: Taking the deployment of a large language model to an industrial edge gateway with 4GB of memory as an example, this paper details the overall process of deploying the downstream task processing model. The downstream task is industrial equipment fault prediction, which outputs fault type prediction and operation and maintenance decision suggestions based on equipment sensor data (such as bearing vibration frequency, motor winding temperature, oil metal particle concentration, etc.).

[0021] Figure 1 This is a flowchart of a deployment method for a downstream task processing model according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain the weight parameters of the initial model to be deployed to the target edge device, the task dataset corresponding to the downstream task, and the computing power limit of the target edge device.

[0022] Specifically, the weight parameters are distributed in the weight matrices of each linear layer (i.e., the linear layer) in the initial model. In the Transformer architecture, the weight matrix of each linear layer... The row index corresponds to the output channel ( Each output neuron), the column index corresponds to the input channel ( (One input feature dimension).

[0023] More specifically, the computing power limit includes, but is not limited to, the memory limit of the target edge device. In some specific embodiments, the computing power limit can also be pre-configured according to the actual hardware specifications of the target edge device and the real-time requirements of downstream tasks.

[0024] In this embodiment, the task dataset is a task calibration dataset consisting of 128 pieces of equipment sensor data, including operating parameters such as bearing vibration spectrum, motor winding temperature curve, and oil metal particle concentration, as well as corresponding fault type labels (such as early wear of gearbox, aging of motor insulation, and leakage of hydraulic system).

[0025] Step S102: Input the task dataset into the initial model, perform forward propagation and backward propagation while freezing the model parameters, and obtain the gradient information of each weight parameter relative to the task loss of the downstream task.

[0026] Specifically, the task dataset (e.g., the 128 device sensor data points mentioned above) is input into the initial model. Forward and backward propagation are performed with the model parameters frozen (requires_grad=False, disabling weight updates). During forward propagation, the model calculates and outputs predictions layer by layer, and the task loss is calculated based on the predictions and the true labels. (e.g., cross-entropy loss). During backpropagation, the task loss is calculated. For each weight parameter partial derivatives Gradient information is obtained by taking the absolute value. .

[0027] The gradient information is used to characterize the importance of the corresponding weight parameter to the downstream task. This gradient information refers to the instantaneous rate of change of the task loss when only a small perturbation is applied to this weight parameter while keeping all other weight parameters constant. Therefore, the larger the magnitude of the gradient information, the more significant the influence of the weight parameter on the inference result of the downstream task, indicating that the weight parameter encodes features directly related to the downstream task. Conversely, the smaller the magnitude of the gradient information, the smaller the influence of the weight parameter on the inference result of the current downstream task; even if a certain numerical deviation occurs, it will not cause significant fluctuations in the task loss. In other words, in this embodiment, the larger the magnitude of the gradient information, the more important the weight parameter is to the industrial equipment fault prediction task.

[0028] Step S103: For any weight parameter, construct the comprehensive sensitivity of the weight parameter based on the gradient information, quantization error estimation and magnitude information of the weight parameter.

[0029] Specifically, for any weight parameter Based on the gradient information, quantization error estimation, and magnitude information of the weight parameters, a comprehensive sensitivity of the weight parameters is constructed. Among them, the overall sensitivity is used to evaluate the sensitivity contribution of the weight parameters to the task loss of downstream tasks, the magnitude information is used to characterize the structural stability of the weight parameters themselves (i.e. the magnitude of the absolute value of the weights, with larger weights usually encoding more significant features), and the quantization error estimation is used to characterize the inherent vulnerability of the weight parameters to quantization noise (i.e. the deviation between the coarse quantization result and the original value).

[0030] In some specific embodiments, step S103 above includes steps S1031-S1033: Step S1031: Calculate the quantization error estimate.

[0031] Specifically, for any weight parameter The weight parameters are coarsely quantized to obtain quantized proxy values. The coarse quantization process uses a standard quantization proxy, which performs quantization according to a fixed rule of linear mapping between the minimum and maximum channel ranges.

[0032] in, This is a fixed scaling factor pre-calculated based on the maximum absolute value of the channel.

[0033] Then, based on the weight parameters With quantified proxy value The absolute difference is used to calculate the quantization error estimate of the weight parameters:

[0034] This quantization error estimate characterizes the inherent vulnerability of the weighting parameter to quantization noise: a larger quantization error estimate indicates that the weighting parameter is more easily damaged by coarse quantization; conversely, a smaller quantization error estimate indicates that the weighting parameter has natural robustness to quantization noise and can maintain approximate functionality even after quantization. The following example illustrates this: Weight parameters This weight parameter is located in a certain output channel and corresponds to the mapping relationship from the input feature of "abnormal bearing vibration frequency" to the output feature of "gearbox fault". When using a standard quantization proxy for coarse quantization, a fixed scaling factor is set. The surrogate value after coarse quantization of this weight is... At this point, the quantization error estimate for this weight parameter is: The large error estimate indicates that the weighting parameter is highly vulnerable to quantization noise. If it is not carefully processed in actual quantization, the strength of the critical prediction path "abnormal bearing vibration frequency → gearbox failure" will be weakened, thereby causing a decrease in the accuracy of industrial equipment failure prediction tasks.

[0035] Step S1032: Obtain the magnitude information of the weight parameters.

[0036] Specifically, the magnitude information is the absolute value of the weighting parameters. Or the preset exponent of the absolute value of the weight parameter. ( =1).

[0037] More specifically, the magnitude information of weight parameters is used to characterize the structural stability of the weight parameters themselves: weight parameters with large absolute values ​​usually correspond to feature associations that the model has stably learned and that have strong significance (such as the mapping relationship of key terms in sensor data of industrial equipment or the association of core concepts in legal documents). Their values ​​themselves have structural stability and play a dominant role in the inference process; while weight parameters with absolute values ​​close to zero often encode noise or task-irrelevant subtle patterns, and their values ​​are easily affected by gradient noise. Therefore, this magnitude information, as a structural prior, is used to characterize the inherent importance hierarchy of weight parameters in the model parameter space, preventing sensitivity misjudgments caused by random fluctuations in gradient information on small weight parameters, and ensuring that the comprehensive sensitivity measurement focuses on weight parameters that are numerically reliable and structurally significant. The following example illustrates this: Weight parameters This weight corresponds to the mapping from the input feature "high-frequency bearing vibration" to the output "early gearbox wear". Its absolute value... The large magnitude indicates that the connection was sufficiently strengthened during the pre-training process of the initial model, stably encoding the strong industrial association between "high-frequency bearing vibration" and "early gearbox wear". Even though the gradient information generated by this weight due to sample randomness in the current batch of calibration data (only 128 device sensor logs) is at a moderate level, its large magnitude information still ensures that the weight occupies a high position in the overall sensitivity, thus receiving priority protection - because large weights usually carry more significant and robust semantic structures.

[0038] Step S1033: The gradient information, quantization error estimation and amplitude information of the weight parameters are fused to obtain the comprehensive sensitivity of the weight parameters.

[0039] Specifically, the overall sensitivity is positively correlated with gradient information, quantization error estimation, and amplitude information; that is, an increase in any one of these three factors leads to a corresponding increase in overall sensitivity. The result is obtained by combining the products of the three, as shown below:

[0040] in, Indicates the first The output channel The weight parameters corresponding to each input channel Indicates mission loss. The partial derivative operator is represented. Represents weight parameters The gradient information, that is, the absolute value of the partial derivative of the task loss with respect to the weight parameter; This indicates coarse quantization. Represents weight parameters Quantization error estimation, This indicates the preset index (usually set to 1). Represents weight parameters Amplitude information, Represents weight parameters Overall sensitivity to downstream tasks.

[0041] In this embodiment, the overall sensitivity is not a single-dimensional evaluation, but rather considers the joint performance of the weight parameter across three dimensions: task relevance (gradient information), quantization vulnerability (quantization error estimation), and structural stability (amplitude information). Specifically, a high overall sensitivity score is obtained only when a weight parameter simultaneously satisfies the following conditions: "important to downstream tasks" (large gradient amplitude), "easily damaged by quantization" (large quantization error), and "significant in value" (large amplitude). Therefore, the overall sensitivity accurately assesses the degree of sensitivity contribution to the performance loss of downstream tasks caused by numerical deviations during the quantization process of the weight parameter (i.e., compressing from full precision to low bit depth, such as compressing the weight parameter from 16 bits to 3 bits). A higher overall sensitivity indicates that the weight parameter is more likely to cause a significant increase in task loss after quantization, and therefore requires more emphasis on protection during quantization optimization.

[0042] Step S104: For any output channel in the weight matrix, calculate the importance score of the output channel based on the comprehensive sensitivity of all weight parameters corresponding to the output channel.

[0043] Specifically, for the output channels in the weight matrix All corresponding weight parameters (i.e., the first) Weight parameters of all columns corresponding to a row Overall sensitivity Summation is performed to obtain the output channel. Initial importance score :

[0044] Then, the initial importance scores of all output channels are normalized to obtain the total importance score. This embodiment uses mean-standard-deviation normalization:

[0045] in, This is the mean of the initial importance scores for all output channels. Standard deviation, To prevent division by zero of extremely small constants.

[0046] In this embodiment, the importance score is a channel-level score obtained by aggregating (e.g., summing) the overall sensitivity of all weight parameters under the same output channel along the input channel dimension and then normalizing it. In a neural network, an output channel corresponds to a set of weight vectors that map input features to a specific semantic space. Its overall behavior determines the type of features extracted or transformed by the channel during inference (such as fault description features in device logs or liability clause features in legal documents). Therefore, the importance score characterizes the degree of sensitivity contribution of the output channel to the loss of downstream tasks: the higher the importance score, the more task-critical information the output channel encodes, and its quantization distortion will be directly transmitted to subsequent network layers, leading to a significant degradation in task performance; the lower the importance score, the more general or marginal features the output channel mainly encodes, and its moderate quantization has a limited impact on the final task result.

[0047] Step S105: Determine the target quantization bit width based on the upper limit of computing power, and calculate the optimal quantization ratio based on the importance score of the output channel.

[0048] In this embodiment, the original model occupies approximately 13GB, far exceeding the 4GB memory limit. After evaluation, the target quantization bit width was determined to be 3 (INT3), resulting in a quantized model size of approximately 2.5GB, which meets the computing power constraint. The target quantization bit width is the uniform bit width supported by the target edge device, meaning all output channels uniformly use a 3-bit integer representation.

[0049] In some specific embodiments, step S105 above includes steps S1051-S1054: Step S1051: For any preset quantization ratio among multiple preset quantization ratios, use the preset quantization ratio to quantize the weight parameters in the weight matrix corresponding to the output channel to obtain the quantized weight parameters.

[0050] Specifically, for multiple preset quantization ratios ( Any preset quantization ratio in ) Use preset quantization ratio For each weight parameter corresponding to the output channel Quantization is performed to obtain the quantized weight parameters. .

[0051] Here, the quantization process can be understood as performing scaling, rounding, and restoration operations sequentially using the following formulas:

[0052] For example: when any weight parameter is 0.024 and the preset quantization ratio is 0.008, the weight parameter after scaling is 0.024 / 0.008 = 3.0; then, 3.0 is rounded (e.g., rounded to the nearest integer) to obtain the rounded weight parameter. Finally, the quantized weight parameters are obtained by restoring 3 using a preset quantization ratio. .

[0053] Step S1052: Calculate the first output feature based on the quantized weight parameters and the task dataset, calculate the second output feature based on the weight parameters and the task dataset, and calculate the reconstruction error based on the first output feature and the second output feature.

[0054] Specifically, the activation input passed from the preceding network layer to the current linear layer is obtained. ( (Number of calibration samples in the task dataset): based on the quantized weight parameters and activation input Calculate the first output feature: According to the weight parameters and activation input Calculate the second output feature: The reconstruction error is calculated based on the first and second output features.

[0055] Step S1053: Multiply the reconstruction error by the importance score of the output channel to calculate the importance-weighted reconstruction error corresponding to the preset quantization ratio.

[0056] Specifically, the importance-weighted reconstruction error is calculated as follows:

[0057] in, Indicates output channel Importance score.

[0058] Step S1054: Compare the importance-weighted reconstruction errors corresponding to each preset quantization ratio, and determine the preset quantization ratio corresponding to the minimum importance-weighted reconstruction error as the optimal quantization ratio for the output channel.

[0059] Specifically, multiple preset quantization ratios are iterated to obtain the importance-weighted reconstruction error corresponding to each preset quantization ratio. The importance-weighted reconstruction errors corresponding to each preset quantization ratio are compared, and the preset quantization ratio corresponding to the minimum importance-weighted reconstruction error is determined as the optimal quantization ratio for the output channel. Among them, due to Larger channels (such as those used for fault prediction in industrial equipment) have higher weights assigned to their reconstruction errors, and the search algorithm will select more refined channels for them. ;and Smaller channels allow for the selection of coarser textures. .

[0060] This application's embodiments achieve a refined search for the optimal quantization ratio for each channel by traversing multiple preset quantization ratios and calculating the corresponding importance-weighted reconstruction errors. By multiplying the reconstruction error by the importance score of the output channel, the reconstruction error of high-importance channels is given a higher weight in the objective function, thereby forcing the search algorithm to select a finer quantization ratio for high-importance channels, prioritizing the protection of mission-critical structures; at the same time, it allows low-importance channels to occupy a smaller weight in the objective function, making them acceptable with a coarser quantization ratio. This "biased" search mechanism achieves a dynamic tilt of the precision budget towards mission-critical channels without increasing additional storage overhead, significantly improving the performance of the uniform bit-width quantization model on specific downstream tasks.

[0061] In some specific embodiments, the optimal quantization ratio is calculated using the following formula:

[0062] in, Indicates output channel The optimal quantization ratio, Indicates output channel Importance score Represents the weight matrix Middle The weight parameters of all columns corresponding to the row. This represents the activation input obtained from the task dataset. Represents the weight matrix Middle The weight parameters of all columns corresponding to the row are set using a preset quantization ratio. Quantification is performed.

[0063] Step S106: Based on the target quantization bit width and the optimal quantization ratio, quantize all weight parameters corresponding to the output channel in the initial model to obtain a task processing model for downstream tasks. Deploy the task processing model on the target edge device. The runtime computing power requirement of the task processing model is less than the upper limit of computing power.

[0064] In some specific embodiments, step S106 above includes steps S1061-S1064: Step S1061: Determine the zero offset of the output channel and determine the range of integer values ​​based on the target quantization bit width.

[0065] Specifically, the zero-point offset of the output channel is determined (symmetric quantization is used in this embodiment). The range of integer values ​​is determined based on the target quantization bit width. For example, at that time The range of integer values ​​is [ 4,3].

[0066] Step S1062: For any weight parameter corresponding to the output channel, perform quantization processing on the weight parameter according to the optimal quantization ratio to obtain the quantization processing result.

[0067] Step S1063: Round the quantization result, sum the rounded result with the zero offset, and trim the summation result to the range of integer values ​​to obtain the quantized integer value.

[0068] In steps S1062-S1063, firstly, for any weight parameter corresponding to the output channel... Based on the optimal quantization ratio of this output channel The weight parameters are scaled to obtain the scaled result. Then, the scaling result is rounded to the nearest integer to obtain the rounded result. Finally, the rounded result is summed with the zero-point offset, and the summation result is then trimmed to the nearest integer value. This yields the quantized integer value.

[0069] Step S1064: Multiply the quantized integer value by the optimal quantization ratio to obtain the quantized weight parameters of the output channel.

[0070] Specifically, the quantized weight parameters are calculated using the following formula. :

[0071] The initial model is quantized by iterating through all weight parameters of all output channels. The optimal quantization ratio for each output channel is different (obtained independently based on their respective importance scores), and all output channels share the same target quantization bit width (uniform bit width). This ensures that the quantized task processing model can directly utilize the standard low-bit integer arithmetic units of edge devices for efficient inference.

[0072] This application embodiment determines the target quantization bit width based on the upper limit of computing power, and determines the zero-point offset and integer value range through the target quantization bit width. It then sequentially performs scaling, rounding, zero-point offset, and clipping operations, establishing a standardized channel-level affine quantization process to ensure the reproducibility and numerical stability of the quantization results. By multiplying the quantized integer value of each output channel by its optimal quantization ratio, while maintaining a uniform bit width (all output channels share the same target quantization bit width), it achieves differentiated precision preservation for each channel: mission-critical channels... Smaller size achieves higher effective accuracy, edge channels due to A larger bit width allows for a higher compression ratio. This design, which "maintains a uniform bit width under the constraint of computing power limit and combines it with differentiated ratios," enables the quantized task processing model to both directly utilize the low-bit integer arithmetic units of edge devices for efficient inference (hardware friendly) and maintain high fidelity on the key structures of the task (task-aware), thus achieving a balance between device computing power constraints and downstream task performance.

[0073] In some specific embodiments, the downstream task is a vertical domain cognitive task oriented towards the target edge device; the vertical domain cognitive task includes at least one of legal document analysis, code generation, intelligent decision-making in industrial scenarios, and multilingual translation. Specifically: Scenario 1: Legal Document Analysis Task When the downstream task is legal document analysis, the task dataset includes legal contract texts, judgments, and legal provisions. The target edge devices are mobile terminals or government edge computing nodes. The quantized model can achieve intelligent parsing of legal texts in an isolated network environment.

[0074] Scenario 2: Code Generation Task When the downstream task is a code generation task, the task dataset includes code snippets, function comments, or programming question-and-answer pairs. The target edge device is the developer's workstation's built-in AI acceleration module or an edge code completion terminal.

[0075] Scenario 3: Intelligent Decision-Making Tasks in Industrial Scenarios When the downstream task is an intelligent decision-making task in an industrial scenario, the task dataset includes equipment sensor logs, process parameter records, and fault diagnosis reports. The target edge device is an industrial gateway or a production line edge controller. The computing power of the industrial gateway does not support mixed-precision inference acceleration, therefore the unified bit-width quantization scheme of this application must be adopted.

[0076] Scenario 4: Multilingual Translation Task When the downstream task is a multilingual translation task, the task dataset includes parallel corpora and a domain-specific terminology lookup table. The target edge device is an IoT translation terminal or a smart wearable device.

[0077] This application's embodiments limit the downstream tasks to specific areas such as legal document analysis, code generation, intelligent decision-making in industrial scenarios, and multilingual translation. Correspondingly, they are equipped with domain-specific datasets such as court judgments, code snippets, sensor logs, and parallel corpora. This allows the quantization process to be task-aware and optimized for the real-world data distribution within the vertical domain, avoiding performance degradation caused by inconsistencies between general corpora and the target task distribution. By specifying the target edge devices as government edge computing nodes and industrial gateways, the irreplaceable nature of the unified bit-width quantization scheme under specific hardware constraints is clarified, enhancing the scenario adaptability and feasibility of the technical solution.

[0078] Example 2: Corresponding to the implementation of the deployment method for the upstream and downstream task processing models, this application embodiment also provides a deployment apparatus for a downstream task processing model, used to execute the deployment method for the downstream task processing model of the above embodiments. For example... Figure 2 As shown, the deployment device for this downstream task processing model includes: The data acquisition module is used to acquire the weight parameters of the initial model to be deployed to the target edge device, the task dataset corresponding to the downstream task, and the computing power limit of the target edge device; the weight parameters are distributed in the weight matrix of the linear layer in the initial model; The gradient information acquisition module is used to input the task dataset into the initial model, perform forward propagation and backward propagation under the condition of freezing the model parameters, and acquire the gradient information of each weight parameter relative to the task loss of the downstream task; the gradient information is used to characterize the importance of the corresponding weight parameter to the downstream task. A comprehensive sensitivity construction module is used to construct a comprehensive sensitivity of any weight parameter based on its gradient information, quantization error estimation, and amplitude information. The comprehensive sensitivity is used to evaluate the sensitivity contribution of the weight parameter to the task loss of the downstream task. The amplitude information is used to characterize the structural stability of the weight parameter itself. The quantization error estimation is used to characterize the inherent vulnerability of the weight parameter to quantization noise. The importance score calculation module is used to calculate the importance score of any output channel in the weight matrix based on the comprehensive sensitivity of all weight parameters corresponding to the output channel. The importance score is used to characterize the sensitivity contribution of the output channel to the task loss of the downstream task. The row index of the weight matrix corresponds to the output channel, and the column index corresponds to the input channel. The optimal quantization ratio calculation module is used to determine the target quantization bit width based on the upper limit of computing power, and to calculate the optimal quantization ratio based on the importance score of the output channel; the target quantization bit width is the uniform bit width supported by the target edge device; The model deployment module is used to quantize all weight parameters corresponding to the output channel in the initial model according to the target quantization bit width and the optimal quantization ratio to obtain a task processing model for the downstream task, and deploy the task processing model on the target edge device; the runtime computing power requirement of the task processing model is less than the computing power upper limit.

[0079] Optionally, the comprehensive sensitivity construction module is further configured to: perform coarse quantization processing on any weight parameter to obtain a quantization proxy value; calculate the quantization error estimate of the weight parameter based on the absolute difference between the weight parameter and the quantization proxy value; obtain the amplitude information of the weight parameter, wherein the amplitude information is the absolute value of the weight parameter or a preset exponent power of the absolute value of the weight parameter; and fuse the gradient information, the quantization error estimate, and the amplitude information of the weight parameter to obtain the comprehensive sensitivity of the weight parameter, wherein the comprehensive sensitivity is positively correlated with the gradient information, the quantization error estimate, and the amplitude information.

[0080] Optionally, the optimal quantization ratio calculation module is further configured to: quantize the weight parameters corresponding to the output channel in the weight matrix using any one of a plurality of preset quantization ratios to obtain quantized weight parameters; calculate a first output feature based on the quantized weight parameters and the task dataset; calculate a second output feature based on the weight parameters and the task dataset; calculate a reconstruction error based on the first output feature and the second output feature; multiply the reconstruction error by the importance score of the output channel to calculate the importance-weighted reconstruction error corresponding to the preset quantization ratio; compare the importance-weighted reconstruction errors corresponding to each preset quantization ratio, and determine the preset quantization ratio corresponding to the minimum importance-weighted reconstruction error as the optimal quantization ratio for the output channel.

[0081] Optionally, the downstream task is a vertical domain cognitive task for the target edge device; the vertical domain cognitive task includes at least one of legal document analysis, code generation, intelligent decision-making in industrial scenarios, and multilingual translation.

[0082] Optionally, the model deployment module is further configured to determine the zero-point offset of the output channel and determine the integer value range based on the target quantization bit width; for any weight parameter corresponding to the output channel, the weight parameter is quantized according to the optimal quantization ratio to obtain a quantization result; the quantization result is rounded, the rounded result is summed with the zero-point offset, and the summation result is clipped to the integer value range to obtain a quantized integer value; the quantized integer value is multiplied by the optimal quantization ratio to obtain the quantized weight parameter of the output channel; wherein the optimal quantization ratios of each output channel are different, and all output channels share the same target quantization bit width.

[0083] The deployment apparatus for the downstream task processing model provided in the above embodiments of this application and the deployment method for the downstream task processing model provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0084] This application also provides a computer device for executing the deployment method of the downstream task processing model described above. Please refer to... Figure 3 This illustrates a schematic diagram of a computer device provided by some embodiments of this application. For example... Figure 3 As shown, the computer device 3 includes: a processor 300, a memory 301, a bus 302, and a communication interface 303. The processor 300, the communication interface 303, and the memory 301 are connected via the bus 302. The memory 301 stores a computer program that can run on the processor 300. When the processor 300 runs the computer program, it executes the deployment method of the downstream task processing model provided in the foregoing embodiments of this application.

[0085] The memory 301 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 303 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0086] Bus 302 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Memory 301 is used to store programs. After receiving an execution instruction, the processor 300 executes the program. The deployment method of the downstream task processing model disclosed in the foregoing embodiments can be applied to the processor 300, or implemented by the processor 300.

[0087] The processor 300 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 300 or by instructions in software form. The processor 300 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 301. The processor 300 reads the information in memory 301 and, in conjunction with its hardware, completes the steps of the above method.

[0088] The computer device and the deployment method of the downstream task processing model provided in this application are based on the same inventive concept and have the same beneficial effects as the methods they adopt, run or implement.

[0089] This application also provides a computer-readable storage medium corresponding to the deployment method of the downstream task processing model provided in the foregoing embodiments. Please refer to [link / reference]. Figure 4 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the deployment method of the downstream task processing model provided in any of the foregoing embodiments.

[0090] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0091] The computer-readable storage medium provided in the above embodiments of this application and the deployment method of the downstream task processing model provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0092] It should be noted that: Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0093] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0094] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0095] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for deploying a downstream task processing model, characterized in that, The method includes: Obtain the weight parameters of the initial model to be deployed to the target edge device, the task dataset corresponding to the downstream task, and the computing power limit of the target edge device; the weight parameters are distributed in the weight matrix of the linear layer in the initial model; The task dataset is input into the initial model, and forward and backward propagation are performed with the model parameters frozen to obtain the gradient information of each weight parameter relative to the task loss of the downstream task; the gradient information is used to characterize the importance of the corresponding weight parameter to the downstream task. For any weight parameter, a comprehensive sensitivity of the weight parameter is constructed based on its gradient information, quantization error estimation, and amplitude information. The comprehensive sensitivity is used to evaluate the sensitivity contribution of the weight parameter to the task loss of the downstream task. The amplitude information is used to characterize the structural stability of the weight parameter itself. The quantization error estimation is used to characterize the inherent vulnerability of the weight parameter to quantization noise. For any output channel in the weight matrix, an importance score is calculated based on the comprehensive sensitivity of all weight parameters corresponding to the output channel. The importance score is used to characterize the sensitivity contribution of the output channel to the task loss of the downstream task. The row index of the weight matrix corresponds to the output channel, and the column index corresponds to the input channel. The target quantization bit width is determined based on the upper limit of computing power, and the optimal quantization ratio is calculated based on the importance score of the output channel; the target quantization bit width is the uniform bit width supported by the target edge device. Based on the target quantization bit width and the optimal quantization ratio, all weight parameters corresponding to the output channel in the initial model are quantized to obtain a task processing model for the downstream task, and the task processing model is deployed on the target edge device; the runtime computing power requirement of the task processing model is less than the upper limit of computing power.

2. The method according to claim 1, characterized in that, Based on the gradient information, the quantization error estimate of each weight parameter, and the magnitude information, a comprehensive sensitivity is constructed, including: For any weight parameter, coarse quantization is performed on the weight parameter to obtain a quantization proxy value, and the quantization error estimate of the weight parameter is calculated based on the absolute difference between the weight parameter and the quantization proxy value. Obtain the amplitude information of the weight parameter, wherein the amplitude information is the absolute value of the weight parameter or a preset exponent power of the absolute value of the weight parameter; The gradient information, quantization error estimation, and amplitude information of the weight parameters are fused to obtain the comprehensive sensitivity of the weight parameters. The comprehensive sensitivity is positively correlated with the gradient information, quantization error estimation, and amplitude information.

3. The method according to claim 2, characterized in that, The overall sensitivity is calculated using the following formula: in, Indicates the first The output channel The weight parameters corresponding to each input channel Indicates mission loss. The partial derivative operator is represented. Represents weight parameters gradient information, This indicates coarse quantization. Represents weight parameters Quantization error estimation, This indicates the preset index. Represents weight parameters Amplitude information, Represents weight parameters Overall sensitivity to the downstream tasks.

4. The method according to claim 1, characterized in that, Calculate the optimal quantization ratio based on the importance score of the output channel, including: For any one of a plurality of preset quantization ratios, the preset quantization ratio is used to quantize the weight parameters in the weight matrix corresponding to the output channel to obtain quantized weight parameters. The first output feature is calculated based on the quantized weight parameters and the task dataset; the second output feature is calculated based on the weight parameters and the task dataset; and the reconstruction error is calculated based on the first output feature and the second output feature. The importance-weighted reconstruction error corresponding to the preset quantization ratio is calculated by multiplying the reconstruction error by the importance score of the output channel. Compare the importance-weighted reconstruction errors corresponding to each preset quantization ratio, and determine the preset quantization ratio corresponding to the minimum importance-weighted reconstruction error as the optimal quantization ratio for the output channel.

5. The method according to claim 4, characterized in that, The optimal quantization ratio is calculated using the following formula: in, Indicates output channel The optimal quantization ratio, Indicates output channel Importance score Represents the weight matrix Middle The weight parameters of all columns corresponding to the row. Represents the task dataset, Represents the weight matrix Middle The weight parameters of all columns corresponding to the row are set using a preset quantization ratio. Quantification is performed.

6. The method according to claim 1 or 2, characterized in that, The downstream task is a vertical domain cognitive task for the target edge device; the vertical domain cognitive task includes at least one of legal document analysis, code generation, intelligent decision-making in industrial scenarios, and multilingual translation.

7. The method according to claim 6, characterized in that, When the downstream task is a legal document analysis task, the task dataset includes legal contract texts, judgments, and legal provisions; the target edge device is a mobile terminal or a government edge computing node. When the downstream task is a code generation task, the task dataset includes code snippets, function comments, or programming question-and-answer pairs; the target edge device is the built-in AI acceleration module of the developer's workstation or an edge code completion terminal. When the downstream task is an intelligent decision-making task in an industrial scenario, the task dataset includes equipment sensor logs, process parameter records, and fault diagnosis reports; the target edge device is an industrial gateway or a production line edge controller. When the downstream task is a multilingual translation task, the task dataset includes parallel corpora and a domain-specific terminology lookup table; the target edge device is an IoT translation terminal or a smart wearable device.

8. The method according to claim 1 or 2, characterized in that, Based on the target quantization bit width and the optimal quantization ratio, all weight parameters corresponding to the output channel in the initial model are quantized to obtain a task processing model for the downstream task, including: Determine the zero-point offset of the output channel, and determine the range of integer values ​​based on the target quantization bit width; For any weight parameter corresponding to the output channel, the weight parameter is quantized according to the optimal quantization ratio to obtain the quantization result; The quantization result is rounded down, the rounded result is summed with the zero offset, and the summation result is clipped to the range of integer values ​​to obtain the quantized integer value. The quantized integer value is multiplied by the optimal quantization ratio to obtain the quantized weight parameter of the output channel; wherein the optimal quantization ratio of each output channel is different from each other, and all output channels share the same target quantization bit width.

9. A deployment device for a downstream task processing model, characterized in that, The device includes: The data acquisition module is used to acquire the weight parameters of the initial model to be deployed to the target edge device, the task dataset corresponding to the downstream task, and the computing power limit of the target edge device; the weight parameters are distributed in the weight matrix of the linear layer in the initial model; The gradient information acquisition module is used to input the task dataset into the initial model, perform forward propagation and backward propagation under the condition of freezing the model parameters, and acquire the gradient information of each weight parameter relative to the task loss of the downstream task; the gradient information is used to characterize the importance of the corresponding weight parameter to the downstream task. A comprehensive sensitivity construction module is used to construct a comprehensive sensitivity of any weight parameter based on its gradient information, quantization error estimation, and amplitude information. The comprehensive sensitivity is used to evaluate the sensitivity contribution of the weight parameter to the task loss of the downstream task. The amplitude information is used to characterize the structural stability of the weight parameter itself. The quantization error estimation is used to characterize the inherent vulnerability of the weight parameter to quantization noise. The importance score calculation module is used to calculate the importance score of any output channel in the weight matrix based on the comprehensive sensitivity of all weight parameters corresponding to the output channel. The importance score is used to characterize the sensitivity contribution of the output channel to the task loss of the downstream task. The row index of the weight matrix corresponds to the output channel, and the column index corresponds to the input channel. The optimal quantization ratio calculation module is used to determine the target quantization bit width based on the upper limit of computing power, and to calculate the optimal quantization ratio based on the importance score of the output channel; the target quantization bit width is the uniform bit width supported by the target edge device; The model deployment module is used to quantize all weight parameters corresponding to the output channel in the initial model according to the target quantization bit width and the optimal quantization ratio to obtain a task processing model for the downstream task, and deploy the task processing model on the target edge device; the runtime computing power requirement of the task processing model is less than the computing power upper limit.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 8.

Images