Model quantization method and device, quantized model, and hardware platform

Abstract

Embodiments of the present application provide a model quantization method and device, a quantized model and a hardware platform. The method comprises: in an initial model iteration quantization process, performing global search on candidate models according to a preset search strategy, and flipping the quantization state of a plurality of target operators to generate new candidate models; combining constraint information supported by the hardware platform on operator categories and data types, and performing feasibility checking on the new candidate models based on an evaluation cost index to filter out non-deployable schemes; and under the condition of meeting an accuracy threshold, continuously optimizing the candidate models in the direction of model deployment effect. The method aims to balance model accuracy and model deployment effect.

Description

Technical Field

[0001] This application relates to the field of model quantization, and more particularly to a model quantization method, apparatus, quantization model, and hardware platform. Background Technology

[0002] When neural network models are deployed on edge computing devices, mixed precision quantization is typically used to reduce storage bandwidth usage and inference computation overhead, and intermediate representations and compilation tools are combined to complete model conversion.

[0003] However, existing quantization schemes often struggle to adapt to the operator constraints and data type limitations of different hardware in FP8 scenarios, and the searched strategies may not be directly deployable. Furthermore, their evaluations typically focus on a single metric, making it difficult to consider factors such as accuracy, bandwidth, computation, and on-chip storage. Additionally, the quantization parameters in the intermediate representation are prone to deviation from the actual numerical behavior of the hardware, leading to a decrease in accuracy and stability after deployment.

[0004] Existing technologies struggle to balance hardware constraints and accuracy-efficiency during the quantization search process, resulting in poor adaptability, significant accuracy loss, or excessive resource consumption in actual hardware deployments. Summary of the Invention

[0005] This application provides a model quantization method, apparatus, quantization model, and hardware platform to balance model accuracy and model deployment effectiveness.

[0006] In a first aspect, embodiments of this application provide a model quantization method, including: flipping the quantization state of multiple target operators in a first candidate model to obtain a second candidate model, wherein the target operators are selected by quantization search of the first candidate model based on a preset quantization search strategy, and the first candidate model and the second candidate model are obtained by iterative quantization of an initial model to be quantized;

[0007] Based on hardware constraint information and the evaluation cost index of the second candidate model, the feasibility of the second candidate model is verified. The hardware constraint information is used to indicate the constraints of the hardware platform of the model to be deployed on the operator category. The feasibility verification is used to filter out candidate models that are not executable. The evaluation cost index includes: cosine similarity, resource dimension index and coverage index.

[0008] If the feasibility verification is passed, and the cosine similarity of the first candidate model reaches a preset accuracy threshold, then the first candidate model is deployed and optimized based on the resource dimension indicators and / or coverage indicators of the second candidate model and the first candidate model, until the iteration round is satisfied.

[0009] In one possible implementation, the deployment optimization of the first candidate model is performed by comparing the resource dimension metrics and / or coverage metrics of the second candidate model with those of the first candidate model, including:

[0010] If the cosine similarity of the second candidate model is greater than or equal to the preset accuracy threshold, the deployment priority of the second candidate model is calculated based on at least one of the resource dimension index and coverage index of the second candidate model.

[0011] If the deployment priority of the second candidate model is greater than or equal to the deployment priority of the first candidate model, then the second candidate model replaces the first candidate model for subsequent iterative quantization.

[0012] If the deployment priority of the second candidate model is lower than that of the first candidate model, the first candidate model is kept unchanged for subsequent iterative quantization. The deployment optimization allows the cosine similarity of the second candidate model to be lower than that of the first candidate model.

[0013] And / or, the method further includes:

[0014] If the cosine similarity of the second candidate model is less than the preset accuracy threshold, the first candidate model remains unchanged.

[0015] And / or, before flipping the quantization states of multiple target operators in the first candidate model to obtain the second candidate model, the method further includes:

[0016] Based on the acceptance rate of the second candidate model within a preset time window, a cooling factor for adjusting the temperature value is dynamically determined, wherein the acceptance rate is used to indicate the probability of replacing the first candidate model with the second candidate model within the preset time window;

[0017] Based on the cooling factor of the second candidate model, the temperature value of the current iteration is determined, and the temperature value decreases with each iteration.

[0018] According to the search quantity corresponding to the temperature value, based on the preset quantization search strategy, each operator in the first candidate model is quantized and searched to determine multiple target operators, and the temperature value is positively correlated with the search quantity;

[0019] And / or, if the feasibility verification passes, and the cosine similarity of the first candidate model does not reach a preset accuracy threshold, the method further includes:

[0020] Compare the cosine similarity between the second candidate model and the first candidate model;

[0021] When the cosine similarity of the second candidate model is greater than or equal to the cosine similarity of the first candidate model, the second candidate model replaces the first candidate model for subsequent iterative quantization.

[0022] When the cosine similarity of the second candidate model is less than that of the first candidate model, the first candidate model remains unchanged until the cosine similarity of the first candidate model reaches the preset accuracy threshold.

[0023] In one possible implementation, the feasibility verification includes: hardware verification and performance verification. The feasibility verification of the second candidate model based on hardware constraint information and the evaluation cost index of the second candidate model includes:

[0024] Compare each of the target operators with the hardware constraint information;

[0025] If all the target operators satisfy the hardware constraint information, the second candidate model is determined to pass the hardware verification.

[0026] The cosine similarity, resource dimension index, and coverage index of the second candidate model are weighted to obtain the comprehensive cost of the second candidate model;

[0027] If the overall cost of the second candidate model is less than or equal to the overall cost of the first candidate model, the performance verification is deemed successful.

[0028] In one possible implementation, the weighting of the cosine similarity, resource dimension index, and coverage index of the second candidate model to obtain the comprehensive cost of the second candidate model includes:

[0029] Based on the temperature value of the current iteration and the preset mapping relationship, the cost conversion coefficient of the cosine similarity penalty is determined, and the cost conversion coefficients of the resource dimension index and / or coverage index are adjusted accordingly.

[0030] Based on the cost conversion coefficients adjusted from each of the aforementioned evaluation cost indicators, the evaluation cost indicators of the second candidate model are weighted to obtain the comprehensive cost of the second candidate model.

[0031] And / or, if the overall cost of the second candidate model is greater than the overall cost of the first candidate model;

[0032] The performance verification is determined based on the obtained random factor.

[0033] In one possible implementation, the method further includes:

[0034] The first candidate model corresponding to the termination iteration round is compiled in multiple stages and then deployed on the hardware platform.

[0035] And / or, when the current iteration is the initial iteration, before performing quantization search on each operator in the first candidate model based on a preset quantization search strategy, and flipping the quantization states of the selected target operators to obtain the second candidate model, the method further includes:

[0036] The precision of the quantizable operators in the initial model is quantized into a preset format to obtain a fully quantized model.

[0037] Model inference is performed on the fully quantized model and the initial model using calibration samples from the pre-stored calibration dataset.

[0038] Calculate the cosine similarity between the model output of the fully quantized model and the model output of the initial model;

[0039] Compare the cosine similarity with the preset precision threshold;

[0040] When the cosine similarity reaches the preset accuracy threshold, the fully quantized model is compiled in multiple stages and then deployed on the hardware platform.

[0041] In one possible implementation, the first candidate model or the fully quantized model is compiled at multiple levels and then deployed on the hardware platform, including:

[0042] Using a programmable precision description language, the quantization parameters of each operator of the fully quantized model or the first candidate model corresponding to the termination iteration round are mapped step by step to a hardware numerical format, and the quantization parameters of the fully quantized model or the first candidate model operator mapped to the hardware numerical format are corrected by a hardware simulator.

[0043] The modified full-quantization model or the first candidate model is deployed on the hardware platform.

[0044] In one possible implementation, the hardware numerical format includes: a first-level intermediate representation, a second-level intermediate representation, and a third-level intermediate representation. The step of mapping the quantization parameters of each operator of the fully quantized model or the first candidate model to the hardware numerical format level by level using a programmable precision description language, and correcting the quantization parameters of the operators of the fully quantized model or the first candidate model mapped to the hardware numerical format using a hardware simulator, includes:

[0045] The quantization parameters of the fully quantized model or the first candidate model operator are mapped to a first-level intermediate representation using a programmable precision description language.

[0046] The first-level intermediate representation is input into a preset hardware adaptation rule base, and a second-level intermediate representation is generated through rule-driven hardware adaptation transformation.

[0047] The secondary intermediate representation is loaded into a pre-built hardware simulator for simulation verification to obtain the simulation verification results;

[0048] Based on the simulation verification results, the quantization parameters of the fully quantized model or the first candidate model operator are corrected to obtain the three-level intermediate representation.

[0049] Secondly, embodiments of this application provide a model quantization device, including: an iterative quantization module, used to flip the quantization state of multiple target operators in a first candidate model to obtain a second candidate model, wherein the target operators are selected by quantization search of the first candidate model based on a preset quantization search strategy, and the first candidate model and the second candidate model are obtained by iterative quantization of an initial model to be quantized;

[0050] The verification module is used to perform feasibility verification on the second candidate model based on hardware constraint information and the evaluation cost index of the second candidate model. The hardware constraint information is used to indicate the constraints of the hardware platform of the model to be deployed on the operator category. The feasibility verification is used to filter out candidate models that are not executable. The evaluation cost index includes: cosine similarity, resource dimension index and coverage index.

[0051] The decision module is used to optimize the deployment of the first candidate model by comparing the resource dimension index and / or coverage index of the second candidate model with the first candidate model if the cosine similarity of the first candidate model reaches a preset accuracy threshold after the feasibility verification is passed, until the iteration round is satisfied.

[0052] Thirdly, embodiments of this application provide a quantization model, which is obtained by quantizing an initial model based on the first aspect and / or various possible implementations of the first aspect.

[0053] Fourthly, embodiments of this application provide a hardware platform, wherein the memory stores computer-executable instructions, and a quantization model is deployed on the hardware platform;

[0054] The processor executes computer execution instructions stored in the memory to load the quantization model and perform inference calculations on the input data to obtain inference results. The quantization model is obtained by quantizing the initial model based on the first aspect and / or various possible implementations of the first aspect.

[0055] Fifthly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0056] The memory stores computer-executed instructions;

[0057] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0058] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0059] In a seventh aspect, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0060] The model quantization method, apparatus, quantization model, and hardware platform provided in this application, by performing joint flipping on multiple target operators on a first candidate model, enable the search process to adjust the quantization configuration across the entire model range, thereby gradually approaching the quantization configuration that meets deployment requirements. By jointly introducing hardware constraint information and evaluation cost metrics into the candidate model screening process, quantization configurations that cannot be executed on the target hardware platform are first excluded from the perspective of deployability, and then cost judgment is performed on the remaining candidates, improving the feasibility of deploying the second candidate model and reducing repeated iterations due to subsequent compilation failures or runtime anomalies. By using a cosine similarity threshold as a switching condition for deployment optimization, the search process first reaches an acceptable lower limit of accuracy, and then continuously iterates around at least one of the resource dimension metrics and coverage metrics, allowing a controlled decrease in cosine similarity between the second candidate model and the first candidate model, thereby establishing a deployment-oriented dynamic balance between model accuracy and deployment effect, improving the deployability of the quantized model and the consistency of model output. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0062] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of this application;

[0063] Figure 2 A flowchart illustrating a model quantization method provided in this application embodiment. Figure 1 ;

[0064] Figure 3 A flowchart illustrating a model quantization method provided in this application embodiment. Figure 2 ;

[0065] Figure 4 A flowchart illustrating a model quantization method provided in this application embodiment. Figure 3 ;

[0066] Figure 5 A flowchart illustrating a model quantization method provided in this application embodiment. Figure 4 ;

[0067] Figure 6 This is a schematic diagram of the structure of a model quantization device provided in an embodiment of this application;

[0068] Figure 7 This is a schematic diagram of a hardware platform provided in an embodiment of this application.

[0069] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0070] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0071] First, let me explain the terms used in this application:

[0072] Model quantization refers to a model compression technique that reduces memory and computational requirements by lowering the precision of operators in a neural network model while controlling the model's inference accuracy.

[0073] Figure 1 This is a schematic diagram of an application scenario provided in an embodiment of this application, such as... Figure 1 As shown, in the prior art, when the trained neural network model 11 is applied to the edge computing device 12, since the edge computing device 12 requires high throughput, low power consumption and stable inference accuracy under limited bandwidth, computing resources and on-chip storage conditions, it is usually necessary to compress the neural network model 11 through quantization technology.

[0074] Existing solutions typically reduce resource consumption by assigning different data precisions to different operators in the neural network model 11. Their working method generally involves determining the quantization state of each operator at the model graph level, then evaluating the quantization results based on the estimated precision or resource overhead, thereby determining the quantization model.

[0075] However, in model quantization scenarios, different hardware platforms often have differences in their support for operator input types, data type conversions, and operator categories, which may cause quantization strategies obtained solely from the model side to fail during deployment.

[0076] Therefore, how to improve the deployability of model quantization strategies while meeting hardware constraints has become an urgent technical problem to be solved.

[0077] To address the aforementioned issues, this application provides a model quantization method. During the iterative quantization process of an initial model to form candidate models, a quantization search is performed on the first candidate model based on a preset quantization search strategy, and the quantization states of multiple target operators are flipped to obtain a second candidate model. Then, the feasibility of the second candidate model is verified by combining hardware constraint information and evaluation cost metrics to filter out unexecutable candidate models. When the feasibility verification passes and the cosine similarity of the first candidate model to the initial model reaches a preset accuracy threshold, the first candidate model is deployed and optimized based on at least one of resource dimension metrics and coverage metrics, thereby achieving a balance between accuracy and resource efficiency while ensuring deployment feasibility.

[0078] Figure 2 A flowchart illustrating a model quantization method provided in this application embodiment. Figure 1 ,like Figure 2 As shown, the method includes:

[0079] S201. Flip the quantization states of multiple target operators in the first candidate model to obtain the second candidate model.

[0080] The target operator is selected by quantizing the set of reversible operators of the first candidate model based on a preset quantization search strategy. The first and second candidate models are obtained by iteratively quantizing the initial model to be quantized. The first candidate model serves as the baseline model in the current iteration round, storing the current quantization state of each quantizable operator. The second candidate model is the model to be evaluated in the current iteration round, and its inclusion in subsequent iteration rounds is determined through subsequent verification. The second candidate model is then used as the new first candidate model.

[0081] The quantization search strategy can be a user-specified configuration parameter or dynamically determined based on preset rules, the hardware platform on which the neural network model is deployed, and the application domain of the neural network model. Quantization search strategies can include dispatching strategies, all_quant strategies, and zero strategies.

[0082] The quantization state indicates whether the operator is executed in a high-precision format or a low-precision format in the current candidate model. High-precision formats include, for example, 32-bit floating-point (FP32) and double-precision floating-point (FP64). In this embodiment, the low-precision format is 8-bit floating-point (FP8).

[0083] Flipping refers to the transformation of the quantization state of a target operator. For example, if at least two target operators are selected from the first candidate model, before flipping, one target operator is in FP32 quantization state and the other is in FP8 quantization state. Flipping the quantization states of these two target operators will result in the target operators that were originally in FP32 quantization state becoming FP8 quantization state in the formed second candidate model, and vice versa.

[0084] The initial model is a trained neural network model ready to be deployed to the target hardware platform. Quantizable operators in the model include at least convolution, fully connected layers, matrix multiplication, linear transformations in attention, or other operators. Quantization search means that the selection of target operators is not limited to a single layer, a single module, or a locally adjacent structure, but rather determines target operators across the entire model.

[0085] In this step, during the model iterative quantization process, multiple target operators are selected within the full model range of the first candidate model according to a preset quantization search strategy, and quantization state flipping is performed to generate the second candidate model.

[0086] By performing joint flipping on multiple target operators on the first candidate model, the search process can adjust the quantization configuration across the entire model range, thereby gradually approaching the quantization configuration that meets the deployment requirements.

[0087] Specifically, when the current iteration is the initial iteration, the initial model to be quantized can be used as the first candidate model for iterative quantization. Alternatively, the precision of the quantizable operators in the initial model to be quantized can be quantized to FP8 format to obtain a fully quantized model, which can then be used as the first candidate model for iterative quantization.

[0088] Optionally, during the search, the number of target operators can be a fixed value given based on the number of quantizable operators in the initial model, or it can be a dynamic value that gradually decreases with each iteration. For example, the dynamic value can decrease linearly with each iteration, or it can be dynamically determined based on the temperature value of each iteration. In the initial stage of the search (high temperature stage), the quantization state of multiple operators is flipped at once to achieve extensive exploration. As the temperature decreases, the number of flips is gradually reduced, eventually degenerating into single operator flipping for fine adjustment.

[0089] S202. Based on hardware constraint information and the evaluation cost index of the second candidate model, the feasibility of the second candidate model is verified. The evaluation cost index includes: cosine similarity, resource dimension index and coverage index.

[0090] The hardware constraint information indicates the constraints imposed on the operator categories by the hardware platform of the model to be deployed. The hardware platform can be a terminal device integrating a Neural Processing Unit (NPU), Digital Signal Processor (DSP), or similar device. Feasibility verification is used to filter out unexecutable candidate models.

[0091] The evaluation cost metrics are used to comprehensively describe the deployment quality of the second candidate model. Resource-related metrics may include at least one of the following: bandwidth cost, normalized multiply-accumulate (MAC) cost, and static random-access memory (SRAM) overflow value. Coverage metrics may be the percentage of high-precision format operators, the percentage of preset-precision format operators, or other coverage-related metrics. Bandwidth cost refers to the normalized value of memory bandwidth required for model inference. Multiply-accumulate cost refers to the amount of multiply-accumulate operations required for model inference.

[0092] Since the quantized model needs to be deployed on a hardware platform, it is necessary to verify it in conjunction with hardware constraint information after each second candidate model is determined.

[0093] In this step, it is determined whether the selected target operator meets the hardware constraint information, and the comprehensive cost of the second candidate model is calculated based on the evaluation cost index of the second candidate model. The feasibility of the second candidate model is then verified based on the comprehensive cost.

[0094] Specifically, the hardware constraints of the hardware platform are organized into constraint rules related to the operator category to obtain hardware constraint information. Based on the hardware constraint information, the deployability of the second candidate model is judged, and the second candidate model is screened in combination with the evaluation cost index. If the second candidate model does not meet the hardware platform constraints, it is marked as unexecutable and filtered out. If the second candidate model meets the hardware platform constraints, the comprehensive cost of the second candidate model is calculated according to the evaluation cost index, and the feasibility of the second candidate model is further verified based on the comprehensive cost and preset quality conditions.

[0095] For example, the preset quality pass condition could be a cost threshold, meaning that if the second candidate model satisfies the hardware platform constraints and its overall cost is lower than the cost threshold, then the second candidate model passes the feasibility check. Alternatively, the preset quality pass condition could be that the overall cost of the second candidate model does not exceed the overall cost of the first candidate model; that is, by comparing the overall costs of the first and second candidate models, if the second candidate model satisfies the hardware platform constraints and its overall cost is lower than the overall cost of the first candidate model, then the second candidate model passes the feasibility check.

[0096] Optionally, if the second candidate model meets the hardware platform constraints, the feasibility verification can be further randomly passed if the comprehensive cost does not meet the preset quality pass condition, so as to increase the randomness of model iteration quantization.

[0097] By combining hardware constraint information with evaluation cost metrics into the candidate model screening process, quantitative configurations that cannot be executed on the target hardware platform are first excluded from the perspective of deployability. Then, the cost of the second candidate model is judged, which improves the feasibility of deploying the second candidate model and reduces repeated iterations of subsequent compilation failures or runtime anomalies.

[0098] S203. If the feasibility verification is passed, and the cosine similarity between the output results of the first candidate model and the initial model reaches the preset accuracy threshold, then the first candidate model is deployed and optimized based on the resource dimension indicators and / or coverage indicators of the second candidate model and the first candidate model until the iteration round is satisfied.

[0099] In this context, deployment optimization allows the cosine similarity of the second candidate model to be lower than that of the first candidate model. Cosine similarity quantifies the degree of consistency between the output of the first or second candidate model and the output of the initial model, characterizing model accuracy. A preset accuracy threshold defines the minimum accuracy boundary. Deployment optimization allows the cosine similarity of the second candidate model to be lower than that of the first candidate model. The iteration rounds are a constraint on the number of iteration steps; these can be pre-set by the user based on the original model or be a default value.

[0100] In order to control the inference accuracy of the model after deployment on the hardware platform, the cosine similarity between the first candidate model and the initial model is used as a prerequisite for deployment optimization. That is, only when the accuracy of the first candidate model has reached the preset accuracy threshold will the quantization objective shift from optimizing the model accuracy to optimizing the deployment effect of the model.

[0101] In this step, after the cosine similarity between the output results of the first candidate model and the initial model reaches a preset accuracy threshold, the resource dimension indicators and / or coverage indicators of the second candidate model and the first candidate model are compared, and a decision is made on whether to use the second candidate model to replace the first candidate model to continue participating in subsequent iterations.

[0102] A second candidate model that outperforms the first candidate model in resource dimension metrics and / or coverage metrics may have higher or lower accuracy than the first candidate model. During deployment optimization, a lower cosine similarity between the second and first candidate models is permissible. Therefore, under the current constraints, a limited degree of accuracy regression is acceptable in exchange for lower deployment costs and improved model quantization randomness, avoiding getting trapped in local optima.

[0103] In practice, based on a pre-prepared calibration dataset, model inference is performed on the initial model, the first candidate model, and the second candidate model after feasibility verification. Model inference can be performed under the conditions of uniform input batches, uniform preprocessing procedures, and fixed random seeds to ensure that the outputs of different candidate models are comparable.

[0104] For each calibration sample, the initial model output vector y0, the first candidate model output vector y1, and the second candidate model output vector y2 are extracted. The cosine similarity CoS1 between the first candidate model and the initial model, and the cosine similarity CoS2 between the second candidate model and the initial model are calculated. When CoS1 is greater than or equal to the preset accuracy threshold, it indicates that the current first candidate model has met the accuracy admission condition, and deployment optimization is then initiated.

[0105] Specifically, before deployment optimization, based on the optimization goals of the quantification task, at least one evaluation cost indicator that needs to be optimized is determined from resource dimension indicators and coverage indicators; the optimization goal can be user input or determined based on the current compilation stage of the quantification task.

[0106] After entering the deployment optimization phase, the resource dimension indicators and / or coverage indicators of the first and second candidate models are calculated and compared. Based on the comparison results of the resource dimension indicators and / or coverage indicators between the second and first candidate models, it is determined whether to accept the second candidate model as the new first candidate model. As the iteration progresses, the search process repeats steps S201 to S203 until a preset termination iteration round is reached. Upon termination, the first candidate model corresponding to that round is determined as the output quantization model.

[0107] By using a cosine similarity threshold as a switching condition for deployment optimization, the search process first reaches an acceptable lower limit of accuracy, and then iterates continuously around at least one of the resource dimension indicators and coverage indicators. It also allows the second candidate model to exhibit a controlled decrease in cosine similarity relative to the first candidate model, thereby establishing a deployment-oriented dynamic balance between model accuracy and deployment effect.

[0108] In one possible implementation, the above-mentioned optimization of the deployment of the first candidate model by comparing the resource dimension indicators and / or coverage indicators of the second candidate model with those of the first candidate model is described in detail, including: if the cosine similarity of the second candidate model is greater than or equal to a preset accuracy threshold, calculating the deployment priority of the second candidate model based on at least one of the resource dimension indicators and coverage indicators of the second candidate model; comparing the deployment priorities of the second candidate model with those of the first candidate model; if the deployment priority of the second candidate model is greater than or equal to that of the first candidate model, then replacing the first candidate model with the second candidate model for subsequent iterative quantization; if the deployment priority of the second candidate model is less than that of the first candidate model, then keeping the first candidate model unchanged for subsequent iterative quantization.

[0109] The deployment priority is used to indicate the deployment effect after deploying the first candidate model or the second candidate model on the hardware platform. The deployment priority includes: hardware resource cost and / or model quantization effect: hardware resource cost is used to indicate the hardware resources required for model inference after the model is deployed on the hardware platform; model quantization effect refers to the proportion of preset format operators or the proportion of high-precision format operators.

[0110] In actual implementation, if the first candidate model meets the preset accuracy threshold, the resource dimension indicators and / or coverage indicators corresponding to the second candidate model are read respectively, and the deployment priority of the second candidate model is calculated based on the resource dimension indicators and / or coverage indicators.

[0111] When optimizing hardware resources based solely on any one of the resource-level metrics and / or coverage metrics, the evaluation cost metric is directly mapped to the deployment priority. When optimizing hardware resources based on multiple evaluation cost metrics among the resource-level metrics and / or coverage metrics, the deployment priority is obtained by normalizing and weighting the multiple evaluation cost metrics among the resource-level metrics and / or coverage metrics.

[0112] Compare the deployment priorities of the first candidate model and the second candidate model. If the deployment priority of the second candidate model is greater than or equal to that of the first candidate model, it means that the second candidate model does not reduce bandwidth cost while meeting accuracy constraints. In this case, the second candidate model is retained as the result of the current iteration and used as the input for subsequent quantization inversion. If the deployment priority of the second candidate model is lower than that of the first candidate model, it means that the first candidate model has a better bandwidth cost while meeting accuracy constraints. In this case, the first candidate model is kept unchanged and continues to participate in subsequent iterations of quantization.

[0113] After the first candidate model passes the accuracy threshold, it is selected and retained according to the deployment priority, so that the subsequent iterative quantization evolves in the direction of the optimization goal, improving the deployment effect of the model while maintaining the usable accuracy of deployment.

[0114] For example, when optimizing bandwidth cost is the optimization objective, bandwidth cost is mapped to deployment priority. The lower the bandwidth cost, the higher the deployment priority, thus prioritizing candidate models with low bandwidth cost to enter the subsequent stages. While maintaining the accuracy of deployment, the hardware resource overhead of the model is reduced, improving the model's execution adaptability and resource utilization efficiency on the hardware platform.

[0115] For example, when optimizing the proportion of preset format operators is the optimization goal, the proportion of preset format operators is mapped to the deployment priority. The higher the proportion of preset format operators, the higher the deployment priority. Thus, candidate models with a high proportion of preset format operators are selected first to enter the subsequent stage. While maintaining the usable accuracy of deployment, the coverage of quantization operators and the deployment effect of the model are improved.

[0116] For example, when performing joint optimization with the optimization goals of optimizing bandwidth cost and the proportion of preset format operators, it is necessary to normalize and weight the bandwidth cost and the proportion of preset format operators, and then normalize the result of the weighted sum to obtain the deployment priority.

[0117] Optionally, the method further includes: keeping the first candidate model unchanged if the cosine similarity of the second candidate model is less than a preset accuracy threshold.

[0118] Specifically, the model output of the second candidate model is obtained, and the cosine similarity between this output and the initial model output is calculated. When the calculated cosine similarity is less than the preset accuracy threshold, it indicates that the second candidate model has not yet reached the accuracy requirement of the current stage. Therefore, the first candidate model is maintained as the benchmark model for subsequent iterations of quantization to avoid the model accuracy falling below the preset accuracy threshold, which would require re-optimization of accuracy.

[0119] By adopting this approach, the second candidate model will not overwrite the first candidate model when the accuracy is insufficient. This ensures that the quantization iteration always revolves around the model that meets the preset accuracy threshold, thus ensuring that the selection result of the first candidate model is consistent with the preset accuracy threshold and improving the stability of subsequent feasibility verification and deployment optimization.

[0120] This application provides a model quantization method. By performing joint flipping on multiple target operators on a first candidate model, the search process can adjust the quantization configuration across the entire model range, thereby gradually approaching the quantization configuration that meets deployment requirements. By jointly introducing hardware constraint information and evaluation cost metrics into the candidate model screening process, quantization configurations that cannot be executed on the target hardware platform are first excluded from the perspective of deployability. Then, cost judgment is performed on the second candidate model, improving the feasibility of deploying the second candidate model and reducing repeated iterations due to subsequent compilation failures or runtime anomalies. By using a cosine similarity threshold as a switching condition for deployment optimization, the search process first reaches an acceptable lower limit of accuracy, and then continuously iterates around at least one of the resource dimension metrics and coverage metrics. It also allows for a controlled decrease in cosine similarity between the second candidate model and the first candidate model, thereby establishing a deployment-oriented dynamic balance between model accuracy and deployment effect, improving the deployability of the quantized model and the consistency of model output.

[0121] Existing technologies are insufficient to accurately reflect the quality of hardware deployment during the search process. Figure 3 A flowchart illustrating a model quantization method provided in this application embodiment. Figure 2 In this embodiment, feasibility verification includes: hardware verification and performance verification. Figure 2 Based on the previous examples, the feasibility verification of the second candidate model is further explained in detail, including the evaluation cost index based on hardware constraint information and the second candidate model. Figure 3 As shown, the method includes:

[0122] S301. Compare each target operator with the hardware constraint information; if each target operator satisfies the hardware constraint information, determine that the second candidate model passes the hardware verification.

[0123] The hardware constraints include: operators that do not support FP8 inputs, operators that do not support dtype conversion, operators whose outputs must remain FP8, operators whose inputs are prohibited from being converted from FP16, and operators that require all inputs to have the same dtype.

[0124] Specifically, the category, input / output precision, and quantization state of each target operator are compared one by one with the hardware constraint information. If any target operator does not meet the hardware constraints—for example, if the operator type is not in the hardware support list, or if the input data type of the operator is inconsistent with the hardware requirements—the hardware verification is deemed to have failed. If all target operators meet the hardware constraint information, the hardware verification is deemed to have passed.

[0125] Optionally, if hardware verification fails, the target operator and category that caused the hardware verification to fail are recorded, and it is ensured that no new second candidate model is generated based on the target operator and category in the future.

[0126] S302. Weight the cosine similarity, resource dimension index and coverage index of the second candidate model to obtain the comprehensive cost of the second candidate model.

[0127] The overall cost is used to quantify the penalty for hardware adaptability and accuracy of the first and second candidate models.

[0128] In this step, the benefit-type indicators among cosine similarity, resource dimension indicators, and coverage indicators are converted into cost-type indicators, and then a weighted sum is performed based on the cost conversion coefficients of each cost evaluation indicator to obtain the comprehensive cost of the second candidate model.

[0129] For example, resource-level metrics could be normalized bandwidth cost, normalized MAC cost, and SRAM overflow, while coverage metrics could be the proportion of high-precision format operators—all cost-based metrics. Cosine similarity, on the other hand, is a benefit-based metric. When calculating the overall cost of the first or second candidate model, the cosine similarity needs to be converted into a cost-based metric to determine the overall cost. Specifically, the weighted sum of normalized bandwidth cost, normalized MAC cost, SRAM overflow, the proportion of high-precision format operators, and the cosine similarity (minus 1) is used to obtain the overall cost of the second candidate model.

[0130] Optionally, the overall cost of the first candidate model can be calculated using the same weighting rule on its corresponding evaluation cost metric to ensure consistency in the comparison benchmark. The cost conversion coefficient can also be dynamically adjusted with each iteration. For example, as the number of iterations increases, the cost conversion coefficient for cosine similarity can be reduced, and the cost conversion coefficients for resource dimension metrics and coverage metrics can be adjusted accordingly.

[0131] S303. If the overall cost of the second candidate model is less than or equal to the overall cost of the first candidate model, the performance verification is deemed successful.

[0132] When the synthesis cost of the second candidate model is less than or equal to the synthesis cost of the first candidate model, it indicates that the second candidate model satisfies the hardware constraints without degrading the synthesis performance, and thus the performance verification is passed.

[0133] This application provides a model quantization method that compares each target operator with hardware constraint information. If all target operators satisfy the hardware constraints, the hardware verification is deemed successful. The method then weights the cosine similarity, resource dimension index, and coverage index of the second candidate model to obtain its comprehensive cost. If the comprehensive cost of the second candidate model is less than or equal to the comprehensive cost of the first candidate model, the performance verification is deemed successful. This dual-judgment mechanism establishes a criterion between operator executability and overall model performance, allowing the second candidate model to undergo performance screening only under the premise of hardware compatibility. This reduces the probability of undeployable candidates entering the evaluation process and ensures that the retained second candidate models maintain comprehensive performance comparable to or better than the first candidate model, avoiding the limitations of traditional single-objective optimization.

[0134] Based on the aforementioned embodiments, the cosine similarity, resource dimension index, and coverage index of the second candidate model are further weighted to obtain the comprehensive cost of the second candidate model. This includes: determining the cost conversion coefficient of the cosine similarity penalty based on the temperature value of the current iteration round and the preset mapping relationship, and adjusting the cost conversion coefficients of the resource dimension index and / or coverage index accordingly; and weighting the evaluation cost indexes of the second candidate model based on the adjusted cost conversion coefficients of each evaluation cost index to obtain the comprehensive cost of the second candidate model.

[0135] The mapping relationship can be determined by a lookup table or a piecewise function.

[0136] In practical implementation, after hardware verification, the cosine similarity cost conversion coefficient is determined based on the temperature value of the current iteration and a preset mapping relationship, so that different weight allocations correspond to different iteration stages. Along with the change in cosine similarity weight, the cost conversion coefficients of other evaluation cost indicators are simultaneously increased or decreased to maintain the preset constraints on the total weight of each indicator. Then, the adjusted cost conversion coefficients are used to weight each evaluation cost indicator to obtain the comprehensive cost of the second candidate model.

[0137] Optionally, when comparing the overall costs, the overall cost of the first candidate model can be calculated based on the cost conversion coefficients determined in the previous iteration rounds, or the overall cost of the first candidate model can be updated in the current iteration round according to the newly determined cost conversion coefficients to reflect the overall requirements of the current iteration stage on maintaining accuracy and consuming resources.

[0138] By dynamically adjusting the cost conversion coefficient of cosine similarity, a dynamic balance between accuracy preservation and resource consumption is achieved, and a stable basis for comparison is provided for subsequent candidate model selection.

[0139] In one possible implementation, before flipping the quantization states of multiple target operators in the first candidate model to obtain the second candidate model, the method further includes: dynamically determining a cooling factor for adjusting the temperature value based on the acceptance rate of the second candidate model within a preset time window; determining the temperature value of the current iteration based on the cooling factor of the second candidate model; and performing a quantization search on each operator in the first candidate model according to the search quantity corresponding to the temperature value, based on a preset quantization search strategy, to determine multiple target operators.

[0140] The acceptance rate indicates the probability of replacing the first candidate model with the second candidate model within a preset time window. The temperature value decreases with each iteration and is positively correlated with the number of searches. A cooling factor is used to dynamically adjust the temperature value. The temperature value is used to determine the number of target operators to search.

[0141] After each round of comparison between the first and second candidate models, the number of times the second candidate model is accepted within a preset time window is recorded, and the acceptance rate is calculated accordingly. Based on the acceptance rate, a cooling factor is determined for dynamically adjusting the temperature value. Subsequently, the temperature value is calculated based on the cooling factor and the current iteration round. Temperature updates can employ exponential decay or piecewise decay to ensure the temperature value maintains a downward trend throughout the iteration process. Then, based on the positive correlation between temperature value and the number of searches, the number of target operators to be quantized in the next iteration is determined. In subsequent iterations of quantization, a preset quantization search strategy is used to traverse, sample, and combine the operators in the first candidate model to generate a new second candidate model.

[0142] When the acceptance rate is high within the preset time window, it indicates that the current search direction is more likely to be accepted, so the cooling rate is reduced to make the temperature drop more slowly; when the acceptance rate is low, the cooling rate is increased to keep the number of searches at a low level in order to enhance the local convergence of the operators in the first candidate model.

[0143] Based on the number of searches corresponding to the temperature values, the operator is subjected to one or more global combination searches, and a new second candidate model is output for subsequent verification and substitution judgment.

[0144] By dynamically adjusting the cooling factor based on the acceptance rate, the temperature value and the number of searches are determined, thus realizing the adaptive change of the quantization search intensity with the acceptance of the second candidate model. In the iterative quantization process, the search range and update efficiency are taken into account, and the operator search of the first candidate model is made more in line with the current search state.

[0145] Based on the foregoing embodiments, the method further includes:

[0146] If the overall cost of the second candidate model is greater than that of the first candidate model, the performance verification is determined based on the obtained random factor.

[0147] In practical implementation, when the comparison result of the comprehensive cost shows that the comprehensive cost of the second candidate model is greater than that of the first candidate model, the performance verification is not directly judged as failing. Instead, a random factor is read to randomly determine whether the performance verification passes or fails. If it is judged as passing, the second candidate model is retained and continues to be used for subsequent iterations of quantization; if it is judged as failing, the first candidate model is maintained as the current retained model, and the second candidate model is not replaced.

[0148] Through the above processing, when the overall cost of the second candidate model is higher than that of the first candidate model, the performance verification no longer relies entirely on the deterministic threshold judgment. Instead, a random factor is introduced for supplementary judgment, which makes the candidate model selection process more flexible. While maintaining controllability, it improves the flexibility of retaining the first candidate model and iterative exploration, and makes it easier for quantization search to cover model states with potential deployment value. This improves the randomness of model quantization and avoids local optima.

[0149] Based on the aforementioned embodiments, further, if the feasibility verification passes and the cosine similarity between the output results of the first candidate model and the initial model does not reach the preset accuracy threshold, the method further includes: comparing the cosine similarity between the second candidate model and the first candidate model; if the cosine similarity between the second candidate model and the first candidate model is greater than or equal to the cosine similarity between the second candidate model and the first candidate model, replacing the first candidate model with the second candidate model for subsequent iterative quantization; if the cosine similarity between the second candidate model and the first candidate model is less than the cosine similarity between the second candidate model and the first candidate model, keeping the first candidate model unchanged until the cosine similarity between the first candidate model and the preset accuracy threshold is reached.

[0150] In practice, based on a pre-prepared calibration dataset, model inference is performed on the initial model, the first candidate model, and the second candidate model after feasibility verification. The cosine similarity between the first and second candidate models and the initial model is calculated. If the cosine similarity of the second candidate model is greater than or equal to that of the first candidate model, the second candidate model is retained as the new first candidate model and continues to be used as the benchmark in subsequent iterations. If the cosine similarity of the second candidate model is less than that of the first candidate model, the current first candidate model remains unchanged.

[0151] By directly measuring the consistency between the output of the first and second candidate models and the initial model using cosine similarity, the current first model is continuously replaced by the second candidate model that is closer to the threshold requirement, thereby driving the quantization state to converge toward higher precision and providing a candidate basis that meets the precision constraints for subsequent deployment and optimization.

[0152] In one possible implementation, the method further includes: compiling the first candidate model corresponding to the terminating iteration round in multiple stages and then deploying it on the hardware platform.

[0153] In practical implementation, after reaching a certain iteration round, the first candidate model corresponding to the terminating iteration round undergoes multi-level compilation, enabling the compiled first candidate model to be loaded and run on the hardware platform. The compiled first candidate model is then distributed to the hardware platform, where the hardware platform's runtime environment handles loading, memory allocation, and inference calls, thus putting the first candidate model into actual deployment.

[0154] After multi-level compilation, the first candidate model is deployed on the hardware platform to form a deployable model, reducing the adaptation deviation during the model conversion process.

[0155] Optionally, when the current iteration is the initial iteration, the first candidate model is the initial model to be quantized. Before performing quantization search on each operator in the first candidate model based on a preset quantization search strategy and flipping the quantization state of multiple target operators selected in the search to obtain the second candidate model, the method further includes: quantizing the precision of the quantizable operators in the initial model into a preset format to obtain a fully quantized model; performing model inference on the fully quantized model and the initial model respectively using calibration samples in a pre-stored calibration dataset; calculating the cosine similarity between the model output of the fully quantized model and the model output of the initial model; comparing the cosine similarity with a preset precision threshold; and deploying the fully quantized model on a hardware platform after multi-level compilation when the cosine similarity reaches the preset precision threshold.

[0156] Among them, quantizable operators are uniformly marked before iterative quantization, and the preset format can be, for example, FP8 format, NVFP4 format, etc.

[0157] In this step, the precision of quantizable operators in the initial model is quantized to a preset format to generate a fully quantized model. Calibration samples are fed into both the fully quantized model and the initial model for model inference, yielding their outputs and calculating cosine similarity. The cosine similarity of the outputs of the fully quantized model and the initial model is compared with a preset precision threshold. If the cosine similarity of the outputs of the fully quantized model and the initial model is greater than or equal to the preset precision threshold, the fully quantized model is directly compiled at multiple levels and deployed on the hardware platform.

[0158] Furthermore, if the cosine similarity between the output results of the fully quantized model and the initial model is less than a preset accuracy threshold, then the initial model is iteratively quantized.

[0159] In the initial iteration phase, a full quantization pre-check is performed first, so that the full quantization model that meets the accuracy requirements can directly enter the compilation and deployment chain, skipping iterative quantization and reducing the consumption of computing resources.

[0160] Figure 4 A flowchart illustrating a model quantization method provided in this application embodiment. Figure 3 This embodiment provides a detailed explanation of the multi-level compilation of the model and its deployment on the hardware platform, such as... Figure 4 As shown, the method includes:

[0161] S401. Using a programmable precision description language, the quantization parameters of the first candidate model operator corresponding to the fully quantized model or the terminating iteration round are mapped to a first-level intermediate representation.

[0162] Read the quantization parameters of each operator in the fully quantized model or the first candidate model, and generate a first-level intermediate representation adapted to the hardware platform based on the programmable precision description language.

[0163] S402. Input the first-level intermediate representation into the preset hardware adaptation rule library, and generate the second-level intermediate representation through rule-driven hardware adaptation transformation.

[0164] The intermediate representation is mapped step by step to a hardware-recognizable data bit width, exponent range, and mantissa encoding format to generate a second-level intermediate representation.

[0165] S403. Load the secondary intermediate representation into a pre-built hardware simulator for simulation verification to obtain the simulation verification results.

[0166] The secondary intermediate representation is loaded into a pre-built hardware simulator, and the input, output, and intermediate accumulation results are simulated and verified to obtain the simulation verification results.

[0167] S404. Based on the simulation verification results, correct the quantization parameters of the fully quantized model or the first candidate model operator to obtain a three-level intermediate representation.

[0168] Based on the simulation verification results, the quantization parameters of the first candidate model operator are corrected to obtain a three-level intermediate representation that is consistent with the actual execution behavior of the hardware.

[0169] Through the above multi-level compilation and simulation correction, the consistency of numerical behavior between the model and the hardware platform is ensured, the execution deviation caused by format differences is reduced, and the stability of the finally deployed model on the hardware platform is improved.

[0170] S405. Deploy the revised full-quantization model or the first candidate model on the hardware platform.

[0171] The modified full-quantization model or the first candidate model is loaded into the execution environment of the hardware platform and run. In actual deployment, other models of hardware simulators can also be selected, and this application does not limit this.

[0172] This application provides a model quantization method that uses a programmable precision description language to map the quantization parameters of each operator in a fully quantized model or a first candidate model to a hardware numerical format step by step. A hardware simulator is then used to correct the quantization parameters of the operators in the fully quantized model or the first candidate model mapped to the hardware numerical format. The corrected fully quantized model or the first candidate model is then deployed on a hardware platform. Through this multi-level compilation and simulation correction, the consistency of the numerical behavior between the model and the hardware platform is ensured, execution deviations caused by format differences are reduced, and the stability of the finally deployed model on the hardware platform is improved. By completing multiple rounds of format adaptation and parameter correction before deployment, the repeated adjustment costs during the deployment phase are reduced.

[0173] Figure 5 A flowchart illustrating a model quantization method provided in this application embodiment. Figure 4 ,like Figure 5 As shown, this method includes a data input phase, a full-quantization rapid pre-detection phase, an iterative quantization phase, and a deployment phase. The iterative quantization phase includes a feasibility verification phase and a two-stage acceptance decision phase.

[0174] Data Input Phase: This phase receives configuration parameters, the initial model to be quantized, and the calibration dataset. Configuration parameters include hardware constraints, maximum number of iterations, preset accuracy threshold, and quantization search algorithm. For example, it receives the YOLOv5 model to be quantized, configuration parameters, and the calibration dataset. The configuration parameters specify a dispatching strategy as the quantization search algorithm, a preset accuracy threshold of 0.95, and a maximum number of iterations of 1000.

[0175] The full-quantization rapid pre-detection stage involves quantizing the quantizable operators in the initial model into a preset format to obtain a full-quantized model. Then, using calibration samples from a pre-stored calibration dataset, model inference is performed on both the full-quantized model and the initial model. The cosine similarity between the outputs of the full-quantized model and the initial model is calculated and compared to a preset accuracy threshold. If the cosine similarity reaches the preset accuracy threshold, the full-quantized model is directly compiled in multiple stages and deployed to the hardware platform. If the cosine similarity does not reach the preset accuracy threshold, the iterative quantization stage begins.

[0176] Iterative quantization phase:

[0177] After entering the iterative quantification stage, the first stage is the feasibility verification stage. After the feasibility verification is passed, the two-stage decision-making stage begins.

[0178] In the first iteration, the number of target operators is determined based on the initial temperature value. In subsequent iterations, the acceptance rate of the second candidate model replacing the first candidate model within a preset time window is statistically analyzed to determine a cooling factor. Then, based on the cooling factor corresponding to the second candidate model, the temperature value of the current iteration round is determined. According to the search quantity corresponding to the temperature value, quantization search is performed on each operator in the first candidate model based on a preset quantization search strategy. The temperature value is positively correlated with the search quantity. The quantization states of the multiple target operators found are flipped to obtain the second candidate model. Both the first and second candidate models are candidate models obtained through continuous iteration of the initial model to be quantized.

[0179] Feasibility Verification Phase: After obtaining the second candidate model, a feasibility verification is performed based on the hardware constraint information and the cosine similarity, resource dimension indicators, and coverage indicators of the second candidate model. Hardware verification passes when all target operators satisfy the hardware constraints. After passing hardware verification, the cost conversion coefficient corresponding to the cosine similarity is determined based on the temperature value of the current iteration and the preset mapping relationship (in the first iteration, the cost conversion coefficient is given in the configuration parameters or is a default value). Simultaneously, the cost conversion coefficients of other evaluation cost indicators are adjusted. The adjusted cost conversion coefficients are then used to weight the cosine similarity, resource dimension indicators, and coverage indicators to obtain the comprehensive cost of the second candidate model. If any target operator does not satisfy the hardware constraints, hardware verification fails, and a new second candidate model is determined.

[0180] The comprehensive cost of the second candidate model is compared with that of the first candidate model. If the comprehensive cost of the second candidate model is less than or equal to that of the first candidate model, the performance verification is considered successful, and the process proceeds to the subsequent two-stage decision-making stage. If the comprehensive cost of the second candidate model is greater than that of the first candidate model, the performance verification is further determined by incorporating the obtained random factors. If the feasibility verification fails, the second candidate model is filtered out, and the first candidate model remains unchanged for the next round of search.

[0181] After the feasibility verification is passed, the two-stage decision-making phase begins.

[0182] The two-stage decision-making phase involves determining whether the cosine similarity between the output of the first candidate model and the initial model reaches a preset accuracy threshold. If the cosine similarity of the first candidate model reaches this threshold, deployment optimization is performed, allowing the cosine similarity of the second candidate model to be lower than that of the first candidate model. During deployment optimization, deployment priorities are determined based on resource dimension metrics and / or coverage metrics. The deployment priorities in the cost metrics are compared between the second and first candidate models. If the deployment priority of the second candidate model is greater than or equal to that of the first candidate model, the second candidate model replaces the first candidate model, and subsequent iterations continue. If the deployment priority of the second candidate model is less than that of the first candidate model, the first candidate model remains unchanged, and subsequent iterations continue. During the deployment optimization phase, any second candidate model whose accuracy falls below the preset accuracy threshold is rejected.

[0183] When the feasibility verification passes but the cosine similarity between the output of the first candidate model and the initial model does not reach the preset accuracy threshold, accuracy optimization is performed. By comparing the cosine similarity between the second candidate model and the first candidate model, it is determined whether to replace the first candidate model with the second candidate model. In subsequent iterations, the first candidate model is continuously updated according to the process of searching, flipping the quantization state, feasibility verification and candidate model comparison, until the cosine similarity of the first candidate model reaches the preset accuracy threshold or the iteration round is terminated.

[0184] Deployment phase:

[0185] Upon reaching the termination iteration round, the first candidate model corresponding to that iteration round is compiled in multiple stages and deployed on the hardware platform. If the fully quantized model already meets the preset accuracy threshold in the initial iteration stage, the fully quantized model that meets the conditions can also be compiled and deployed in the same way. During multi-stage compilation, the quantization parameters of each operator in the first candidate model or the fully quantized model that meets the conditions are mapped to the hardware numerical format level by level through a programmable precision description language. The hardware numerical format includes a first-level intermediate representation, a second-level intermediate representation, and a third-level intermediate representation.

[0186] In this example, the first-level intermediate representation is `iq.json`, which records the mapping between quantization parameters and operator types. This first-level intermediate representation is then input into a pre-defined hardware adaptation rule library, and a second-level intermediate representation `hw.json` is generated through rule-driven hardware adaptation transformation. Subsequently, the second-level intermediate representation `hw.json` is loaded into a pre-built hardware simulator for simulation verification, and the simulation results are fused to generate the third-level intermediate representation `sv.json`. The SVLog is parsed and injected back into the model description. Based on the simulation verification results, the quantization parameters of each operator in the model are corrected to obtain the third-level intermediate representation `sv.json`. Finally, the corrected model is deployed on the hardware platform.

[0187] Figure 6 This is a schematic diagram of the structure of a model quantization device provided in an embodiment of this application, as shown below. Figure 6 As shown, the model quantization device 60 provided in this embodiment includes: an iterative quantization module 601, a verification module 602, and a decision module 603;

[0188] Among them, the iterative quantization module 601 is used to flip the quantization state of multiple target operators in the first candidate model to obtain the second candidate model. The target operators are selected by quantization search of the first candidate model based on a preset quantization search strategy. The first candidate model and the second candidate model are obtained by iterative quantization of the initial model to be quantized.

[0189] The verification module 602 is used to perform feasibility verification on the second candidate model based on hardware constraint information and the evaluation cost index of the second candidate model. The hardware constraint information is used to indicate the constraints of the hardware platform of the model to be deployed on the operator category. The feasibility verification is used to filter out candidate models that are not executable. The evaluation cost index includes: cosine similarity, resource dimension index and coverage index.

[0190] The decision module 603 is used to optimize the deployment of the first candidate model based on the resource dimension indicators and / or coverage indicators of the second candidate model and the first candidate model if the cosine similarity of the model output results of the first candidate model and the initial model reaches a preset accuracy threshold after the feasibility verification is passed. This optimization is performed until the iteration round is met. The deployment optimization allows the cosine similarity of the second candidate model to be lower than that of the first candidate model.

[0191] In one possible implementation, the decision module 603 is further configured to: calculate the deployment priority of the second candidate model based on at least one of the resource dimension indicators and coverage indicators of the second candidate model, if the cosine similarity of the second candidate model is greater than or equal to a preset accuracy threshold; compare the deployment priority of the second candidate model with that of the first candidate model; if the deployment priority of the second candidate model is greater than or equal to that of the first candidate model, then replace the first candidate model with the second candidate model for subsequent iterative quantization; if the deployment priority of the second candidate model is less than that of the first candidate model, then keep the first candidate model unchanged for subsequent iterative quantization.

[0192] The decision module 603 is also used to keep the first candidate model unchanged if the cosine similarity of the second candidate model is less than a preset accuracy threshold.

[0193] In one possible implementation, the feasibility verification includes hardware verification and performance verification. The verification module 602 is also used to compare each target operator with the hardware constraint information. If each target operator satisfies the hardware constraint information, the second candidate model is determined to pass the hardware verification. The cosine similarity, resource dimension index, and coverage index of the second candidate model are weighted to obtain the comprehensive cost of the second candidate model. If the comprehensive cost of the second candidate model is less than or equal to the comprehensive cost of the first candidate model, the performance verification is determined to pass.

[0194] In one possible implementation, the verification module 602 is further configured to determine the evaluation cost index of the second candidate model; based on the temperature value of the current iteration round and the preset mapping relationship, determine the cost conversion coefficient of the cosine similarity penalty, and adjust the cost conversion coefficients of the resource dimension index and / or coverage index accordingly; based on the adjusted cost conversion coefficients of each evaluation cost index, weight the evaluation cost index of the second candidate model to obtain the comprehensive cost of the second candidate model.

[0195] In one possible implementation, the iterative quantization module 601 is further configured to dynamically determine a cooling factor for adjusting the temperature value based on the acceptance rate of the second candidate model within a preset time window, wherein the acceptance rate indicates the probability of replacing the first candidate model with the second candidate model within the preset time window; determine the temperature value of the current iteration based on the cooling factor of the second candidate model, wherein the temperature value decreases with each iteration; and perform quantization search on each operator in the first candidate model according to the search quantity corresponding to the temperature value, based on a preset quantization search strategy, to determine multiple target operators, wherein the temperature value is positively correlated with the search quantity.

[0196] In one possible implementation, the verification module 602 is further configured to determine whether the performance verification passes based on the obtained random factor if the comprehensive cost of the second candidate model is greater than the comprehensive cost of the first candidate model.

[0197] In one possible implementation, if the feasibility verification passes and the cosine similarity between the output of the first candidate model and the initial model does not reach the preset accuracy threshold, the decision module 603 is further used to compare the cosine similarity between the second candidate model and the first candidate model. If the cosine similarity between the second candidate model and the first candidate model is greater than or equal to the cosine similarity between the second candidate model and the first candidate model, the second candidate model is used to replace the first candidate model for subsequent iterative quantization.

[0198] When the cosine similarity of the second candidate model is less than that of the first candidate model, the first candidate model remains unchanged until its cosine similarity reaches a preset accuracy threshold.

[0199] In one possible implementation, the model quantization device 60 further includes: a deployment module 604; the deployment module 604 is used to deploy the first candidate model corresponding to the termination iteration round on the hardware platform after multi-level compilation;

[0200] In the case that the current iteration is the initial iteration, the verification module 602 is also used to quantize the precision of the quantizable operators in the initial model into a preset format to obtain a fully quantized model; to perform model inference on the fully quantized model and the initial model respectively using calibration samples in the pre-stored calibration dataset; to calculate the cosine similarity between the model output of the fully quantized model and the model output of the initial model; and to compare the cosine similarity with the preset precision threshold.

[0201] When the cosine similarity reaches the preset accuracy threshold, the deployment module 604 is also used to deploy the fully quantized model on the hardware platform after multi-level compilation.

[0202] In one possible implementation, the deployment module 604 is further configured to map the quantization parameters of each operator of the full quantization model or the first candidate model to a hardware numerical format step by step using a programmable precision description language, and to correct the quantization parameters of the operators of the full quantization model or the first candidate model mapped to the hardware numerical format using a hardware simulator; and to deploy the corrected full quantization model or the first candidate model on a hardware platform.

[0203] In one possible implementation, the hardware numerical format includes: a first-level intermediate representation, a second-level intermediate representation, and a third-level intermediate representation. The deployment module 604 is further used to map the quantization parameters of the fully quantized model or the first candidate model operator to the first-level intermediate representation using a programmable precision description language; input the first-level intermediate representation into a preset hardware adaptation rule base, and generate the second-level intermediate representation through rule-driven hardware adaptation transformation; load the second-level intermediate representation into a pre-built hardware simulator for simulation verification to obtain the simulation verification result; and correct the quantization parameters of the fully quantized model or the first candidate model operator based on the simulation verification result to obtain the third-level intermediate representation.

[0204] The model quantization device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0205] This application provides a quantization model, which is obtained by quantizing an initial model based on the model quantization method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0206] Figure 7 This is a schematic diagram of a hardware platform provided in an embodiment of this application. Figure 7 As shown, the hardware platform 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the hardware platform 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0207] In the specific implementation process, at least one processor 701 executes the computer execution instructions stored in the memory 702, so that at least one processor 701 executes the computer execution instructions stored in the memory 702 to load the quantization model and perform inference calculations on the input data to obtain the inference result. The quantization model is obtained by quantizing the initial model based on the model quantization method provided in the above method embodiment.

[0208] This application also provides an electronic device, including a memory and a processor. The specific implementation process of the processor of the electronic device can be found in the above method embodiments, and its implementation principle and technical effect are similar. Therefore, it will not be repeated here.

[0209] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0210] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0211] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0212] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the model quantization method described above.

[0213] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the aforementioned model quantization method.

[0214] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0215] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0216] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0217] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0218] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0219] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0220] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0221] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A model quantization method, characterized in that, include: The quantization states of multiple target operators in the first candidate model are flipped to obtain the second candidate model. The target operators are selected by quantization search of the first candidate model based on a preset quantization search strategy. The first candidate model and the second candidate model are obtained by iterative quantization of the initial model to be quantized. Based on hardware constraint information and the evaluation cost index of the second candidate model, the feasibility of the second candidate model is verified. The hardware constraint information is used to indicate the constraints of the hardware platform of the model to be deployed on the operator category. The feasibility verification is used to filter out candidate models that are not executable. The evaluation cost index includes: cosine similarity, resource dimension index and coverage index. If the feasibility verification is passed, and the cosine similarity of the first candidate model reaches a preset accuracy threshold, then the first candidate model is deployed and optimized based on the resource dimension indicators and / or coverage indicators of the second candidate model and the first candidate model, until the iteration round is satisfied.

2. The method according to claim 1, characterized in that, The step of optimizing the deployment of the first candidate model based on the resource dimension metrics and / or coverage metrics of the second candidate model and the first candidate model includes: If the cosine similarity of the second candidate model is greater than or equal to the preset accuracy threshold, the deployment priority of the second candidate model is calculated based on at least one of the resource dimension index and coverage index of the second candidate model. If the deployment priority of the second candidate model is greater than or equal to the deployment priority of the first candidate model, then the second candidate model replaces the first candidate model for subsequent iterative quantization. If the deployment priority of the second candidate model is lower than that of the first candidate model, the first candidate model is kept unchanged for subsequent iterative quantization. The deployment optimization allows the cosine similarity of the second candidate model to be lower than that of the first candidate model. And / or, the method further includes: If the cosine similarity of the second candidate model is less than the preset accuracy threshold, the first candidate model remains unchanged. And / or, before flipping the quantization states of multiple target operators in the first candidate model to obtain the second candidate model, the method further includes: Based on the acceptance rate of the second candidate model within a preset time window, a cooling factor for adjusting the temperature value is dynamically determined, wherein the acceptance rate is used to indicate the probability of replacing the first candidate model with the second candidate model within the preset time window; Based on the cooling factor of the second candidate model, the temperature value of the current iteration is determined, and the temperature value decreases with each iteration. According to the search quantity corresponding to the temperature value, based on the preset quantization search strategy, each operator in the first candidate model is quantized and searched to determine multiple target operators, and the temperature value is positively correlated with the search quantity; And / or, if the feasibility verification passes, and the cosine similarity of the first candidate model does not reach a preset accuracy threshold, the method further includes: Compare the cosine similarity between the second candidate model and the first candidate model; When the cosine similarity of the second candidate model is greater than or equal to the cosine similarity of the first candidate model, the second candidate model replaces the first candidate model for subsequent iterative quantization. When the cosine similarity of the second candidate model is less than that of the first candidate model, the first candidate model remains unchanged until the cosine similarity of the first candidate model reaches the preset accuracy threshold.

3. The method according to claim 1, characterized in that, The feasibility verification includes hardware verification and performance verification. The feasibility verification of the second candidate model based on hardware constraint information and the evaluation cost index of the second candidate model includes: Compare each of the target operators with the hardware constraint information; If all the target operators satisfy the hardware constraint information, the second candidate model is determined to pass the hardware verification. The cosine similarity, resource dimension index, and coverage index of the second candidate model are weighted to obtain the comprehensive cost of the second candidate model; If the overall cost of the second candidate model is less than or equal to the overall cost of the first candidate model, the performance verification is deemed successful.

4. The method according to claim 3, characterized in that, The weighted average of the cosine similarity, resource dimension indicators, and coverage indicators of the second candidate model yields the comprehensive cost of the second candidate model, including: Based on the temperature value of the current iteration round and the preset mapping relationship, the cost conversion coefficient of the cosine similarity is determined, and the cost conversion coefficients of the resource dimension index and / or coverage index are adjusted accordingly. Based on the cost conversion coefficients adjusted from each of the aforementioned evaluation cost indicators, the evaluation cost indicators of the second candidate model are weighted to obtain the comprehensive cost of the second candidate model. And / or, if the overall cost of the second candidate model is greater than the overall cost of the first candidate model; The performance verification is determined based on the obtained random factor.

5. The method according to claim 1, characterized in that, The method further includes: The first candidate model corresponding to the termination iteration round is compiled in multiple stages and then deployed on the hardware platform. And / or, when the current iteration is the initial iteration, before performing quantization search on each operator in the first candidate model based on a preset quantization search strategy, and flipping the quantization states of the selected target operators to obtain the second candidate model, the method further includes: The precision of the quantizable operators in the initial model is quantized into a preset format to obtain a fully quantized model. Model inference is performed on the fully quantized model and the initial model using calibration samples from the pre-stored calibration dataset. Calculate the cosine similarity between the model output of the fully quantized model and the model output of the initial model; When the cosine similarity reaches the preset accuracy threshold, the fully quantized model is compiled in multiple stages and then deployed on the hardware platform.

6. The method according to claim 5, characterized in that, Deploying the first candidate model or the fully quantized model on the hardware platform after multi-level compilation includes: Using a programmable precision description language, the quantization parameters of each operator of the fully quantized model or the first candidate model corresponding to the termination iteration round are mapped to hardware numerical format step by step, and the quantization parameters of the fully quantized model or the first candidate model operator mapped to hardware numerical format are corrected by a hardware simulator. The modified full-quantization model or the first candidate model is deployed on the hardware platform.

7. The method according to claim 6, characterized in that, The hardware numerical format includes: a first-level intermediate representation, a second-level intermediate representation, and a third-level intermediate representation. The process involves mapping the quantization parameters of each operator in the fully quantized model or the first candidate model to the hardware numerical format level by level using a programmable precision description language, and then correcting the quantization parameters of the operators in the fully quantized model or the first candidate model mapped to the hardware numerical format using a hardware simulator. This includes: The quantization parameters of the fully quantized model or the first candidate model operator are mapped to a first-level intermediate representation using a programmable precision description language. The first-level intermediate representation is input into a preset hardware adaptation rule base, and a second-level intermediate representation is generated through rule-driven hardware adaptation transformation. The secondary intermediate representation is loaded into a pre-built hardware simulator for simulation verification to obtain the simulation verification results; Based on the simulation verification results, the quantization parameters of the fully quantized model or the first candidate model operator are corrected to obtain the three-level intermediate representation.

8. A model quantization device, characterized in that, include: The iterative quantization module is used to flip the quantization state of multiple target operators in the first candidate model to obtain the second candidate model. The target operators are selected by quantization search of the first candidate model based on a preset quantization search strategy. The first candidate model and the second candidate model are obtained by iterative quantization of the initial model to be quantized. The verification module is used to perform feasibility verification on the second candidate model based on hardware constraint information and the evaluation cost index of the second candidate model. The hardware constraint information is used to indicate the constraints of the hardware platform of the model to be deployed on the operator category. The feasibility verification is used to filter out candidate models that are not executable. The evaluation cost index includes: cosine similarity, resource dimension index and coverage index. The decision module is used to optimize the deployment of the first candidate model by comparing the resource dimension index and / or coverage index of the second candidate model with the first candidate model if the cosine similarity of the first candidate model reaches a preset accuracy threshold after the feasibility verification is passed, until the iteration round is satisfied.

9. A quantization model, characterized in that, include: The quantization model is obtained by quantizing the initial model based on the model quantization method described in any one of claims 1-7.

10. A hardware platform, characterized in that, include: Memory, processor; The memory stores computer-executed instructions, and the hardware platform is deployed with a quantization model. The processor executes the computer execution instructions stored in the memory to load the quantization model and perform inference calculations on the input data to obtain inference results. The quantization model is obtained by quantizing an initial model based on the model quantization method as described in any one of claims 1-7.

Images