Model quantization method and apparatus, electronic device, and storage medium

Abstract

The application relates to a model quantization method and device, electronic equipment and a storage medium. The method comprises the following steps: arranging modules in a neural network model in descending order according to module importance, wherein the modules are linear layers or convolutional layers; according to the module importance ranking, the complexity of each module in a quantization simulation model and an estimated quantization error are determined to determine a candidate code set of the corresponding module; according to the module importance ranking, each candidate code in each candidate code set is used to quantize the weight of the corresponding module in the quantization simulation model, and the quantization result is corrected to reduce the deviation between the quantization result and the non-quantization result of the neural network model; and the optimal code is selected from the candidate code set based on the loss value of the corrected quantization result, and the optimal code of each module in the quantization simulation model is used to quantize the corresponding module. The application can improve the model quantization precision.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and in particular to a model quantization method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of neural networks, artificial intelligence has been widely applied in many fields such as image recognition, natural language processing, and autonomous driving. However, the high computational complexity and huge storage requirements of deep neural networks make them difficult to deploy efficiently on resource-constrained edge devices such as IoT devices and mobile terminals.

[0003] To address this issue, quantization techniques have emerged. By mapping floating-point weights and activation values ​​to low-bit integer representations, quantization effectively reduces the model's storage footprint and computational overhead. Specifically, model quantization transforms the unquantized high-precision 32-bit floating-point (FP32) model into a 4-bit or 8-bit quantized simulation model. Its core objective is to reduce storage and computational costs while preserving model performance as much as possible.

[0004] However, the static encoding strategy commonly used in the current quantization process has obvious limitations: on the one hand, this strategy ignores the propagation and accumulation effect of quantization error between different layers, which can easily lead to a decrease in the overall accuracy of the model; on the other hand, the encoding selection is based only on simple statistical indicators such as minimum-maximum and histogram, which fails to fully adapt to the characteristics of each module, limits the performance of the quantization model, and results in low quantization accuracy of the modules. Summary of the Invention

[0005] This application provides a model quantization method, apparatus, electronic device, and storage medium to address the problem of low module quantization accuracy.

[0006] Firstly, this application provides a model quantization method, the method comprising:

[0007] The modules in the neural network model are arranged in descending order of importance, wherein each module is a linear layer or a convolutional layer;

[0008] Based on the importance of the modules, the candidate encoding set of each module in the quantization simulation model is determined one by one according to the complexity of each module and the estimated quantization error.

[0009] According to the importance of the modules, the weights of the corresponding modules in the quantization simulation model are quantized one by one using each candidate code in each candidate code set, and the quantization results are corrected to reduce the deviation between the quantization results and the non-quantization results of the neural network model.

[0010] The optimal code is selected from the candidate code set based on the loss value of the corrected quantization result, and the corresponding module is quantized using the optimal code of each module in the quantization simulation model.

[0011] Optionally, before determining the candidate encoding set for each module in the quantization simulation model according to its complexity and estimated quantization error, based on the module importance ranking, the method further includes:

[0012] Obtain the input dataset, wherein the input dataset contains multiple data batches;

[0013] The sampling frequency of the module is determined based on the computational complexity of the module and the estimated quantization error, wherein the sampling frequency is used to indicate the number of input data batches allocated to the module;

[0014] After inputting the input dataset into the neural network model and the quantization simulation model respectively, the modules are sorted according to their importance. Then, the input activation tensors of each module in the neural network model and the quantization simulation model are collected one by one according to the number of input data batches of each module. The number of input data batches is positively correlated with the number of input activation tensors. The input activation tensors are intermediate data passed from the previous module to the current module.

[0015] Optionally, the candidate encoding set for each module is determined according to its importance, based on the complexity of each module in the quantization simulation model and the estimated quantization error.

[0016] Based on the importance of the modules, perform the following processing on each module in the quantization simulation model one by one:

[0017] The number of candidate codes for the module is determined based on the module's complexity and the estimated quantization error.

[0018] The maximum candidate code for the target channel is determined based on the maximum weight of the module in the target channel, the candidate code index, and the number of candidate codes.

[0019] The minimum candidate code for the target channel is determined based on the minimum weight of the module in the target channel, the candidate code index, and the number of candidate codes.

[0020] Within the range of the maximum and minimum candidate codes, a set of candidate codes corresponding to the number of candidate codes is generated.

[0021] Optionally, according to the importance of the modules, the weights of the corresponding modules in the quantization simulation model are quantized one by one using each candidate code from each candidate code set, and the quantization results are corrected to reduce the deviation between the quantization results and the non-quantization results of the neural network model, including:

[0022] Based on the importance of the modules, perform the following processing on each module in the quantization simulation model one by one:

[0023] The weights of the module are quantized using each candidate code in the candidate code set of the module to obtain the quantized weights;

[0024] The quantized weights are calculated based on the input activation tensor of the module to obtain the quantization result of the quantization simulation model.

[0025] The original weights of the modules are calculated based on the input activation tensors of the corresponding modules in the neural network model to obtain the non-quantized result of the neural network model.

[0026] Based on the deviation between the quantized result and the non-quantized result, the quantized result is corrected by a dynamic compensation factor to reduce the deviation.

[0027] Optionally, the calculation process of the dynamic compensation factor includes:

[0028] Determine the first variance of the non-quantized result of the neural network model, and determine the second variance of the quantized result of the quantized simulation model;

[0029] The dynamic compensation factor is determined based on the first variance, the second variance, and the preset stability parameter.

[0030] Optionally, selecting the optimal code from the candidate code set based on the loss value of the corrected quantization result includes:

[0031] Determine the loss value of the corrected quantization result for each candidate code corresponding to the module;

[0032] The minimum loss value is selected from multiple loss values, and the candidate code corresponding to the minimum loss value is taken as the optimal code.

[0033] Optionally, the modules in the neural network model can be sorted in descending order of importance, including:

[0034] Determine the loss of the neural network model on the sample dataset;

[0035] The sensitivity of each module is obtained by calculating the sensitivity of the loss to the weights of each module through backpropagation.

[0036] The importance of each module is determined based on its sensitivity, with a positive correlation between module sensitivity and module importance.

[0037] Secondly, this application provides a model quantization apparatus, the apparatus comprising:

[0038] The sorting module is used to sort the modules in the neural network model in descending order of module importance, wherein each module is a linear layer or a convolutional layer;

[0039] The module is determined by prioritizing modules according to their importance and determining the candidate encoding set for each module based on its own complexity and the estimated quantization error in the quantization simulation model.

[0040] The correction module is used to sort the modules according to their importance, quantize the weights of the corresponding modules in the quantization simulation model one by one using each candidate code in each candidate code set, and correct the quantization results to reduce the deviation between the quantization results and the non-quantization results of the neural network model.

[0041] The quantization module is used to select the optimal code from the candidate code set based on the loss value of the corrected quantization result, and to quantize the corresponding module using the optimal code of each module in the quantization simulation model.

[0042] Thirdly, this application provides an electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus.

[0043] Fourthly, this application also provides a computer storage medium storing computer-executable instructions for executing the model quantization method described in any of the preceding claims of this application.

[0044] Compared with the prior art, the technical solution provided in this application has the following advantages: First, the linear layers and convolutional layers in the neural network are arranged in descending order of module importance to determine module priority; then, according to the module order, the candidate encoding set for each module is dynamically determined based on the module's own complexity and the estimated quantization error, so as to dynamically adjust the encoding range of the module and avoid the accuracy limitation caused by a fixed range for complex modules; subsequently, according to the module order, the weights of the corresponding modules are quantized according to the multiple candidate quantization codes of each module, and the deviation between the quantization result and the original non-quantization output is corrected through dynamic error compensation, directly improving the quantization accuracy of the module; finally, the optimal encoding with the minimum loss for each module is selected to quantize the corresponding module, ensuring that each module adopts the most suitable scheme and avoiding accuracy loss caused by improper encoding. In this process, the module importance ranking realizes reasonable resource allocation, the candidate encoding set adaptively determines the encoding range, and dynamic error compensation reduces quantization error, all of which jointly improve the quantization accuracy of the model. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0048] Figure 1 A flowchart of a model quantization method provided in an embodiment of this application;

[0049] Figure 2 A complete flowchart of a model quantization method provided in this application embodiment is shown below;

[0050] Figure 3 This is a schematic diagram of the structure of a model quantization device provided in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0054] To address the issue of low model quantization accuracy mentioned in the background art, this application embodiment achieves adaptive sequential quantization of each module in the model through adaptive encoding and dynamic error compensation, thereby improving the overall quantization accuracy of the model.

[0055] The following will describe in detail a model quantization method provided in this application embodiment, taking its application to a server as an example, with specific implementation details. Figure 1 As shown, the specific steps are as follows:

[0056] Step 101: Sort the modules in the neural network model in descending order of importance, where each module is a linear layer or a convolutional layer;

[0057] Step 102: Sort by module importance, and determine the candidate code set for each module according to its own complexity and estimated quantization error in the quantization simulation model;

[0058] Step 103: Sort by module importance, quantize the weight of the corresponding module in the simulation model one by one using each candidate code in each candidate code set, and correct the quantization result to reduce the deviation between the quantization result and the non-quantization result of the neural network model;

[0059] Step 104: Select the optimal code from the candidate code set based on the loss value of the corrected quantization result, and quantize the corresponding module using the optimal code of each module in the quantization simulation model.

[0060] First, the terms involved in the embodiments of this application will be explained, including the following content.

[0061] Neural network model: refers to the original 32-bit floating-point (FP32) neural network model that has not undergone quantization processing. It includes the weight parameters of each layer and the calculation logic, and can output high-precision original results (non-quantized results). It is the reference benchmark for quantized simulation models.

[0062] Quantization simulation model: A model used to simulate the low-bit quantization process. It has the same structure as the neural network model, but the weights or activation values ​​can be converted into low-precision representations (such as integers) through quantization encoding. It is the object of quantization optimization in this algorithm.

[0063] Module: refers to the structural unit in a neural network that supports quantization, specifically a linear layer or a convolutional layer. Each module contains independent weight parameters.

[0064] Module importance: An indicator that measures the degree of influence of a module on the model output. It is calculated based on the sensitivity of the model loss to the module weights (the higher the sensitivity, the higher the importance) and determines the optimization priority of the module.

[0065] Candidate encoding set: A set of multiple quantization encoding schemes generated for each module. The encoding is used to define the mapping rules of weights from high precision to low precision (such as the maximum / minimum candidate encoding range). The number of encodings is dynamically adjusted according to the importance of the module and the quantization error.

[0066] Weight quantization: The process of converting the high-precision weights of a module (such as FP32) into a low-precision representation (such as low-bit integers). The mapping is achieved through candidate encoding, with the aim of reducing model storage and computational overhead.

[0067] Quantization results: The output results (low precision) obtained by the quantization simulation model using the quantized weights need to be compared with the non-quantization results of the neural network model to evaluate the error.

[0068] Non-quantized results: The output (FP32) calculated using the original high-precision weights of the neural network model is used as a benchmark to measure the error of the quantized results.

[0069] Correcting quantization results: The quantization results are adjusted through a dynamic error compensation mechanism (such as adding compensation terms) to make the quantization results closer to the non-quantization results, thereby reducing quantization errors.

[0070] Optimal coding: The coding scheme selected from the candidate coding set that minimizes the loss value of the corrected quantization result, which is ultimately used for the weight quantization of the module.

[0071] In step 101, firstly, a 32-bit floating-point precision neural network model and its corresponding quantization simulation model are input. The system identifies the modules in the neural network model that support quantization, specifically linear layers and convolutional layers. To be compatible with existing quantization configurations, input or output activation quantizers, as well as weight quantizers of non-supported modules or user-specified excluded modules, are temporarily disabled, while weight quantizers belonging to supported modules and not excluded by the user are in their initial state.

[0072] Subsequently, the weight quantizer initializes the weight quantization encoding for each supporting module, capturing the minimum and maximum value ranges of the weights. Simultaneously, to improve quantization accuracy, the system introduces a module importance evaluation mechanism, sorting modules in descending order of importance, prioritizing optimization of layers that have a greater impact on the model output.

[0073] Different modules in a neural network have varying impacts on the overall model performance. Even small changes in the weights of highly important modules can significantly affect the model's output. Prioritizing these modules can better preserve model accuracy with limited resources, clarifying optimization priorities for subsequent quantization steps and avoiding resource waste. For example, in a neural network for text classification, the convolutional layer responsible for extracting key semantic features is more important than subsequent linear layers; therefore, this convolutional layer will be processed first after ranking.

[0074] In step 102, based on the module importance ranking obtained in step 101, the system processes each module in the quantization simulation model one by one. For the current module, its computational complexity and estimated quantization error are analyzed first, and the encoding range is dynamically adjusted accordingly. The computational complexity is determined based on the module's floating-point operations and memory usage, while the estimated quantization error refers to the degree of precision loss that may result from module quantization.

[0075] For modules with high computational complexity, such as layers with large floating-point operations and large memory consumption, the coding coverage will be expanded to fully include the rich and subtle weight features in the module, avoiding the omission of important weight information due to a narrow range. For modules with large estimated quantization errors, such as layers with scattered weight values ​​and significant numerical differences, the coding coverage will also be expanded to include a wider range of weight values, reducing information loss caused by range limitations during quantization.

[0076] This dynamic adjustment allows the quantization range of the candidate coding set to better adapt to the characteristics of the module. Whether it is a complex module or a high-error quantization module, more weight details can be retained through a wider coding coverage. This avoids the precision limitations caused by a fixed range for complex modules and reduces the information loss during quantization of high-error-risk modules, thereby improving the accuracy and flexibility of weight quantization coding.

[0077] In step 103, the system processes each module sequentially according to their importance. For each module, it quantizes the weights using each candidate code in its candidate code set, obtaining quantized weights. Then, it combines these quantized weights with the input activation tensor to calculate the quantization result of the quantized simulation model. Simultaneously, the neural network model calculates the non-quantized result using the original weights and the same input activation tensor.

[0078] Because the precision of weights decreases during quantization, there will be a deviation between the quantized and non-quantized results. To reduce this deviation, the system introduces a dynamic error compensation mechanism. By calculating the difference between the quantized and non-quantized results, a compensation term is generated to correct the quantized result, making the corrected quantized result closer to the non-quantized result.

[0079] By comparing the differences between the results before and after quantization and performing targeted compensation, the accuracy loss caused by quantization can be effectively reduced, thereby improving the accuracy of the quantization results. This is especially true for highly important modules, as it better preserves their impact on the model output. For example, if the quantized result of a convolutional layer deviates by 12% from the unquantized result, dynamic error compensation can reduce the deviation to 5%.

[0080] In step 104, for each module, the system calculates the loss value of the corrected quantization result corresponding to each candidate code in its candidate code set. The smaller the loss value, the smaller the deviation between the quantization result and the unquantized result corresponding to that candidate code. From these loss values, the system selects the candidate code corresponding to the minimum loss value as the optimal code for that module. After determining the optimal code for each module, the system uses this optimal code to perform weighted quantization on the corresponding module until the quantization processing of each module is completed.

[0081] The system evaluates the quantization effect of different candidate codes by using loss values. Selecting the optimal code ensures that the module retains its original performance to the maximum extent after quantization, allowing each module to be processed in the optimal quantization method. Ultimately, this enables the entire quantization simulation model to maintain high accuracy while reducing storage and computational overhead. For example, in the candidate code set for a certain linear layer, if a certain candidate code corresponds to the one with the smallest loss value after correction quantization, using this candidate code to quantize the linear layer keeps the model's accuracy loss within an acceptable range, while significantly reducing storage requirements.

[0082] In this application, the linear and convolutional layers in the neural network are first arranged in descending order of module importance to determine module priority. Then, based on the module order, a candidate encoding set for each module is dynamically determined according to its own complexity and estimated quantization error, dynamically adjusting the encoding range of the module to avoid the accuracy limitations caused by a fixed range for complex modules. Next, based on the module order, the weights of each module are quantized according to multiple candidate quantization codes, and dynamic error compensation corrects the deviation between the quantization result and the original non-quantized output, directly improving the quantization accuracy of the module. Finally, the optimal code with the minimum loss for each module is selected for quantization, ensuring that each module uses the most suitable scheme and avoiding accuracy loss due to improper encoding. In this process, the module importance ranking achieves reasonable resource allocation, the candidate encoding set adaptively determines the encoding range, and dynamic error compensation reduces quantization error, collectively improving the model's quantization accuracy.

[0083] As an optional implementation, in step 101, sorting the modules in the neural network model in descending order of module importance includes: determining the loss of the neural network model on the sample dataset; calculating the sensitivity of the loss to the weights of each module through backpropagation to obtain the sensitivity of each module; and determining the importance of the corresponding module based on the sensitivity of each module, wherein the module sensitivity is positively correlated with the module importance.

[0084] First, the system inputs the sample dataset into the neural network model, calculates the model's prediction results through forward propagation, and then compares the prediction results with the true labels in the sample dataset to obtain the model's loss on that dataset, such as cross-entropy loss and mean squared error, which are used to measure the deviation between the predicted value and the true value.

[0085] Subsequently, the system uses the backpropagation algorithm to calculate the sensitivity of the loss to the weights of each module, starting from the loss value. This sensitivity is the overall strength of the partial derivative of the loss function with respect to the weight parameters of each module. A higher sensitivity value indicates that a small change in the weights of that module causes more drastic fluctuations in the overall model loss. Finally, the system uses sensitivity as a measure of module importance; module sensitivity is positively correlated with module importance, and all modules are ranked from highest to lowest importance.

[0086] The formula for module sensitivity is as follows:

[0087]

[0088] in, The loss of the model on the sample dataset (X, Y) W represents the backpropagation calculation. i This represents the weight parameters of the i-th layer of the neural network model.

[0089] This application uses loss and backpropagation-based sensitivity calculation to objectively quantify the impact of modules on model performance. Ranking modules by importance clarifies the priority of subsequent processing, allowing high-sensitivity modules to be prioritized in the quantification process. This eliminates the need to invest too much effort in low-sensitivity modules, instead concentrating computational resources and coding optimization on key high-sensitivity modules, thereby maximizing the preservation of model accuracy with limited resources.

[0090] As an optional implementation, before step 102, the method further includes the following:

[0091] Step S11: Obtain the input dataset, which contains multiple data batches;

[0092] Step S12: Determine the sampling frequency of the module based on the computational complexity of the module and the estimated quantization error, wherein the sampling frequency is used to indicate the number of input data batches allocated to the module;

[0093] Step S13: After inputting the input dataset into the neural network model and the quantization simulation model respectively, sort them according to the importance of the modules, and collect the input activation tensors of each module in the neural network model and the quantization simulation model one by one according to the number of input data batches of each module. The number of input data batches is positively correlated with the number of input activation tensors. The input activation tensors are the intermediate data passed from the previous module to the current module.

[0094] In step S11, the system caches several input batches from the data loader provided by the user and generates a cached dataset. The cached dataset contains multiple data batches, each batch containing multiple data samples and corresponding input tensors. For example, a cached dataset for an image classification task may contain 10 batches, each batch containing 32 images.

[0095] If the user provides a checkpoint configuration file, the algorithm will further sample intermediate activation values ​​at the breakpoint modules specified in the configuration file through forward propagation, generating a subset of data. Intermediate activation values ​​refer to the intermediate results generated by the module during data processing. For example, in an object detection model, the checkpoint configuration file may specify sampling at the 3rd convolutional layer and the 2nd linear layer. The algorithm will organize the feature data output by these layers into separate subsets for subsequent more refined quantitative analysis.

[0096] By using cached datasets and subsets as input datasets, we can both utilize the original input data to cover the overall quantization scenario and focus on the intermediate states of key modules through checkpoints, thereby reducing the cost of processing invalid data.

[0097] In step S12, the system determines the sampling frequency for each module in the quantization simulation model based on its computational complexity and estimated quantization error. The sampling frequency is used to indicate the number of input data batches allocated to that module.

[0098] The sampling frequency is positively correlated with the computational complexity and estimated quantization error of the module. The more complex the module is and the larger the estimated quantization error, the higher the sampling frequency will be. This means that more batches of input data will be allocated to it. For example, 8 batches of data will be allocated to a convolutional layer with high complexity and high initial error, while 2 batches of data will be allocated to a linear layer with low complexity and low initial error, so as to capture the activation characteristics of different modules more fully.

[0099] The computational complexity is primarily assessed by combining the floating-point operations and memory usage of a module. For example, deep convolutional layers typically contain more parameters, requiring more multiplication and addition operations in a single forward computation, resulting in greater memory consumption and higher computational complexity than shallow linear layers. The estimated quantization error is pre-estimated based on the distribution characteristics of the module weights. For instance, modules with a wide range of weight values ​​and subtle differences are prone to larger errors during quantization due to precision limitations, leading to higher estimated quantization errors compared to modules with concentrated weight distributions.

[0100] The formula for calculating the module sampling frequency is shown below.

[0101]

[0102] Among them, f iIndicates the module's sampling frequency; L i Represents the i-th quantization module of the quantization simulation model, Comp(L) i ) reflects L i The computational complexity of the module; Err(L) i ) for L i The module's estimated quantization error; B is the number of input batches configured by the user, typically 10-100.

[0103] In step S13, the system inputs the cached dataset and sub-datasets (if they exist) into the neural network model (FP32 precision) and the quantization simulation model respectively, and then processes each module one by one according to the importance of the modules (from high to low).

[0104] For each module, the system uses the sampling frequency f determined in step S12. i (i.e., the number of batches of input data), the input activation tensors of this module in the two models are collected separately through the forward hook mechanism. The forward hook mechanism is a monitoring tool embedded in the forward propagation process of the model. It can capture the intermediate data received by the module in real time. The input activation tensor refers to the intermediate data passed from the previous module to the current module. For example, the feature map tensor output by the previous convolutional layer will be used as the input activation tensor of the current linear layer.

[0105] Since the number of input data batches is positively correlated with the number of input activation tensors, modules with higher sampling frequencies will collect more input activation tensors. For example, a module with a sampling frequency of 8 will collect input activation tensors corresponding to 8 batches, while a module with a sampling frequency of 2 will only collect input activation tensors from 2 batches. Simultaneously, the algorithm will filter data from the collected activation tensors that meet the mode requirements based on the user-configured input symmetry mode (symmetric or asymmetric). For example, when the user configures a symmetric mode, it will filter activation tensors whose numerical distribution is centered at 0 and whose positive and negative ranges are approximately equal, as the basis for subsequent quantization analysis.

[0106] As an optional implementation, in step 102, the candidate encoding set for each module is determined according to its importance, based on the complexity of each module in the quantization simulation model and the estimated quantization error. This includes the following:

[0107] Based on the importance of the modules, perform the following processing on each module in the quantization simulation model one by one:

[0108] Step S21: Determine the number of candidate codes for the module based on the module's complexity and the estimated quantization error;

[0109] Step S22: Determine the maximum candidate code for the target channel based on the module's maximum weight in the target channel, the candidate code index, and the number of candidate codes;

[0110] Step S23: Determine the minimum candidate code for the target channel based on the module's minimum weight in the target channel, the candidate code index, and the number of candidate codes;

[0111] Step S24: Within the range of the maximum and minimum candidate codes, generate a set of candidate codes that matches the number of candidate codes.

[0112] In step S21, the system processes each module in the quantization simulation model one by one according to their importance (from high to low). For the current module, its computational complexity and estimated quantization error are comprehensively analyzed to determine the number of candidate codes for that module. The higher the computational complexity and the larger the estimated quantization error, the more candidate codes are assigned. For example, a high-complexity convolutional layer with a large estimated quantization error may be assigned 20 candidate codes; while a low-importance, low-complexity, and low-error linear layer may only be assigned 5 candidate codes. This dynamic allocation ensures that key modules have a wider range of coding schemes to choose from.

[0113] The formula for calculating the number of candidate codes is as follows.

[0114]

[0115] Where, N cand，i N represents the number of candidate codes for the module. cand The number of global candidates configured for the user, the default value is 20. Imp(L) i ) for L i Module sensitivity, Err(L) i ) for L i The module's estimated quantization error.

[0116] In step S22, for each target channel in the module (e.g., the output channel of a convolutional layer, the feature channel of a linear layer), the algorithm first obtains the maximum weight value of that channel. Then, combining the candidate encoding index (ranging from 1 to the number of candidate encodings in the module) and the number of candidate encodings, the maximum candidate encoding for that channel is calculated.

[0117] The calculation logic is to allocate the maximum weight range proportionally. Specifically, the smaller the candidate code index, the closer the corresponding maximum candidate code is to the channel's minimum weight side; the larger the candidate code index, the closer it is to the channel's maximum weight. For example, if the maximum weight of a target channel is 10 and the number of candidate codes is 5, when the candidate code index is 3, the maximum candidate code is 10*(3 / 5) = 6, meaning the code corresponding to this index can cover a maximum weight value of 6. In this way, the maximum candidate codes of different indices are distributed in an orderly manner within the channel's maximum weight range.

[0118] In step S23, similar to step S22, for the target channel in the module, the minimum weight value of the channel is first obtained. Then, combined with the candidate coding index and the number of candidate codes, the minimum candidate code of the channel is calculated, ensuring that the minimum candidate code and the maximum candidate code correspond one-to-one in the index. For example, if the minimum weight of a target channel is -10 and the number of candidate codes is 5, when the candidate coding index is 3, the minimum candidate code is -10 × (3 / 5) = -6, forming a symmetrical range with the maximum candidate code 6 corresponding to index 3. If the module adopts an asymmetric quantization mode, the calculation of the minimum candidate code may be based on the actual minimum weight of the channel, rather than being strictly symmetrical, but still maintaining a proportional relationship with the index.

[0119] In step S24, the system generates a corresponding set of candidate codes within the determined range of maximum and minimum candidate codes, based on the number of candidate codes. The generation method involves allocating specific code values ​​between the maximum and minimum candidate codes corresponding to each index at uniform intervals or with a weighted distribution density, ensuring that the codes cover the main weight features within the channel.

[0120] For example, if a target channel has 5 candidate codes, and the maximum candidate code for index 3 is 6 and the minimum candidate code is -6, then under that index, the code set may contain 5 codes: -6, -3, 0, 3, and 6, completely covering the weight range from -6 to 6, with even spacing to suit the weight distribution of that channel. Finally, the candidate codes for all target channels within the module are aggregated to form the candidate code set for that module.

[0121] In this application, the number of candidate codes is dynamically allocated based on the module's computational complexity and estimated quantization error. This allows high-priority modules to receive more coding resources, while lower-priority modules receive fewer codes, saving computational costs and achieving a balance between accuracy and efficiency. By calculating the maximum and minimum candidate codes at the channel level, the coding range is closely correlated with the actual weight distribution (maximum and minimum weights) of each channel, accurately covering the effective range and reducing accuracy loss caused by invalid codes. Finally, a candidate code set is generated within the precise range, ensuring that the codes can meticulously capture subtle changes in weights. For example, channels with dense weight distributions can retain details through dense coding intervals, while sparsely distributed channels can be covered with loose intervals. Ultimately, this makes the candidate code set more suitable for module characteristics, providing a high-quality foundation for subsequent optimal code selection and thus improving overall quantization accuracy.

[0122] As an optional implementation, in step 103, the weights of the corresponding modules in the simulation model are quantized one by one using each candidate code from each candidate code set, according to the importance of the modules, and the quantization results are corrected to reduce the deviation between the quantization results and the non-quantization results of the neural network model, including the following:

[0123] Based on the importance of the modules, perform the following processing on each module in the quantization simulation model one by one:

[0124] Step S31: Quantize the weight of the module using each candidate code in the candidate code set of the module to obtain the quantized weight;

[0125] Step S32: Calculate the quantized weights based on the input activation tensor of the module to obtain the quantization result of the quantization simulation model;

[0126] Step S33: Calculate the original weights of the modules based on the input activation tensors of the corresponding modules in the neural network model to obtain the non-quantized result of the neural network model;

[0127] Step S34: Based on the deviation between the quantified and non-quantified results, correct the quantified results using a dynamic compensation factor to reduce the deviation.

[0128] In step S31, the system sorts the modules according to their importance (from high to low) and processes each module in the quantization simulation model one by one. For the current module, each candidate code is selected from the candidate code set of the module, and the weight of the current module is quantized in turn to obtain the quantized weight.

[0129] The specific process of quantization is as follows: the system converts the original high-precision weights (such as 32-bit floating-point precision) in the module into low-precision representations (such as 8-bit integers). For example, the weights of a convolutional layer, originally stored as 32-bit floating-point, are converted into 8-bit integers after quantization using a candidate encoding. The aim is to reduce the model's storage footprint (e.g., 32-bit to 8-bit conversion can reduce storage by 75%) and computational overhead (lower precision calculations are faster) by reducing weight precision, thus enabling the model to be deployed on resource-constrained devices.

[0130] In step S32, after weight quantization is completed, the system calls the previously collected input activation tensor of the module and performs module calculations (such as convolution operation of convolutional layers and matrix multiplication of linear layers) with the quantized weights to obtain the output result of the quantized simulation model in the module, i.e., the quantization result.

[0131] For example, the input activation tensor of a linear layer is a feature matrix of shape (128, 64), and the quantized weights are a low-precision matrix of shape (64, 32). Matrix multiplication yields a quantized result of shape (128, 32). The core of this step is to simulate the module output under low-precision weights, laying the foundation for subsequent comparison with high-precision results.

[0132] In step S33, while calculating the quantization result, the system processes the corresponding module in the neural network model (original 32-bit floating-point precision model), calls the same input activation tensor as the quantization simulation model (to ensure input consistency), and performs the same module calculation (such as convolution and matrix multiplication) with the original weights of the module (32-bit floating-point precision, not quantized) to obtain the output result of the neural network model in the module, i.e., the non-quantized result.

[0133] The non-quantized result is a high-precision benchmark. For example, the original weights of the linear layer above are a 32-bit floating-point matrix of (64,32). After calculation with the same input activation tensor, a 32-bit floating-point output of (128,32) is obtained, which serves as a reference standard for measuring the deviation of the quantization result.

[0134] In step S34, the system compares the quantized results with the non-quantized results and calculates the deviation between them (such as mean square error and absolute error). Based on this deviation, a dynamic compensation term is introduced for error compensation: a dynamic compensation factor is calculated based on the magnitude of the deviation (e.g., the compensation factor is positively correlated with the deviation, and the larger the deviation, the stronger the compensation). The product of the compensation factor and the deviation is then used as the compensation term and added to the quantization results to obtain the corrected quantization results.

[0135] For example, if the deviation between the quantization result and the non-quantization result of the module is 0.05, and the calculated dynamic compensation factor is 1.2, then the compensation term is 0.05 × 1.2 = 0.06. The corrected quantization result is the original quantization result plus 0.06, reducing the deviation to within 0.01. This step directly reduces the accuracy loss caused by quantization by dynamically adjusting the compensation strength.

[0136] The calculation formula for dynamic compensation is as follows.

[0137] ΔY i，b =α*(Y i，b -Y i，b，q )

[0138] ΔY′ i，b，q =Y i，b，q +ΔY i，b

[0139] Where, ΔY i，b Y is the compensation term, α is the dynamic compensation factor, which dynamically controls the compensation range to avoid overcompensation or undercompensation. i，b For module L i The non-quantitative result Y in the b-th batch i，b，q For module L i The quantization result on the b-th batch, Y′ i,b,q For Y i,b,q The corrected quantification results.

[0140] The formula for calculating the dynamic compensation factor α is as follows.

[0141]

[0142] Among them, Var(Y i,b Y is the non-quantitative result. i,b The first variance, Var(Y) i，b，q Y represents the quantification result. i，b，q The second variance, ∈, is a stability parameter, usually set to 10. -10 .

[0143] In this application, the quantized and non-quantized results are calculated based on the same input activation tensor, ensuring the objectivity of the deviation assessment, providing an accurate basis for error compensation, and avoiding misjudgment due to inconsistent inputs; dynamic error compensation corrects the deviation, making the quantized result close to the original high-precision result.

[0144] As an optional implementation, in step 104, selecting the optimal code from the candidate code set based on the loss value of the corrected quantization result includes the following: determining the loss value of the corrected quantization result for each candidate code corresponding to the module; selecting the minimum loss value from multiple loss values, and taking the candidate code corresponding to the minimum loss value as the optimal code.

[0145] In this application, for each module, the system calculates the loss value of the corrected quantization result corresponding to each candidate code in the candidate code set for that module. The loss value is used to measure the deviation between the corrected quantization result and the unquantized result of the neural network model. Subsequently, the system selects the minimum loss value from all candidate codes; the candidate code corresponding to this minimum loss value is the optimal code for that module. This application evaluates the effect of each candidate code through loss value quantization and selects the optimal code corresponding to the minimum loss value, ensuring that the deviation between the quantization result of the module and the original high-precision result is minimized, thus preserving the model accuracy to the maximum extent.

[0146] After determining the optimal encoding for all supporting modules (linear layers and convolutional layers), the system will re-enable the previously temporarily disabled activation quantizer and weight quantizer. At this point, using the input dataset and the optimal encoding, the activation quantizer and the weight quantizer for non-supporting modules are computed. The activation quantizer's role is to convert the activation values ​​(the output values ​​of a layer during model computation) from high precision to low precision. Calculating its encoding determines the quantization rules (such as numerical range and intervals), reducing the storage space and computational cost of activation values ​​while maintaining model accuracy.

[0147] This application provides a complete schematic diagram of the model quantization process, such as Figure 2 As shown, the steps include the following.

[0148] Step 201: Sort the modules according to their importance.

[0149] Determine the loss of the neural network model on the sample dataset, calculate the sensitivity of the loss to the weights of each module through backpropagation, determine the importance of the modules based on the sensitivity, and rank the modules in descending order of importance.

[0150] Step 202: Obtain the input dataset.

[0151] Multiple input batches are cached from the user-provided data loader to generate a cached dataset. If a checkpoint configuration file exists, intermediate activation values ​​(output values ​​of a certain layer in the model computation) are sampled at specified breakpoint modules through forward propagation to generate a subset dataset. The cached dataset and the subset dataset are used together as the input dataset.

[0152] Step 203: Determine the sampling frequency based on the input dataset.

[0153] By combining the computational complexity of the modules and the estimated quantization error, the sampling frequency of each module is dynamically adjusted, with higher sampling frequencies for modules with high complexity or high error risk.

[0154] Step 204: Collect input activation tensors according to module order.

[0155] The input dataset is fed into the neural network model (FP32 precision) and the quantization simulation model. The input activation tensors (intermediate data passed from the previous module) of each module in the two models are collected one by one through the forward hook mechanism according to the importance of the modules. The data is then filtered according to the symmetric or asymmetric mode configured by the user.

[0156] Step 205: Determine the number of candidate codes according to the module sorting.

[0157] Modules are ranked by importance, and a number of candidate codes are allocated to each module based on module complexity and estimated quantization error (modules with higher importance and larger errors are allocated more codes).

[0158] Step 206: Determine the candidate code set based on the number of candidate codes.

[0159] For each target channel of the module, the maximum and minimum candidate codes are calculated based on the channel's maximum and minimum weights, candidate code indices, and quantities. Within the range of the maximum and minimum candidate codes, a corresponding number of candidate codes are generated, forming the module's candidate code set.

[0160] Step 207: Sort by module and dynamically correct the quantization result using each code in the candidate code set.

[0161] Modules are ranked by importance, and the weights of corresponding modules are quantized using each code in each candidate code set to obtain quantized weights. The quantized results of the quantized simulation model and the unquantized results of the neural network model are calculated using the input activation tensor. A dynamic compensation factor is introduced based on the deviation between the two to correct the quantization results and reduce bias.

[0162] Step 208: Determine the optimal encoding and use the optimal encoding for quantization.

[0163] Calculate the loss value of the corrected quantization result corresponding to each candidate code, select the code corresponding to the minimum loss value as the optimal code of the module, and use the optimal code of each module to perform quantization of the corresponding module.

[0164] This application has the following beneficial effects.

[0165] 1. Improve quantization accuracy. Through a dynamic error compensation mechanism, an adaptive compensation factor is designed based on variance to calculate the compensation term, effectively correcting the non-quantized result of the quantization result and the original output, reducing quantization error, making the output of the quantized model closer to the original FP32 model, significantly improving quantization accuracy, and enhancing the robustness of the model.

[0166] 2. Optimize resource utilization efficiency. By employing an adaptive sampling strategy, the sampling frequency is dynamically adjusted based on the computational complexity of the modules and the estimated quantization error. This allows modules with high complexity and high error to obtain more sampling resources, improving the targeting and efficiency of activation data collection. Simultaneously, through an adaptive encoding search strategy, the number of candidate codes is dynamically adjusted based on module importance and error prediction, generating a better quantization range. This improves the accuracy and flexibility of weighted quantization encoding while optimizing resource allocation, balancing quantization accuracy and computational cost.

[0167] 3. Enhanced applicability and flexibility. This algorithm is compatible with the PyTorch framework and the AIMET quantization simulation tool, supports multiple modules and user configurations, and can meet the needs of different scenarios. It provides an efficient solution for deploying deep learning models in resource-constrained scenarios, and has broad applicability and flexibility.

[0168] Based on the same technical concept, this application provides a model quantization device, such as... Figure 3 As shown, the device includes:

[0169] The sorting module 301 is used to sort the modules in the neural network model in descending order according to their importance, wherein each module is a linear layer or a convolutional layer;

[0170] Module 302 is used to sort modules according to their importance and determine the candidate encoding set of each module in the quantization simulation model based on its own complexity and the estimated quantization error.

[0171] The correction module 303 is used to sort the modules according to their importance, quantize the weights of the corresponding modules in the simulation model one by one using each candidate code in each candidate code set, and correct the quantization results to reduce the deviation between the quantization results and the non-quantization results of the neural network model.

[0172] The quantization module 304 is used to select the optimal code from the candidate code set based on the loss value of the corrected quantization result, and to quantize the corresponding module using the optimal code of each module in the quantization simulation model.

[0173] Optionally, the device is also used for:

[0174] Obtain the input dataset, which contains multiple batches of data;

[0175] The sampling frequency of a module is determined based on its computational complexity and estimated quantization error, where the sampling frequency is used to indicate the number of input data batches allocated to the module.

[0176] After inputting the input dataset into the neural network model and the quantization simulation model respectively, the modules are sorted according to their importance. Then, the input activation tensors of each module in the neural network model and the quantization simulation model are collected one by one according to the number of input data batches for each module. The number of input data batches is positively correlated with the number of input activation tensors. The input activation tensors are the intermediate data passed from the previous module to the current module.

[0177] Optionally, the determining module 302 is used for:

[0178] Based on the importance of the modules, perform the following processing on each module in the quantization simulation model one by one:

[0179] The number of candidate codes for a module is determined based on the module's complexity and the estimated quantization error.

[0180] The maximum candidate code for the target channel is determined based on the module’s maximum weight in the target channel, the candidate code index, and the number of candidate codes.

[0181] The minimum candidate code for the target channel is determined based on the module's minimum weight in the target channel, the candidate code index, and the number of candidate codes.

[0182] Within the range of the maximum and minimum candidate codes, generate a set of candidate codes that matches the number of candidate codes.

[0183] Optionally, the correction module 303 is used for:

[0184] Based on the importance of the modules, perform the following processing on each module in the quantization simulation model one by one:

[0185] The weights of the module are quantized using each candidate code in the candidate code set of the module to obtain the quantized weights;

[0186] The quantization results of the quantization simulation model are obtained by calculating the quantization weights based on the input activation tensor of the module.

[0187] The original weights of the modules are calculated based on the input activation tensors of the corresponding modules in the neural network model to obtain the non-quantized result of the neural network model.

[0188] Based on the deviation between quantified and non-quantified results, the quantified results are corrected through a dynamic compensation factor to reduce the deviation.

[0189] Optionally, the calculation process of the dynamic compensation factor is as follows:

[0190] Determine the first variance of the non-quantized results of the neural network model and the second variance of the quantized results of the quantized simulation model;

[0191] The dynamic compensation factor is determined based on the first variance, the second variance, and the preset stability parameters.

[0192] Optionally, the quantization module 304 is used for:

[0193] Determine the loss value of the corrected quantization result for each candidate code corresponding to the module;

[0194] The minimum loss value is selected from multiple loss values, and the candidate code corresponding to the minimum loss value is taken as the optimal code.

[0195] Optionally, the determining module 301 is used for:

[0196] Determine the loss of the neural network model on the sample dataset;

[0197] The sensitivity of each module is obtained by calculating the loss sensitivity to the weights of each module through backpropagation.

[0198] The importance of each module is determined based on its sensitivity, with a positive correlation between module sensitivity and module importance.

[0199] like Figure 4 As shown, this application provides an electronic device including a processor 401, a communication interface 402, a memory 403, and a communication bus 404, wherein the processor 401, the communication interface 402, and the memory 403 communicate with each other through the communication bus 404.

[0200] Memory 403 is used to store computer programs.

[0201] In one embodiment of this application, when the processor 401 executes the program stored in the memory 403, it implements the model quantization method provided in any of the foregoing method embodiments.

[0202] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the model quantization method provided in any of the foregoing method embodiments.

[0203] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0204] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0205] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0206] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A model quantization method, characterized in that, The method includes: The modules in the neural network model are arranged in descending order of importance, wherein each module is a linear layer or a convolutional layer; Based on the importance of the modules, the candidate encoding set of each module in the quantization simulation model is determined one by one according to the complexity of each module and the estimated quantization error. According to the importance of the modules, the weights of the corresponding modules in the quantization simulation model are quantized one by one using each candidate code in each candidate code set, and the quantization results are corrected to reduce the deviation between the quantization results and the non-quantization results of the neural network model. The optimal code is selected from the candidate code set based on the loss value of the corrected quantization result, and the corresponding module is quantized using the optimal code of each module in the quantization simulation model. The process of quantizing the weights of corresponding modules in the quantization simulation model by sequentially using each candidate code from each candidate code set, according to the importance of the modules, and correcting the quantization results to reduce the deviation between the quantization results and the non-quantization results of the neural network model includes: Based on the importance of the modules, perform the following processing on each module in the quantization simulation model one by one: The weights of the module are quantized using each candidate code in the candidate code set of the module to obtain the quantized weights; The quantized weights are calculated based on the input activation tensor of the module to obtain the quantization result of the quantization simulation model. The original weights of the modules are calculated based on the input activation tensors of the corresponding modules in the neural network model to obtain the non-quantized result of the neural network model. Based on the deviation between the quantization result and the non-quantization result, the quantization result is corrected by a dynamic compensation factor to reduce the deviation; The calculation process for the dynamic compensation factor includes: Determine the first variance of the non-quantized result of the neural network model, and determine the second variance of the quantized result of the quantized simulation model; The dynamic compensation factor is determined based on the first variance, the second variance, and the preset stability parameter.

2. The method according to claim 1, characterized in that, Before determining the candidate encoding set for each module in the quantization simulation model based on its complexity and estimated quantization error, according to the module importance ranking, the method further includes: Obtain the input dataset, wherein the input dataset contains multiple data batches; The sampling frequency of the module is determined based on the computational complexity of the module and the estimated quantization error, wherein the sampling frequency is used to indicate the number of input data batches allocated to the module; After inputting the input dataset into the neural network model and the quantization simulation model respectively, the modules are sorted according to their importance. Then, the input activation tensors of each module in the neural network model and the quantization simulation model are collected one by one according to the number of input data batches of each module. The number of input data batches is positively correlated with the number of input activation tensors. The input activation tensors are intermediate data passed from the previous module to the current module.

3. The method according to claim 1, characterized in that, Based on the importance of each module, the candidate encoding set for each module in the quantization simulation model is determined according to its own complexity and estimated quantization error. Based on the importance of the modules, perform the following processing on each module in the quantization simulation model one by one: The number of candidate codes for the module is determined based on the module's complexity and the estimated quantization error. The maximum candidate code for the target channel is determined based on the maximum weight of the module in the target channel, the candidate code index, and the number of candidate codes. The minimum candidate code for the target channel is determined based on the minimum weight of the module in the target channel, the candidate code index, and the number of candidate codes. Within the range of the maximum and minimum candidate codes, a set of candidate codes corresponding to the number of candidate codes is generated.

4. The method according to claim 1, characterized in that, Selecting the optimal code from the candidate code set based on the loss value of the corrected quantization result includes: Determine the loss value of the corrected quantization result for each candidate code corresponding to the module; The minimum loss value is selected from multiple loss values, and the candidate code corresponding to the minimum loss value is taken as the optimal code.

5. The method according to claim 1, characterized in that, Arrange the modules in the neural network model in descending order of importance, including: Determine the loss of the neural network model on the sample dataset; The sensitivity of each module is obtained by calculating the sensitivity of the loss to the weights of each module through backpropagation. The importance of each module is determined based on its sensitivity, with a positive correlation between module sensitivity and module importance.

6. A model quantization device, characterized in that, The device includes: The sorting module is used to sort the modules in the neural network model in descending order of module importance, wherein each module is a linear layer or a convolutional layer; The module is determined by prioritizing modules according to their importance and determining the candidate encoding set for each module based on its own complexity and the estimated quantization error in the quantization simulation model. The correction module is used to sort the modules according to their importance, quantize the weights of the corresponding modules in the quantization simulation model one by one using each candidate code in each candidate code set, and correct the quantization results to reduce the deviation between the quantization results and the non-quantization results of the neural network model. The quantization module is used to select the optimal code from the candidate code set based on the loss value of the corrected quantization result, and to quantize the corresponding module using the optimal code of each module in the quantization simulation model. The correction module is used for: Based on the importance of the modules, perform the following processing on each module in the quantization simulation model one by one: The weights of the module are quantized using each candidate code in the candidate code set of the module to obtain the quantized weights; The quantized weights are calculated based on the input activation tensor of the module to obtain the quantization result of the quantization simulation model. The original weights of the modules are calculated based on the input activation tensors of the corresponding modules in the neural network model to obtain the non-quantized result of the neural network model. Based on the deviation between the quantization result and the non-quantization result, the quantization result is corrected by a dynamic compensation factor to reduce the deviation; The device is also used for: Determine the first variance of the non-quantized result of the neural network model, and determine the second variance of the quantized result of the quantized simulation model; The dynamic compensation factor is determined based on the first variance, the second variance, and the preset stability parameter.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-5.

Images