Model module level quantization derivation method, apparatus, and storage medium

Abstract

The application discloses a model module level quantization derivation method, equipment and a storage medium, and relates to the technical field of model quantization. The model module level quantization derivation method comprises the following steps: determining a candidate quantization scheme set corresponding to each target module in a to-be-quantized model according to a quantization rule; determining a target quantization scheme for each target module from the candidate quantization scheme set by taking a target compression rate as a constraint and based on the support of an inference kernel corresponding to a target hardware type and a target inference framework; and replacing each target module with a corresponding quantization module according to the quantization statistical information of each target module and the target quantization scheme, and deriving a quantization checkpoint. The application can solve the technical problem that a unified quantization scheme cannot simultaneously consider the differences in quantization sensitivity of different modules and the differences in execution efficiency of a target inference environment.

Description

Technical Field

[0001] This application relates to the field of model quantization technology, and in particular to a method, device and storage medium for model module-level quantization export. Background Technology

[0002] As the parameter size of Transformer models continues to increase, deployment faces challenges such as large model storage volume, high GPU memory consumption, high memory bandwidth pressure, and high inference latency. To reduce deployment costs, improve inference throughput, and shorten response time, model quantization has become a key technical approach in the deployment of large-scale models.

[0003] Existing model quantization methods typically reduce model storage size by compressing the original high-precision floating-point weights into a lower bit representation, and reduce inference computation and data transfer overhead when the corresponding underlying operators are available. Summary of the Invention

[0004] The main purpose of this application is to provide a model module-level quantization export method, device and storage medium, which aims to solve the technical problem that a unified quantization scheme is difficult to balance the differences in module sensitivity and inference environment efficiency.

[0005] To achieve the above objectives, this application provides a model module-level quantization export method, which includes: Based on the quantization rules, determine the set of candidate quantization schemes corresponding to each target module in the model to be quantized; With the target compression ratio as a constraint, and based on the target hardware type and the inference kernel support of the target inference framework, a target quantization scheme is determined for each target module from the set of candidate quantization schemes; Based on the quantization statistics of each target module and the target quantization scheme, each target is replaced with the corresponding quantization module, and quantization checkpoints are derived.

[0006] In one embodiment, determining the set of candidate quantization schemes corresponding to each target module in the model to be quantized according to quantization rules includes: Identify the target modules in the model to be quantized and establish a module path index for each target module; The quantization rule file is parsed to obtain the quantization rule, which includes the module path matching expression and its corresponding quantization method identifier, quantization backend type, and quantization backend parameters. Based on the matching result between the module path matching expression and the module path index, a set of candidate quantization schemes for each target module is determined.

[0007] In one embodiment, with a target compression ratio as a constraint, and based on the target hardware type and the inference kernel support corresponding to the target inference framework, a target quantization scheme is determined for each target module from the set of candidate quantization schemes, including: For each candidate quantization scheme of each target module, calculate the storage cost index, accuracy loss index, and inference latency index of the target module under the candidate quantization scheme, as well as the inference latency index under the target hardware type and the target inference framework. Based on the target hardware type and the target inference framework, query the inference kernel mapping table to determine whether each candidate quantization scheme has a corresponding underlying inference kernel. If not, increase the inference latency index of the candidate quantization scheme or mark it as unavailable. With the constraint that the total storage cost of the entire model does not exceed the storage budget corresponding to the target compression rate, and with the objective of minimizing the weighted sum of the total accuracy loss and inference latency of the entire model, the target quantization scheme of each target module is obtained through joint planning.

[0008] In one embodiment, calculating the storage cost metric, accuracy loss metric, and inference latency metric of the target module under the candidate quantization scheme, and under the target hardware type and the target inference framework, includes: The storage cost index is calculated based on the original weight volume of the target module and the quantization bit width of the candidate quantization scheme. The model to be quantized is subjected to forward computation using calibration data to obtain the output tensor of the target module before quantization; simulated quantization corresponding to the candidate quantization scheme is applied to the weights of the target module, and forward computation is performed again to obtain the output tensor after quantization; the accuracy loss index is determined based on the difference between the output tensor before quantization and the output tensor after quantization. The inference kernel mapping table is used to obtain the estimated execution latency of the underlying inference kernels corresponding to the target hardware type and each quantization backend under the target inference framework, which are stored in advance, as the inference latency index. If the inference kernel mapping table does not contain latency data corresponding to the candidate quantization scheme, the inference latency index is estimated based on the theoretical computational cost of the candidate quantization scheme.

[0009] In one embodiment, the step of jointly planning the target quantization scheme for each target module, with the constraint that the total storage cost of the entire model does not exceed the storage budget corresponding to the target compression ratio, and with the objective of minimizing the weighted sum of the total accuracy loss and inference latency of the entire model, includes: Calculate the sensitivity index for each target module, the sensitivity index being determined based on the accuracy loss index; sort the target modules according to the sensitivity index from high to low; assign candidate quantization schemes to each target module in order of ranking, assigning candidate quantization schemes with smaller accuracy loss to target modules with higher sensitivity indices, until the total storage cost index of the entire model meets the storage budget corresponding to the target compression ratio; or... The candidate quantization schemes for each target module are modeled as a combinatorial optimization problem. The storage cost index is used as a constraint, and the weighted sum of the accuracy loss index and the inference latency index is used as the objective function. Integer programming or dynamic programming is used to solve the problem, and the target quantization schemes for each target module that minimize the objective function are obtained.

[0010] In one embodiment, the step of replacing each target with a corresponding quantization module based on the quantization statistics of each target module and the target quantization scheme, and deriving quantization checkpoints, includes: Forward calculations are performed using calibration data to collect quantitative statistical information for each target module; Based on the target quantization scheme and the quantization statistics, each target module is replaced with the corresponding quantization module to obtain the quantized model; The quantization checkpoints are derived from the quantized model, and the quantization checkpoints include at least the quantized weight parameters and the quantization parameters.

[0011] In one embodiment, after deriving the quantization checkpoint, the method further includes: Based on the model configuration of the model to be quantized and the quantization rules, a quantization skeleton model is constructed, wherein each target module in the quantization skeleton model is a quantization module without loaded quantization weights. The quantization weights and quantization parameters in the quantization checkpoints are loaded into the corresponding positions of each target module in the quantization skeleton model to obtain the reconstructed quantization model. The reconstructed quantization model is validated, and the validation includes at least one of the following: structural consistency validation, parameter key name consistency validation, and short sequence inference validation.

[0012] In one embodiment, before determining the set of candidate quantization schemes corresponding to each target module in the model to be quantized according to the quantization rules, the method further includes: Traverse the internal structure of the model to be quantized and identify special structures in the model to be quantized. The special structures include at least one of the following: expert hybrid structure, shared expert structure, merged projection structure, and key-value cache quantization structure. If the expert hybrid structure or the shared expert structure is identified, the gated projection, up projection and down projection contained in each expert in the expert hybrid structure or the shared expert structure are decomposed into independent logical expert projection targets, and a logical target identifier is assigned to each logical expert projection target. If the merged projection structure is identified, a mapping relationship is established between each split projection target constituting the merged projection structure and the merged projection target. The merged projection structure includes a query / key / value merged projection formed by merging query projection, key projection and value projection, or a gated / up merged projection formed by merging gated projection and up projection. If the key-value cache quantization structure is identified, a key scale buffer and a value scale buffer are preset in the attention module to which the key-value cache quantization structure belongs. The key scale buffer and the value scale buffer are used to write the scale parameters of the key tensor and the value tensor during calibration.

[0013] In addition, to achieve the above objectives, this application also provides a model module-level quantization export device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the model module-level quantization export method described above.

[0014] In addition, to achieve the above objectives, this application also provides a storage medium, which is a computer-readable storage medium, on which a program implementing the model module-level quantization export method is stored. The program implementing the model module-level quantization export method is executed by a processor to implement the steps of the model module-level quantization export method as described above.

[0015] This application provides a model module-level quantization export method. First, it identifies each target module in the model to be quantized and parses the quantization rules to determine a set of candidate quantization schemes for each target module. This set of candidate quantization schemes provides a selectable scheme space for subsequent differentiated quantization configuration. Then, using the target compression rate as a constraint and based on the target hardware type and the inference kernel support of the target inference framework, a target quantization scheme is determined for each target module from the set of candidate quantization schemes. This allows for the selection of quantization schemes that match the target deployment environment for different modules while meeting the overall compression rate requirements, balancing compression efficiency and actual inference performance. After determining the target quantization scheme for each target module, based on the quantization statistics of each target module and the target quantization scheme, each target module is replaced with the corresponding quantization module, and quantization checkpoints are exported. These uniformly exported quantization checkpoints enable stable saving and rapid recovery of the quantized model, thus solving the technical problem that a unified quantization scheme cannot simultaneously address the differences in quantization sensitivity between different modules and the differences in execution efficiency under the target inference environment. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the first embodiment of the model module-level quantization export method in this application. Figure 2 This is a schematic diagram of the module-level rule-driven quantization mechanism provided in Embodiment 10 of the model module-level quantization export method of this application; Figure 3 This is a schematic diagram of the quantization mechanism provided in Embodiment 10 of the model module-level quantization export method of this application; Figure 4 This is a schematic diagram of the quantitative planning provided in Embodiment 10 of the model module-level quantitative derivation method of this application; Figure 5 This is a schematic diagram illustrating the accuracy-delay tradeoff provided in Embodiment 10 of the model module-level quantization export method of this application. Figure 6 A unified adaptation and export diagram provided for Embodiment 10 of the model module-level quantization export method of this application; Figure 7This is a schematic diagram of the hardware structure involved in the low-power control device for the mecha in this application.

[0019] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0020] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not intended to limit this application.

[0021] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0022] Currently, model quantization schemes typically employ a uniform quantization format for each module or rely on manual static configuration, making it difficult to balance the differences in quantization sensitivity between different modules with the differences in execution efficiency of the target inference environment.

[0023] The main solution of this application is as follows: First, a set of candidate quantization schemes corresponding to each target module in the model to be quantized is determined according to quantization rules. Second, a target quantization scheme is determined for each target module from the set of candidate quantization schemes, constrained by the target compression rate and based on the target hardware type and the inference kernel support of the target inference framework. Third, based on the quantization statistics of each target module and the target quantization scheme, each target is replaced with the corresponding quantization module, and quantization checkpoints are exported. By generating a set of candidate quantization schemes independently for each target module, the limitation of a unified scheme is broken. Then, constrained by the target compression rate and based on the inference kernel support, a quantization scheme adapted to the target deployment environment is automatically matched for each module. Finally, by uniformly exporting checkpoints, the differentiated configuration results are stably implemented, thus solving the problem that a unified quantization scheme cannot take into account the differences in quantization sensitivity of different modules and the differences in execution efficiency of the target inference environment.

[0024] It should be noted that the execution entity in this embodiment can be a model module-level quantization export device, or a computing device with data processing and program execution functions such as a server, computer equipment, or cloud computing cluster. This embodiment does not specifically limit it in this way. The following uses a model module-level quantization export device as the execution entity to describe this embodiment and the following embodiments.

[0025] Based on this, Embodiment 1 of this application proposes a model module-level quantization export method, please refer to... Figure 1 The model module-level quantization export method includes steps S10 to S30: Step S10: Determine the set of candidate quantization schemes corresponding to each target module in the model to be quantized according to the quantization rules.

[0026] In this embodiment, the model to be quantized refers to a Transformer model that requires quantization processing, including but not limited to causal language models, dense structure models, and expert hybrid structure models. The target module refers to a quantizable functional module within the model to be quantized, including but not limited to query projection, key projection, value projection, and output projection in the self-attention module; gated projection, up projection, and down projection in the feedforward module; and expert projection modules, shared expert projection modules, key-value cache quantization-related modules, and merged projection modules in the expert hybrid structure. Quantization rules refer to a set of configurations used to describe the target module, quantization method, quantization backend, and their parameters, including at least a module path matching expression, quantization method identifier, quantization backend type, and quantization backend parameters. The candidate quantization scheme set refers to a set of one or more optional quantization schemes corresponding to each target module, where each quantization scheme includes at least a quantization method identifier and a quantization backend type.

[0027] As an optional implementation, the Transformer model to be quantized, the model configuration, and the word segmenter are loaded, and the quantization rule file is read and parsed. Each rule is parsed into a module path matching expression, a quantization method identifier, a quantization backend type, and a quantization backend parameter field. Then, the device determines the corresponding set of candidate quantization schemes for each target module based on the matching result between the module path matching expression and the module path index.

[0028] Optionally, when multiple rules match the same module at the same time, the rule that appears later becomes the final effective rule. Through this sequential overriding mechanism, a broader rule can be used first to specify the global quantization method, and then a more detailed rule can be used to cover the key module, thus achieving flexible module-level differentiated quantization configuration.

[0029] Step S20: With the target compression rate as a constraint, and based on the target hardware type and the inference kernel support corresponding to the target inference framework, determine the target quantization scheme for each target module from the set of candidate quantization schemes.

[0030] In this embodiment, the target compression ratio refers to the target proportion of the quantized model volume to the unquantized model volume, which can be expressed as the ratio of the quantized model size to the unquantized model size, or as its corresponding compression percentage. The target hardware type refers to the type of hardware platform on which the quantized model is actually deployed and run, including but not limited to graphics processors, central processing units, or dedicated acceleration chips. The target inference framework refers to the inference software framework on which the quantized model is actually deployed and run, including but not limited to deep learning frameworks or dedicated inference engines. Inference kernel support refers to whether each quantization scheme has a corresponding low-level efficient computational implementation under the target hardware type and target inference framework, and the actual execution performance of that implementation. The target quantization scheme refers to the quantization scheme finally selected by each target module from the set of candidate quantization schemes, and at least includes a quantization method identifier and a quantization backend type.

[0031] As an optional implementation, a specified target compression ratio, target hardware type, and target inference framework are obtained. For each candidate quantization scheme of each target module, its performance in three dimensions—storage cost, accuracy preservation, and inference latency—is comprehensively evaluated. The storage cost dimension reflects the compression effect of the scheme on model size, the accuracy preservation dimension reflects the impact of the scheme on model output accuracy, and the inference latency dimension reflects the actual execution efficiency of the scheme under the target hardware type and target inference framework. Candidate quantization schemes lacking efficient inference kernel support under the target hardware type and target inference framework are prioritized or excluded during the comprehensive evaluation to avoid selecting schemes with low actual inference efficiency in the target deployment environment. Then, with the total storage cost of the entire model satisfying the target compression ratio as a constraint, and with the overall optimization of the total accuracy loss and inference latency of the entire model as the objective, a target quantization scheme is determined for each target module.

[0032] Step S30: Based on the quantization statistics of each target module and the target quantization scheme, replace each target with the corresponding quantization module and export the quantization checkpoints.

[0033] In this embodiment, quantization statistics refer to the statistics collected during the calibration process required to generate quantization parameters, including but not limited to activation value distribution information, weight distribution information, channel scale information, or importance weight information. A quantization module refers to replacing the original high-precision module with a module capable of performing low-bit computation and outputting results of corresponding precision. A quantization checkpoint refers to a persistent file or set of files used to store the quantized model weights, quantization parameters, configuration information, and related metadata, for loading and restoring the quantized model during subsequent deployment.

[0034] As an optional implementation, after determining the target quantization scheme for each target module, forward computation is performed on the model using calibration data to collect the quantization statistics required by each target module. Based on the target quantization scheme and the collected quantization statistics, the original implementation of each target module is replaced with the corresponding quantization module implementation. After replacement, the quantized model weights, quantization parameters, model configuration information, and related metadata are exported as quantization checkpoints and saved to a specified path. During export, the weights and parameters of each module are organized according to a unified naming convention and data structure to ensure that the parameter organization in the quantization checkpoint corresponds to the structure of the quantized model, ensuring that a deployable quantized model can be stably recovered based on this quantization checkpoint.

[0035] This embodiment identifies each target module and parses the quantization rules to generate a set of candidate quantization schemes independently for different modules, breaking the limitations of a unified quantization method. This allows high-sensitivity modules and low-sensitivity modules to each have differentiated precision option spaces. Using the target compression rate as a constraint, and based on the target hardware type and the support of the inference kernel corresponding to the target inference framework, joint planning is performed. This ensures that the target quantization schemes for each module match the actual execution capabilities of the target deployment environment while meeting the overall compression rate requirements, avoiding the problem of manual static configuration struggling to balance compression rate and inference efficiency. Through a unified quantization checkpoint export mechanism, the differentiated quantization models can be stably saved and quickly restored, ensuring that the decision results can be implemented as deployable quantization models. Therefore, this application overcomes the technical shortcomings of unified quantization or manual static configuration, which cannot simultaneously consider the differences in quantization sensitivity between different modules and the differences in execution efficiency of the target inference environment.

[0036] Based on Embodiment 1, Embodiment 2 of this application proposes a model module-level quantization derivation method, wherein step S10 may include steps S11 to S13: Step S11: Identify the target modules in the model to be quantized and establish the module path index for each target module.

[0037] In this embodiment, the module path index refers to the index relationship that maps the path name of a module in the model to the module object.

[0038] As an optional implementation, after loading the Transformer model to be quantized and its configuration, the internal structure of the model is traversed, recursively visiting each sub-module starting from the root module, and recording the path name of each module and its corresponding module object. For sub-modules within a namespace, a unique module path string is generated according to its hierarchical structure in the model, and this path string and the module object are stored in an index table as key-value pairs.

[0039] Step S12: Parse the quantization rule file to obtain the quantization rules. The quantization rules include the module path matching expression and its corresponding quantization method identifier, quantization backend type, and quantization backend parameters.

[0040] In this embodiment, the quantization rule file is a configuration file describing the quantization method to be used for each target module. The module path matching expression is an expression used to match the path names of modules within the model; regular expressions or wildcard expressions are preferred. The quantization method identifier is the type identifier of the quantization algorithm. The quantization backend type refers to the implementation type of the target module that actually carries out inference computation after quantization. The quantization backend parameters are adjustable parameters that correspond to the quantization backend type.

[0041] As an optional implementation, the quantization rule file is read and parsed line by line. Each rule is broken down into four fields: module path matching expression, quantization method identifier, quantization backend type, and quantization backend parameters. These fields are then stored in the rule set in the order they appear in the file. The quantization backend parameters retain their original data structure during parsing and are stored directly associated with the quantization method identifier and quantization backend type.

[0042] Optionally, when the module path matching expressions of multiple rules match the same target module at the same time, a sequential overriding mechanism is adopted, where the later rule overrides the earlier rule, and the later rule becomes the final effective rule.

[0043] Step S13: Determine the set of candidate quantization schemes for each target module based on the matching result between the module path matching expression and the module path index.

[0044] As an optional implementation, the path name of each target module is matched one by one with the module path matching expression of each rule in the quantization rule set. If a match is successful, the quantization method identifier, quantization backend type, and quantization backend parameters corresponding to that rule are combined into a candidate quantization scheme and added to the candidate quantization scheme set of that target module. If multiple rules are matched for the same target module, the candidate quantization scheme of the module is determined according to the order overriding mechanism, with the last matched rule as the standard; or, the quantization schemes corresponding to all matched rules are added to the candidate quantization scheme set of that module. For target modules that do not match any quantization rules, a default quantization scheme is assigned to them or the original high precision is maintained.

[0045] This embodiment locates target modules of various types by establishing a module path index, parses the quantization rule file and uses a sequential overlay mechanism to achieve a flexible combination of broad and fine rules, and combines the successfully matched rules into a set of candidate quantization schemes, realizing rule-driven module-level quantization scheme generation, providing differentiated quantization options for modules with different quantization sensitivities, without having to write quantization rules separately for each module.

[0046] Based on Embodiment 1, Embodiment 3 of this application proposes a model module-level quantization derivation method, wherein step S20 includes steps S21 to S23: Step S21: For each candidate quantization scheme of each target module, calculate the storage cost index, accuracy loss index, and inference latency index of the target module under the candidate quantization scheme, as well as the inference latency index under the target hardware type and the target inference framework.

[0047] In this embodiment, the storage cost metric is a quantized value characterizing the storage space occupied by the target module after adopting the candidate quantization scheme, used to evaluate the compression effect of the scheme on the model size. The accuracy loss metric is a quantized value characterizing the deviation of the target module's output from the original high-precision output after adopting the candidate quantization scheme, used to evaluate the impact of the scheme on model accuracy. The inference latency metric is a quantized value characterizing the time required for the target module to perform inference using the candidate quantization scheme under the target hardware type and target inference framework, used to evaluate the execution efficiency of the scheme in a real deployment environment.

[0048] As an optional implementation, for each candidate quantization scheme of each target module, storage cost, accuracy loss, and inference latency metrics are calculated separately. The storage cost metric is determined based on the original weight volume of the target module and the quantization bit width of the candidate quantization scheme. The accuracy loss metric is determined by performing forward computation on the model to be quantized using calibration data and comparing the output differences of the target module before and after quantization. The inference latency metric is obtained by querying the inference kernel mapping table to obtain the expected execution latency of each quantization backend under different hardware types and inference frameworks.

[0049] Step S22: Based on the target hardware type and the target inference framework, query the inference kernel mapping table to determine whether each candidate quantization scheme has a corresponding underlying inference kernel. If not, increase the inference latency index of the candidate quantization scheme or mark it as unavailable.

[0050] In this embodiment, the inference kernel mapping table is a pre-established set of mapping relationships used to record the underlying inference kernel types and their performance data corresponding to each quantization backend under different hardware types and inference frameworks. The underlying inference kernel refers to the specific computational implementation that performs underlying operations such as matrix multiplication, data rearrangement, and dequantization in actual inference computation. The same quantization backend may have different underlying inference kernel implementations under different hardware or inference frameworks, and their execution efficiency may vary significantly.

[0051] As an optional implementation, after obtaining the target hardware type and target inference framework, based on the quantization backend type in each candidate quantization scheme, the inference kernel mapping table is queried to determine whether the quantization backend has a corresponding underlying inference kernel under the target hardware type and target inference framework. If a corresponding underlying inference kernel exists, the expected execution latency of the inference kernel is further queried to update the inference latency metric. If no corresponding underlying inference kernel exists, it indicates that the candidate quantization scheme cannot obtain efficient computing support in the current target deployment environment. In this case, the inference latency metric is increased to a value greater than the latency of any scheme with kernel support, or the candidate quantization scheme is directly marked as unavailable, so that it is automatically excluded in subsequent joint planning.

[0052] Step S23: With the total storage cost index of the entire model not exceeding the storage budget corresponding to the target compression rate as a constraint, and with the goal of minimizing the weighted sum of the total accuracy loss index and the inference latency index of the entire model, the target quantization scheme of each target module is jointly planned.

[0053] In this embodiment, the storage budget refers to the upper limit of the total model storage volume corresponding to the target compression ratio. Joint planning refers to the process of allocating the optimal candidate quantization scheme to each target module while comprehensively considering the accuracy loss and inference latency of each module, under the premise of satisfying the global storage budget.

[0054] As an optional implementation, after calculating the storage cost, accuracy loss, and inference latency metrics for each target module under each candidate quantization scheme, and completing the existence check and index correction for the inference kernel, the joint planning phase begins. First, the sensitivity metrics for each target module are determined based on the accuracy loss metrics; modules with higher accuracy loss have higher sensitivity. Then, with the constraint that the total storage cost of the entire model does not exceed the storage budget corresponding to the target compression ratio, the sum of the storage cost metrics of the final selected quantization schemes for all target modules must not exceed the upper limit specified by the storage budget. Next, the optimization objective is to minimize the weighted sum of the total accuracy loss and inference latency metrics of the entire model, where the total accuracy loss is the sum of the accuracy loss metrics of each target module, and the total inference latency is the sum of the inference latency metrics of each target module, with both weighted according to preset weights. Finally, under the above constraints and optimization objectives, joint planning is performed on the candidate quantization schemes for each target module to determine its optimal quantization scheme. When no feasible solution can be found under the constraints, a prompt message is output, suggesting that the user adjust the target compression ratio or expand the range of candidate quantization schemes. Through the above steps, this embodiment realizes the automatic planning of a full model quantization scheme with the goal of achieving the optimal balance between accuracy loss and inference latency under the constraint of target compression ratio. This avoids the inefficiency and unreliability of manual layer-by-layer adaptation, and incorporates the inference kernel support of the target deployment environment into the optimization decision system.

[0055] Optionally, the joint planning uses the target module set, the candidate quantization scheme set for each target module, the storage cost index, accuracy loss index, and inference latency index corresponding to each candidate quantization scheme, the target compression ratio constraint, and the user-specified accuracy-first, latency-first, or balanced mode as core inputs. In the accuracy-first mode, the weight coefficient of the accuracy loss index in the weighted sum is increased, making the planning result more inclined to select the scheme with lower accuracy loss. In the latency-first mode, the weight coefficient of the inference latency index in the weighted sum is increased, making the planning result more inclined to select the scheme with lower inference latency. In the balanced mode, the weight coefficients of the accuracy loss index and the inference latency index are equal. After the joint planning is completed, the final quantization method of each target module, the quantization backend of each target module, and the corresponding derived layout selection result are used as core outputs. The derived layout selection result is determined according to the requirements of the corresponding inference kernel, including the packing order of weight tensors, block size, organization of scale parameters, and whether to use merged storage or separate storage, so that the parameter organization in the quantization checkpoint matches the efficient execution path in the target inference environment. When no combination of candidate quantization schemes for each target module can satisfy the target compression ratio constraint, a prompt message is output and it is suggested to adjust the target compression ratio or expand the range of candidate quantization schemes.

[0056] This embodiment calculates storage cost, accuracy loss, and inference latency metrics for each candidate quantization scheme of each target module. It then determines whether each candidate quantization scheme has a corresponding underlying inference kernel under the target hardware type and target inference framework by querying the inference kernel mapping table. With the total storage cost of the entire model not exceeding the storage budget corresponding to the target compression ratio as a constraint, and with the goal of minimizing the weighted sum of total accuracy loss and inference latency, joint planning is performed. This achieves automatic allocation of quantization schemes for each module while meeting the compression ratio requirements, avoiding the inefficiency and unreliability of manual layer-by-layer trial and error. At the same time, it ensures that the final target quantization scheme has practical executability and high-efficiency inference performance in the target deployment environment.

[0057] Based on Embodiment 3, Embodiment 4 of this application proposes a model module-level quantization derivation method, wherein step S21 may include steps S211 to S213: Step S211: Calculate the storage cost index based on the original weight volume of the target module and the quantization bit width of the candidate quantization scheme.

[0058] As an optional implementation, the original weight volume of the target module is obtained, i.e., the storage space occupied by the original high-precision weights. The quantized weight volume is calculated based on the quantization bit width in the candidate quantization scheme, and then added to the storage overhead of the scale parameters and other additional metadata to obtain the storage cost index of the target module under the candidate quantization scheme. Here, the quantization bit width refers to the number of bits after quantization of the weight parameters in the quantization scheme, and the storage overhead of the scale parameters and other additional metadata is determined according to the quantization backend type and the data organization method specified in the quantization backend parameters.

[0059] Step S212: Perform forward computation on the model to be quantized using calibration data to obtain the output tensor of the target module before quantization; apply simulated quantization corresponding to the candidate quantization scheme to the weights of the target module, and perform forward computation again to obtain the output tensor after quantization; determine the accuracy loss index based on the difference between the output tensor before quantization and the output tensor after quantization.

[0060] As an optional implementation, calibration data is loaded and input into the model to be quantized, and a complete forward computation is performed. During the forward propagation of the target module, the output tensor of the module is captured and saved, denoted as the pre-quantization output tensor. Subsequently, simulated quantization corresponding to the candidate quantization scheme is applied to the weights of the target module. That is, while keeping the original weight data unchanged, a simulation operation of quantization followed by dequantization is performed on the weights to simulate the accuracy loss in the low-bit computation process. The original weights are replaced with the simulated quantized weights, and the forward computation is performed again. The output tensor of the module is captured and saved, denoted as the post-quantization output tensor. The accuracy loss index is determined based on the difference between the pre-quantization output tensor and the post-quantization output tensor. Optionally, this difference can be calculated using metrics such as mean square error, mean absolute error, or cosine similarity. If the target module is a composite module containing multiple projection layers, the above operation is performed on each projection layer, and the differences between the projection layers are weighted and summed as the accuracy loss index of the target module. If the candidate quantization scheme is to maintain the original accuracy without quantization, the accuracy loss index is directly set to zero.

[0061] Step S213: Obtain the expected execution latency of the underlying inference kernels corresponding to the target hardware type and each quantization backend under the target inference framework, which are pre-stored in the inference kernel mapping table, as the inference latency index; if there is no latency data corresponding to the candidate quantization scheme in the inference kernel mapping table, the inference latency index is estimated based on the theoretical computational amount of the candidate quantization scheme.

[0062] In this embodiment, the theoretical computational cost refers to the number of floating-point operations or integer operations corresponding to the candidate quantization scheme, which is used to approximate the inference delay when there is a lack of measured data.

[0063] As an optional implementation, after obtaining the target hardware type and target inference framework, based on the quantization backend type in each candidate quantization scheme, the inference kernel mapping table is queried to determine whether the quantization backend has a corresponding underlying inference kernel under the target hardware type and target inference framework. If a corresponding underlying inference kernel exists, the expected execution latency corresponding to that inference kernel is used as the inference latency metric for that candidate quantization scheme. If the latency data corresponding to that candidate quantization scheme does not exist in the inference kernel mapping table, it is estimated based on the theoretical computational load of that candidate quantization scheme, and the inference latency metric can be obtained through methods such as hardware peak computing power conversion.

[0064] This embodiment calculates storage cost metrics, accuracy loss metrics, and inference latency metrics for each candidate quantization scheme for each target module, quantifying the evaluation of the quantization scheme into three comparable dimensions: storage compression effect, accuracy preservation degree, and actual execution efficiency. By querying the inference kernel mapping table, the inference latency evaluation is bound to the target hardware type and the actual kernel performance under the target inference framework, so that the inference efficiency evaluation of each candidate scheme reflects the real execution situation in the target deployment environment.

[0065] Based on Embodiment 3, Embodiment 5 of this application proposes a model module-level quantization derivation method, where step S23 may include step S231: Step S231: Calculate the sensitivity index of each target module, the sensitivity index being determined based on the accuracy loss index; sort each target module from high to low according to the sensitivity index; assign candidate quantization schemes to each target module in order of sorting, assigning candidate quantization schemes with smaller accuracy loss to target modules with higher sensitivity indices, until the total storage cost index of the entire model meets the storage budget corresponding to the target compression rate.

[0066] As an optional implementation, a sensitivity index is first calculated based on the accuracy loss index of each target module. A module with a larger accuracy loss indicates that it is more sensitive to quantization, and therefore has a higher sensitivity index. Then, the target modules are sorted from highest to lowest sensitivity index, with high-sensitivity modules being assigned candidate quantization schemes first. During the allocation process, target modules with higher sensitivity indices are assigned candidate quantization schemes with smaller accuracy losses, i.e., high-precision quantization schemes are prioritized; target modules with lower sensitivity indices are assigned candidate quantization schemes with larger accuracy losses, i.e., they are given priority for compression tasks. Candidate quantization schemes are assigned to each target module in this order until the total storage cost index of the entire model meets the storage budget corresponding to the target compression ratio. After all modules are allocated, if the total storage cost is still lower than the storage budget, further attempts can be made to reduce the quantization accuracy of some modules to reduce storage usage; if there is still remaining storage space while meeting the storage budget, the current allocation scheme remains unchanged. If the storage budget cannot be met even after all modules are allocated to the maximum compression scheme, a prompt message is output, suggesting that the user adjust the target compression ratio.

[0067] This embodiment calculates a sensitivity index based on the accuracy loss index of each target module. Modules with greater accuracy loss are more sensitive. The target modules are then sorted from highest to lowest sensitivity. Modules with high sensitivity are prioritized for allocation to candidate quantization schemes with lower accuracy loss, while modules with low sensitivity are prioritized for compression tasks, until the total storage cost of the entire model meets the storage budget corresponding to the target compression ratio. This achieves heuristic allocation of quantization schemes based on sensitivity ranking. This method has low computational cost, is simple to implement, and has high execution efficiency. It can generate feasible quantization allocation schemes that meet compression ratio constraints in a short time, making it suitable for application scenarios with requirements for planning speed or large model size.

[0068] Based on Embodiment 3, Embodiment 6 of this application proposes a model module-level quantization derivation method, where step S23 may include step S232: Step S232: Model the candidate quantization schemes of each target module as a combinatorial optimization problem, with the storage cost index as the constraint, and the weighted sum of the accuracy loss index and the inference latency index as the objective function. Solve the problem using integer programming or dynamic programming to obtain the target quantization scheme of each target module that minimizes the objective function.

[0069] As an optional implementation, the selection problem of candidate quantization schemes for each target module is modeled as a combinatorial optimization problem. Specifically, the candidate quantization schemes for each target module are used as decision variables, the constraint that the total storage cost of the entire model does not exceed the storage budget corresponding to the target compression ratio is used, and the weighted sum of the total accuracy loss and inference latency of the entire model is used as the objective function. The accuracy loss index and inference latency index of each target module are pre-calculated in step S21. Integer programming or dynamic programming algorithms are used to solve the above optimization problem to obtain the target quantization schemes for each target module that minimize the objective function.

[0070] This embodiment models the candidate quantization scheme selection problem for each target module as a combinatorial optimization problem. The candidate quantization schemes for each target module are used as decision variables, the constraint is that the total storage cost of the entire model does not exceed the storage budget corresponding to the target compression rate, and the objective function is the weighted sum of the total accuracy loss and inference latency of the entire model. Integer programming or dynamic programming algorithms are used to solve for the target quantization schemes of each target module that minimize the objective function, thus achieving a globally optimal solution for quantization scheme allocation. Theoretically, this method can obtain the optimal solution that satisfies the constraints, ensuring the optimal balance between accuracy loss and inference latency. It is suitable for applications with high requirements for quantization scheme quality or moderate model size.

[0071] Based on Embodiment 1, Embodiment 7 of this application proposes a model module-level quantization derivation method, wherein step S30 may include steps S31 to S33: Step S31: Perform forward calculations using calibration data to collect quantization statistics for each target module.

[0072] In this embodiment, quantization statistics refer to the statistics collected during the calibration process that are required to generate quantization parameters, including but not limited to activation value distribution information, weight distribution information, channel scale information, or importance weight information.

[0073] As an optional implementation, after determining the target quantization scheme for each target module, calibration data is loaded and input into the model to be quantized for forward computation. During the forward propagation of each target module, the input activation value, output activation value, or other statistics required for quantization are obtained through an interception interface. For quantization schemes requiring activation statistics, statistical information such as the distribution range of activation values ​​and the distribution differences between channels is collected; for key-value cache quantization-related modules, the scale information of key tensors and value tensors is collected before attention updates. After collection, the quantization statistics are associated and stored with the corresponding target module and target quantization scheme. Optionally, to reduce peak memory usage, calibration is performed sequentially on a layer-by-layer basis, i.e., only the current layer to be processed is moved into the target device for calibration and statistical information collection each time, and the intermediate cache is released after processing before continuing to the next layer.

[0074] Step S32: Based on the target quantization scheme and the quantization statistics, replace each target module with the corresponding quantization module to obtain the quantized model.

[0075] As an optional implementation, after obtaining the quantization statistics of each target module, the original module is replaced with the corresponding quantization module according to the target quantization scheme of each target module. For quantization schemes that do not require calibration input, quantization replacement is directly completed based on the original weights; for quantization schemes that require calibration input, after the quantization statistics are collected, the quantization parameters are written into the corresponding quantization module before replacement. After replacement, a fast error assessment is performed on the current layer to check whether the difference in output before and after quantization is within an acceptable range.

[0076] Step S33: Derive quantization checkpoints based on the quantized model. The quantization checkpoints include at least the quantized weight parameters and quantization parameters.

[0077] As an optional implementation, after completing the quantization replacement of all target modules, the quantized model weights, quantization parameters, model configuration information, and related metadata are exported as quantization checkpoints and saved to a specified path. During export, the weights and parameters of each module are organized according to a unified naming convention and data structure, ensuring that the parameter organization in the quantization checkpoints corresponds to the structure of the quantized model. For expert hybrid structures and shared expert structures, the expert projection parameters are standardized and expanded according to the logical target naming method, ensuring that the parameter organization in the quantization checkpoints corresponds one-to-one with the logical modules; for key-value cache quantization modules, the corresponding scale parameters and necessary metadata are also exported.

[0078] Optionally, for large models, a split-save approach can be adopted, where quantization checkpoints are divided into multiple files for saving, in order to reduce the size of a single file and facilitate subsequent loading.

[0079] Optionally, the quantization checkpoint also includes inference kernel-related metadata, which includes at least the packing order of weight tensors, block size, organization of scale parameters, whether merged or separate storage is used, and additional metadata for direct invocation of fast paths during inference. This enables the quantization model to directly construct an execution path that matches the target inference environment based on the inference kernel-related metadata during subsequent loading, without the need for further format conversion.

[0080] This embodiment models the candidate quantization scheme selection problem for each target module as a combinatorial optimization problem. The candidate quantization schemes for each target module are used as decision variables, the constraint is that the total storage cost of the entire model does not exceed the storage budget corresponding to the target compression rate, and the objective function is the weighted sum of the total accuracy loss and inference latency of the entire model. Integer programming or dynamic programming algorithms are used to solve for the target quantization schemes of each target module that minimize the objective function, thus achieving a globally optimal solution for quantization scheme allocation. Theoretically, this method can obtain the optimal solution that satisfies the constraints, ensuring the optimal balance between accuracy loss and inference latency. It is suitable for applications with high requirements for quantization scheme quality or moderate model size.

[0081] Based on any of the above embodiments, Embodiment 8 of this application proposes a model module-level quantization derivation method, which may include steps S40-60 after step S30: Step S40: Construct a quantization skeleton model based on the model configuration of the model to be quantized and the quantization rules. Each target module in the quantization skeleton model is a quantization module without loaded quantization weights.

[0082] In this embodiment, the quantization skeleton model refers to the target model structure pre-constructed based on the original model configuration and quantization rules. The target modules have been replaced with the corresponding quantization module types, but specific quantization weights and parameters have not yet been loaded. The structure of the quantization skeleton model is completely identical to the model to be quantized.

[0083] As an optional implementation, after exporting the quantization checkpoints, the original model configuration of the model to be quantized is read, including the model structure definition, the type of each module, and the connection relationships. Based on the quantization method identifier and quantization backend type of each target module defined in the quantization rules, the corresponding modules in the original model are replaced with the specified quantization module types, constructing a complete quantization skeleton model. At this stage, each quantization module only contains module structure and quantization backend type information; quantization weights and quantization parameters remain uninitialized. For expert hybrid structures and shared expert structures, corresponding quantization module placeholders are established in the skeleton model according to the logical target naming convention; for merged projection modules, the corresponding quantization module types are set in the skeleton model according to the mapping relationship between split projection and merged projection. After construction, the structure of the quantization skeleton model remains consistent with the quantized model structure when exporting the quantization checkpoints.

[0084] Step S50: Load the quantization weights and quantization parameters in the quantization checkpoints into the corresponding positions of each target module in the quantization skeleton model to obtain the reconstructed quantization model.

[0085] In this embodiment, the reconstructed quantization model refers to the complete deployable quantization model obtained by loading the quantization weights and quantization parameters stored in the quantization checkpoints into the quantization skeleton model.

[0086] As an optional implementation, the exported quantization checkpoint file is loaded, and the quantization weight parameters, quantization parameters, and related metadata are read. Following the naming and organization of the parameters in each module of the quantization checkpoint, the quantization weights and parameters are loaded one by one into the corresponding positions of each target module in the quantization skeleton model. For expert hybrid structures, each expert projection parameter is loaded into the corresponding expert projection module in the skeleton model according to the standardized naming convention in the checkpoint; for key-value cache quantization modules, the scale parameters are loaded into the corresponding key-value cache buffer in the skeleton model. After loading, the completeness of the quantization weights and quantization parameters of each target module is checked to confirm that all target modules have been loaded successfully.

[0087] S60, Perform verification on the reconstructed quantization model, the verification including at least one of the following: structural consistency verification, parameter key name consistency verification, and short sequence inference verification.

[0088] In this embodiment, verification refers to the process of checking the reconstructed quantization model to confirm its structural correctness, parameter integrity, and inference usability. Structural consistency verification checks whether the structure of the reconstructed quantization model is consistent with the quantization skeleton model, and whether the types and connections of each module are correct. Parameter key name consistency verification checks whether the parameter names in the quantization checkpoints are aligned with the parameter names of each module in the quantization skeleton model. Short sequence inference verification uses a small amount of input data to perform one or more forward inference operations on the reconstructed quantization model to check whether the inference is completed normally and the output results are within the expected range.

[0089] As an optional implementation, after loading, multiple verifications are performed on the reconstructed quantization model. First, structural consistency verification is performed, traversing the reconstructed quantization model to check if its module hierarchy, module types, and connection relationships are consistent with the quantization skeleton model. Next, parameter key name consistency verification is performed, traversing each parameter key name in the quantization checkpoints and comparing it one by one with the expected parameter key names of each module in the quantization skeleton model to confirm that all parameters have been correctly loaded and named in alignment. Finally, short-sequence inference verification is performed, using a small amount of calibration or test data to input into the reconstructed quantization model, performing one or more forward inference operations to check if the inference process completes normally, if there are any errors or anomalies, and if the numerical range of the output results is within the reasonable expected range. If any of the above verification checks fails, a verification failure message is output and the reason for the failure is recorded. If all verifications pass, the reconstructed quantization model is confirmed to be usable, and a verification pass message is output for subsequent deployment.

[0090] This embodiment constructs a quantization skeleton model based on the model configuration and quantization rules of the model to be quantized. The quantization weights and quantization parameters in the quantization checkpoints are loaded into the corresponding positions of each target module in the quantization skeleton model. The reconstructed quantization model is then subjected to structural consistency verification, parameter key name consistency verification, and short sequence inference verification, forming a complete quantization export and reconstruction chain. This makes the structural reconstruction of the quantization model reproducible, and the quantization model can be restored without relying on the original model. Problems such as rule conflicts, missing export items, or loading inconsistencies can be found during the export stage, effectively avoiding loading failures or inference anomalies after the quantization model is deployed. This significantly improves the reproducibility, deployability, and maintainability of the quantization model in engineering deployment.

[0091] Based on any of the above embodiments, Embodiment Nine of this application proposes a model module-level quantization derivation method, which may include steps A10 to A40 before step S10: Step A10: Traverse the internal structure of the model to be quantized and identify special structures in the model to be quantized.

[0092] In this embodiment, special structures refer to structural modules in the Transformer model other than regular linear projection layers, including but not limited to expert hybrid structures, shared expert structures, merged projection structures, and key-value cache quantization structures. Special structures have different organizational methods and parameter layouts within the model and need to be uniformly identified and processed before quantization.

[0093] As an optional implementation, before parsing the quantization rules, the internal structure of the model to be quantized is first traversed, recursively visiting each sub-module starting from the root module to detect whether there are special structures in the model. For each sub-module, it is determined whether it belongs to a special structure based on its type identifier, class name, or attribute information. The identified special structures are recorded in a special structure index table.

[0094] Optionally, specific detection methods include, but are not limited to: identification through the naming convention of module type, identification through whether the module contains expert container class attributes, identification through the dimensional relationship of the projection matrix, and identification through whether the attention module contains key-value cache related attributes.

[0095] Step A20: If the expert hybrid structure or the shared expert structure is identified, the gating projection, up projection and down projection contained in each expert in the expert hybrid structure or the shared expert structure are decomposed into independent logical expert projection targets, and a logical target identifier is assigned to each logical expert projection target.

[0096] In this embodiment, the logical expert projection target refers to a projection layer logic unit extracted from the expert hybrid structure or shared expert structure, which can be independently quantized and configured. Each expert includes three projection layers: gated projection, upper projection, and lower projection. Each projection layer has an independent weight matrix and needs to be identified as an independent quantization target. The logical target identifier is identification information used to uniquely identify each logical expert projection target, including the expert number and projection type.

[0097] As an optional implementation, when an expert hybrid structure is identified, each expert submodule within the structure is traversed. For each expert, its internal gated projection, up projection, and down projection are extracted as independent logical expert projection targets. During extraction, information such as the original weight matrix, input dimension, and output dimension of each projection layer is obtained, and a logical target identifier is assigned to each logical expert projection target. This identifier at least includes the expert number and projection type. For shared expert structures, the same method as for ordinary experts is used to extract their gated projection, up projection, and down projection as independent logical expert projection targets, and a logical target identifier containing the shared expert identifier is assigned. After decomposition, each logical expert projection target is uniformly included in the target module index table, enabling it to be processed normally by subsequent quantization rule matching and candidate quantization scheme generation processes, without needing to consider its nesting level in the original model.

[0098] Step A30: If the merged projection structure is identified, establish a mapping relationship between each split projection target constituting the merged projection structure and the merged projection target. The merged projection structure includes a query / key / value merged projection formed by merging query projection, key projection and value projection, or a gated / up merged projection formed by merging gated projection and up projection.

[0099] In this embodiment, the merged projection structure refers to a structure that combines multiple originally independent linear projection matrices into a larger projection matrix to improve inference performance. The split projection target refers to each independent projection layer before merging, while the merged projection target refers to the larger projection matrix after merging. The mapping relationship refers to associating the logical identifier of the split projection target, the position and dimensional range of its weight matrix in the merged matrix with the merged projection target, enabling quantization rules to establish a correspondence between the split projection targets and the merged projection target.

[0100] As an optional implementation, when a merged projection structure is identified, the original split projection type contained in the merged projection structure is determined. For query key-value merged projection, a mapping relationship is established between the three split projection targets (query projection, key projection, and value projection) and the merged projection target; for gated up-merged projection, a mapping relationship is established between the two split projection targets (gated projection and up-projection) and the merged projection target. In the mapping relationship, the starting position and dimension range of each split projection target in the merged projection matrix are recorded, so that when subsequent quantization rules are matched, even if the rule targets a split projection target, it can be correctly mapped to the merged projection target to perform quantization replacement.

[0101] Step A40: If the key-value cache quantization structure is identified, a key scale buffer and a value scale buffer are preset in the attention module to which the key-value cache quantization structure belongs. The key scale buffer and the value scale buffer are used to write the scale parameters of the key tensor and the value tensor during calibration.

[0102] In this embodiment, the key-value cache quantization structure refers to the module that performs quantization processing on the key-value cache during the autoregressive inference process. The key-scale buffer and value-scale buffer are tensor buffers pre-allocated in the attention module to store the quantization scale parameters of the key tensor and value tensor. The attention module is the core functional unit in the Transformer model that implements feature interaction based on the self-attention mechanism.

[0103] As an optional implementation, when a key-value cache quantization structure is identified, a key-scale buffer and a value-scale buffer are pre-defined in the attention module. These two buffers exist in the attention module as tensors and are not used during the calibration phase; they are only reserved for storage space. During subsequent calibration, when the attention module updates the key-value cache, the scale information of the key and value tensors is collected before the attention update, and the scale parameters are written into the corresponding key-scale buffer and value-scale buffer. When exporting quantization checkpoints, the scale parameters in the key-scale buffer and value-scale buffer are exported along with the model, allowing the inference phase to directly use these scale parameters to perform quantization processing on the key-value cache.

[0104] This embodiment identifies special structures such as expert hybrid structures, shared expert structures, merged projection structures, and key-value cache quantization structures by traversing the internal structure of the model to be quantized. It decomposes the gated projections, up projections, and down projections contained in each expert in the expert hybrid structure or shared expert structure into independent logical expert projection targets and assigns logical target identifiers. It establishes the mapping relationship between each split projection target and the merged projection target in the merged projection structure, and pre-sets key scale buffers and value scale buffers in the attention module to which the key-value cache quantization structure belongs, thus realizing the unified adaptation and export of special structures of complex Transformer models.

[0105] Based on any of the above embodiments, in conjunction with Embodiment Ten of this application... Figures 2 to 6 The overall system architecture and process are explained.

[0106] Optionally, refer to Figure 2The diagram illustrates the modular architecture of the present invention. The system includes a model loading module, a quantization rule parsing module, a model structure identification and adaptation module, a calibration data processing module, a target compression ratio planning module, an inference kernel perception and decision-making module, a calibration acquisition module, a quantization replacement and module materialization module, a checkpoint export module, a quantization skeleton reconstruction module, and a verification module. System inputs include the Transformer model to be quantized, the quantization rule file, calibration data, and information on the target hardware and inference framework. Specifically, the Transformer model to be quantized and its segmenter are read by the model loading module; the quantization rule file is read by the quantization rule parsing module and the rule standardization is performed; the calibration data is processed by the calibration data processing module for subsequent calibration acquisition; and the target hardware and inference framework information are transmitted to the target compression ratio planning module and the inference kernel perception and decision-making module as the basis for decision-making. The model structure identification and adaptation module performs a unified indexing of the target modules within the model and identifies special structures such as expert hybrid structures, shared expert structures, merged projection structures, and key-value cache quantization structures. The target compression ratio planning module and the inference kernel perception and decision-making module generate the final quantization scheme for each module based on quantization rules, target compression ratio, target hardware type, target inference framework, and module sensitivity. The calibration acquisition module collects the statistical information required for quantization under calibration data-driven conditions. The quantization replacement and module materialization module replaces the original modules with quantization modules. The checkpoint export module outputs quantization checkpoints and related quantization metadata. The quantization skeleton reconstruction module supports subsequent recovery of the quantization model from rules and checkpoints. The verification module checks the correctness and deployability of the exported results, ultimately outputting a reconstructable quantization model.

[0107] Optionally, refer to Figure 3This diagram illustrates a module-level rule-driven quantization mechanism. The system establishes a module path identifier for each quantizable target module and associates the module with the corresponding quantization rule through a module path matching expression. The diagram exemplifies the rule matching relationships for target modules such as query projection, key projection, and value projection in the attention module; gated projection in the feedforward module; and expert 1 upward projection and shared expert downward projection in the expert hybrid structure. The quantization rule set contains multiple rules, such as global default rules, attention module rules, key module coverage rules, and expert module rules, employing a sequential coverage mechanism where later rules cover earlier ones. Specifically, taking the key projection of the attention module as an example, the global default rule is matched first. If a more specific attention module rule or key module coverage rule exists subsequently, the later matched rule becomes the final effective rule, enabling differentiated quantization configuration for key modules without requiring separate rule writing for each module. After rule matching, the system outputs a final quantization scheme for each target module, including at least one of 8-bit floating-point quantization, 4-bit floating-point quantization, 4-bit weighted quantization, and high-precision preservation. A corresponding quantization backend is assigned to each quantization scheme, allowing each module to complete quantization export under its respective final quantization scheme.

[0108] Optionally, refer to Figure 4 This paper illustrates the quantization planning mechanism under the constraint of a target compression ratio. For each target module in the target module set, such as module 1, module 2, module 3, up to module N, a corresponding set of candidate quantization schemes is generated. Specifically, the candidate quantization schemes for module 1 include floating-point schemes and scheme A (8-bit weighted quantization); the candidate quantization schemes for module 2 include scheme C (4-bit weighted quantization); the candidate quantization schemes for module 3 include scheme D (high-precision preservation); and the candidate quantization schemes for module N include scheme A (8-bit weighted quantization). The system evaluates the cost of each candidate quantization scheme, calculating its storage cost, precision loss, execution latency, and compatibility. Storage cost reflects the compression effect of the scheme on the model size; precision loss reflects the impact of the scheme on model accuracy; execution latency reflects the actual inference efficiency of the scheme under the target hardware and inference framework; and compatibility reflects whether the scheme has efficient inference kernel support in the target deployment environment. If a scheme lacks efficient kernel support in the target environment, its compatibility score is reduced accordingly. After obtaining the above-mentioned indicators for each candidate scheme of each target module, the system performs joint planning with the constraint that the total storage cost of the whole model does not exceed the storage budget corresponding to the target compression rate, and with the goal of minimizing the weighted sum of the total accuracy loss and inference latency of the whole model. Finally, it outputs the quantitative scheme allocation results of each target module: Module 1 is allocated scheme A, Module 2 is allocated scheme C, Module 3 is allocated scheme D, and Module N is allocated scheme A, so that the whole model can achieve joint optimization of accuracy loss and inference latency while meeting the overall compression rate target.

[0109] Optionally, refer to Figure 5 This diagram illustrates the precision-latency joint decision-making process driven by the inference kernel. Based on the target hardware, target inference architecture, and the mapping relationship between each candidate quantization backend and the inference kernel, the system determines the availability of each quantization backend on the target hardware and the corresponding underlying inference kernel type within the target inference framework. For available kernel types, the system further obtains the weighting and packaging methods, block sizes, scale layouts, and data organization methods supported by each kernel, as well as the corresponding expected execution latency and precision impact. When determining a quantization scheme, the system compares not only the theoretical compression ratio and static error of candidate quantization schemes but also their actual execution cost under the target inference kernel, achieving a precision-latency joint decision-making process. For example, for a low-bit quantization scheme, if it has a higher compression ratio but lacks efficient underlying kernel support in the target hardware and inference framework, the latency or compatibility cost of the scheme is increased to reduce its selection probability. During the export phase, the system generates a kernel-friendly export layout for the target module based on the final quantization backend and the corresponding inference kernel requirements. This includes the packaging order of weight tensors, block size, organization of scale parameters, and whether to use merged or separate storage. The system also writes the expected latency, accuracy impact, and layout information related to the inference kernel into the quantization checkpoint metadata. This allows the quantization model to directly construct the corresponding execution path based on the export metadata during subsequent loading, without needing to perform format conversion again.

[0110] Optionally, refer to Figure 6This demonstrates a unified adaptation and export process for complex structures. For expert hybrid structures and shared expert structures, the system abstracts the gated projection, up projection, and down projection of Expert 1, Expert 2, and the shared expert into independent logical quantization targets. The projection parameters of each expert are written into checkpoints according to a unified naming convention to ensure consistency between the saved results and the logical targets. For the merged projection module, the system establishes a correspondence between query / key / value merged projection and the gated / up projection merge module. When the splitting projection rules in the same merge group are consistent, the splitting rules are automatically merged and applied to the merged projection module. When the splitting rules are inconsistent, the exported configuration is rejected to avoid quantization semantic conflicts. For the key-value cached quantization structure, the system pre-sets the key tensor scale and value tensor scale in the attention module. During the calibration phase, the system collects the scale information of the key tensor and value tensor and performs quantization processing according to a unified logical quantization target. This allows the key-value cached quantization structure to share the same unified logical quantization target framework with the expert hybrid structure, shared expert structure, and merged projection module. In the above manner, the expert hybrid structure, the shared expert structure, the merged projection module, and the key-value cache quantization structure are all mapped to a unified logical quantization target framework. Then, through unified naming export and quantization skeleton reconstruction, reconstructable quantization model checkpoints are generated, enabling the inference stage to directly use these scale parameters to perform quantization processing on the key-value cache.

[0111] This application provides a model module-level quantization export device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the model module-level quantization export method in the above embodiment 1.

[0112] The following is for reference. Figure 7 The diagram illustrates a structural schematic suitable for implementing the model module-level quantization export device in the embodiments of this application. The model module-level quantization export device in the embodiments of this application may include, but is not limited to, computing devices with data processing and program execution functions such as servers, personal computers, laptops, workstations, and cloud computing nodes. Figure 7 The model module-level quantization export device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0113] like Figure 7As shown, the model module-level quantization export device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The random access memory 1004 also stores various programs and data required for the operation of the model module-level quantization export device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the model module-level quantization export device to communicate wirelessly or wiredly with other devices to exchange data. Although model module-level quantization export devices with various systems are shown in the figures, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented alternatively.

[0114] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0115] The model module-level quantization export device provided in this application, employing the model module-level quantization export method described in the above embodiments, can solve the technical problem that a unified quantization scheme struggles to simultaneously address the differences in module sensitivity and inference environment efficiency. Compared with the prior art, the beneficial effects of the model module-level quantization export device provided in this application are the same as those of the model module-level quantization export device provided in the above embodiments, and other technical features in this model module-level quantization export device are the same as those disclosed in the previous embodiment method, and will not be elaborated upon here.

[0116] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0117] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0118] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the model module-level quantization derivation method in the above embodiments.

[0119] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), or any suitable combination thereof.

[0120] The aforementioned computer-readable storage medium may be included in the model module-level quantization export device; or it may exist independently and not be assembled into the model module-level quantization export device.

[0121] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the model module-level quantization export device, enable the model module-level quantization export device to: determine, according to the quantization rules, the set of candidate quantization schemes corresponding to each target module in the model to be quantized; With the target compression rate as a constraint, and based on the target hardware type and the inference kernel support corresponding to the target inference framework, a target quantization scheme is determined for each target module from the candidate quantization scheme set; according to the quantization statistics of each target module and the target quantization scheme, each target is replaced with the corresponding quantization module, and quantization checkpoints are derived.

[0122] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0123] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0124] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0125] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described model module-level quantization derivation method. This solves the technical problem that a unified quantization scheme struggles to balance module sensitivity and inference environment efficiency differences. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the model module-level quantization derivation method provided in the above embodiments, and will not be elaborated upon here.

[0126] This application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the model module-level quantization derivation method described above.

[0127] The computer program product provided in this application can solve the technical problem that a unified quantization scheme is unable to simultaneously address the differences in module sensitivity and inference environment efficiency. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the model module-level quantization derivation method provided in the above embodiments, and will not be elaborated upon here.

[0128] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent scope of this application.

Claims

1. A method for quantization export at the model module level, characterized in that, The model module-level quantization export method includes: Based on the quantization rules, determine the set of candidate quantization schemes corresponding to each target module in the model to be quantized; With the target compression ratio as a constraint, and based on the target hardware type and the inference kernel support of the target inference framework, a target quantization scheme is determined for each target module from the set of candidate quantization schemes; Based on the quantization statistics of each target module and the target quantization scheme, each target is replaced with the corresponding quantization module, and quantization checkpoints are derived.

2. The model module-level quantization export method as described in claim 1, characterized in that, The step of determining the set of candidate quantization schemes corresponding to each target module in the model to be quantized according to the quantization rules includes: Identify the target modules in the model to be quantized and establish a module path index for each target module; The quantization rule file is parsed to obtain the quantization rule, which includes the module path matching expression and its corresponding quantization method identifier, quantization backend type, and quantization backend parameters. Based on the matching result between the module path matching expression and the module path index, a set of candidate quantization schemes for each target module is determined.

3. The model module-level quantization export method as described in claim 1, characterized in that, The step of determining a target quantization scheme for each target module from the candidate quantization scheme set, constrained by the target compression ratio and based on the target hardware type and the inference kernel support corresponding to the target inference framework, includes: For each candidate quantization scheme of each target module, calculate the storage cost index, accuracy loss index, and inference latency index of the target module under the candidate quantization scheme, as well as the inference latency index under the target hardware type and the target inference framework. Based on the target hardware type and the target inference framework, query the inference kernel mapping table to determine whether each candidate quantization scheme has a corresponding underlying inference kernel. If not, increase the inference latency index of the candidate quantization scheme or mark it as unavailable. With the constraint that the total storage cost of the entire model does not exceed the storage budget corresponding to the target compression rate, and with the objective of minimizing the weighted sum of the total accuracy loss and inference latency of the entire model, the target quantization scheme of each target module is obtained through joint planning.

4. The model module-level quantization export method as described in claim 3, characterized in that, The calculation of the target module's storage cost metric, accuracy loss metric, and inference latency metric under the candidate quantization scheme, and under the target hardware type and the target inference framework, includes: The storage cost index is calculated based on the original weight volume of the target module and the quantization bit width of the candidate quantization scheme. The model to be quantized is subjected to forward computation using calibration data to obtain the output tensor of the target module before quantization; simulated quantization corresponding to the candidate quantization scheme is applied to the weights of the target module, and forward computation is performed again to obtain the output tensor after quantization; the accuracy loss index is determined based on the difference between the output tensor before quantization and the output tensor after quantization. The inference kernel mapping table is used to obtain the estimated execution latency of the underlying inference kernels corresponding to the target hardware type and each quantization backend under the target inference framework, which are stored in advance, as the inference latency index. If the inference kernel mapping table does not contain latency data corresponding to the candidate quantization scheme, the inference latency index is estimated based on the theoretical computational cost of the candidate quantization scheme.

5. The model module-level quantization export method as described in claim 3, characterized in that, The method, which constrains the total storage cost of the entire model to not exceed the storage budget corresponding to the target compression rate, and aims to minimize the weighted sum of the total accuracy loss and inference latency of the entire model, jointly plans the target quantization scheme for each target module, including: Calculate the sensitivity index for each target module, the sensitivity index being determined based on the accuracy loss index; sort the target modules according to the sensitivity index from high to low; assign candidate quantization schemes to each target module in order of ranking, assigning candidate quantization schemes with smaller accuracy loss to target modules with higher sensitivity indices, until the total storage cost index of the entire model meets the storage budget corresponding to the target compression ratio; or... The candidate quantization schemes for each target module are modeled as a combinatorial optimization problem. The storage cost index is used as a constraint, and the weighted sum of the accuracy loss index and the inference latency index is used as the objective function. Integer programming or dynamic programming is used to solve the problem, and the target quantization schemes for each target module that minimize the objective function are obtained.

6. The model module-level quantization export method as described in claim 1, characterized in that, The step of replacing each target with a corresponding quantization module based on the quantization statistics of each target module and the target quantization scheme, and deriving quantization checkpoints, includes: Forward calculations are performed using calibration data to collect quantitative statistical information for each target module; Based on the target quantization scheme and the quantization statistics, each target module is replaced with the corresponding quantization module to obtain the quantized model; The quantization checkpoints are derived from the quantized model, and the quantization checkpoints include at least the quantized weight parameters and the quantization parameters.

7. The model module-level quantization export method as described in claim 1, characterized in that, After deriving the quantization checkpoint, the method further includes: Based on the model configuration of the model to be quantized and the quantization rules, a quantization skeleton model is constructed, wherein each target module in the quantization skeleton model is a quantization module without loaded quantization weights. The quantization weights and quantization parameters in the quantization checkpoints are loaded into the corresponding positions of each target module in the quantization skeleton model to obtain the reconstructed quantization model. The reconstructed quantization model is validated, and the validation includes at least one of the following: structural consistency validation, parameter key name consistency validation, and short sequence inference validation.

8. The model module-level quantization export method as described in claim 1, characterized in that, Before determining the set of candidate quantization schemes corresponding to each target module in the model to be quantized according to the quantization rules, the method further includes: Traverse the internal structure of the model to be quantized and identify special structures in the model to be quantized. The special structures include at least one of the following: expert hybrid structure, shared expert structure, merged projection structure, and key-value cache quantization structure. If the expert hybrid structure or the shared expert structure is identified, the gated projection, up projection and down projection contained in each expert in the expert hybrid structure or the shared expert structure are decomposed into independent logical expert projection targets, and a logical target identifier is assigned to each logical expert projection target. If the merged projection structure is identified, a mapping relationship is established between each split projection target constituting the merged projection structure and the merged projection target. The merged projection structure includes a query / key / value merged projection formed by merging query projection, key projection and value projection, or a gated / up merged projection formed by merging gated projection and up projection. If the key-value cache quantization structure is identified, a key scale buffer and a value scale buffer are preset in the attention module to which the key-value cache quantization structure belongs. The key scale buffer and the value scale buffer are used to write the scale parameters of the key tensor and the value tensor during calibration.

9. A model module-level quantization export device, characterized in that, The model module-level quantization export device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the model module-level quantization export method as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the model module-level quantization derivation method as described in any one of claims 1 to 8.

Images