AWQ model optimization algorithm fusing weight quantization perception training

Abstract

The invention discloses an AWQ model optimization algorithm fused with weight quantization perception training, particularly relates to the technical field of deep learning model quantization, and is used for solving the problem that in large-scale neural network reasoning deployment, a weight quantization unified constraint and dynamic optimization mechanism oriented to specific business scenes and hardware resource constraints is lacked, and the algorithm is not suitable for large-scale neural network reasoning deployment. The model performance degradation is uncontrollable and the resource utilization rate is not high under the limitation of time delay, power consumption and storage. A quantization constraint template is constructed by collecting task features and resource features in a target business scene, and the bit number, the scaling factor and the grouping mode are dynamically adjusted according to a weight group and a significant channel under the linkage control of dual-channel training, a quantization track vector and a quantization difficulty graph. Therefore, the task performance of the model is kept basically stable within given time delay, power consumption and storage constraints, and the core problems that performance degradation is uncontrollable and fine optimization is difficult to face specific scenes after existing static weight quantization is deployed are solved.

Description

Technical Field

[0001] This invention relates to the field of deep learning model quantization technology, specifically to an AWQ model optimization algorithm that integrates weighted quantization-aware training. Background Technology

[0002] In current large-scale neural network inference deployments, weight quantization has become a common method to reduce latency, power consumption, and storage footprint. Taking common post-training weight quantization methods as an example, they typically use a small set of general calibration samples to statistically analyze the magnitude distribution of weights in each layer, search for scaling factors or pruning thresholds with a fixed number of bits, and generate a one-time quantization configuration. The advantages of this approach are its simplicity and low overhead. However, quantization decisions are often disconnected from specific business scenarios, rarely incorporating information such as task type, context length distribution, and user terminal differences. They also rarely explicitly model the actual constraints of the target deployment hardware in terms of latency, power consumption, and memory. While some publicly available solutions claim to be "hardware-constraint conscious," they often remain at a coarse-grained device-level limitation level. For example, given a uniform latency or memory limit, they apply a uniform number of bits or a few preset combinations to the entire model. This fails to reflect the differences in quantization difficulty and resource sensitivity between different layers and weight regions. In actual deployment, situations can still easily arise where some key layers are over-compressed, leading to a sudden drop in quality, or the overall configuration is too conservative, resulting in low resource utilization.

[0003] Some technical approaches attempt to introduce quantization-aware training, which considers both task loss and quantization error during the training phase, allowing for fine-tuning of the quantized model to mitigate performance degradation caused by pure post-training quantization. However, these solutions typically operate at the overall model or single-layer level, relying mainly on one-time or limited comparisons of loss values ​​when evaluating quantization effectiveness. They lack fine-grained characterization of the behavioral changes of specific weights throughout the training process. Modifications to quantization strategies often depend on human experience or a few rounds of experimentation, rarely establishing a unified management chain around real business traffic, hardware cost assessment, and the evolution of quantization configurations. Especially in engineering environments with multiple model versions, multiple business scenarios, and multiple types of hardware, existing solutions generally lack unified constraint templates for scenario and hardware combinations, version locking and idempotent management of weight quantization configurations, and a systematic evidence chain from training to deployment to rollback. Many configuration adjustments remain at the script level or are manually recorded, leading to discrepancies between online performance and offline evaluation, and difficulty in promptly locating specific quantization decisions when problems occur.

[0004] In summary, while existing technologies can reduce inference costs to some extent by utilizing weight quantization, they generally suffer from a key problem after convergence: given specific business scenarios and deployment hardware constraints, there is a lack of a mechanism to build a unified constraint template around the entire weight quantization process and to stably and traceably manage the weight quantization strategy at each stage of training, evaluation, and deployment. As a result, it is difficult to stably obtain a quantization model configuration with controllable task performance within the constraints of latency, power consumption, and storage. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides an AWQ model optimization algorithm that integrates weighted quantization-aware training to solve the problems mentioned in the background section.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution: an AWQ model optimization method incorporating weighted quantization-aware training, comprising:

[0009] S1. Collect sample and task information based on the target business scenario, and generate quantitative constraint templates for constraint latency, power consumption and storage based on the computing cost and storage cost of the deployed hardware;

[0010] S2. Using calibration samples, statistically analyze the activation distribution of each layer on the model, identify significant channels, determine the number of bits, scaling factor and grouping method of each weight group, and obtain the initial configuration of weight quantization.

[0011] S3. Construct a dual-path training graph under the quantization constraint template and initial configuration of weight quantization. Simultaneously execute the floating-point path and quantization path for each batch, and construct a composite loss including task loss, layer representation difference and resource regularization term on the quantization path.

[0012] S4. During the training process, record the weight difference, activation difference and gradient norm before and after quantization for each weight group at intervals, and update the quantized trajectory vector of the weight group based on the time series.

[0013] S5. Calculate the difficulty score of the weight group based on the quantized trajectory vector to generate a quantized difficulty map. Adjust the number of bits, scaling factor and grouping method of the weight group under the quantization constraint template, and update the set of significant channels.

[0014] S6. After training iteration and quantization difficulty graph update, when the task performance and resource indicators meet the preset conditions, freeze the quantization configuration, quantize the model weights according to the quantization configuration, and export the deployed model.

[0015] Furthermore, S1 includes:

[0016] Assign scenario identifiers to target business scenarios, collect call logs and task records in the observation window, align task fields, latency, and power in the task records to a unified time axis, and statistically analyze latency and power according to the observation window and remove abnormal records.

[0017] Based on the computational cost and video memory usage registered in the deployment hardware capability library, the latency limit, power limit, and video memory limit are determined according to service level constraints;

[0018] Register quantized constraint templates containing bit ranges for each layer, grouping granularity ranges, and resource budget limits in the configuration library, using scene identifiers and hardware identifiers as keys.

[0019] The template version number and effective time limit the existence of one effective template for the same scenario and hardware combination.

[0020] Furthermore, S2 includes:

[0021] In the target business scenario, extract anonymized historical calls in a hierarchical manner according to task type and context length to form a calibration sample set;

[0022] Record the activation values ​​organized by layer index and channel index on the model to be deployed, generate a significant channel set according to the energy proportion threshold and write it into the configuration library;

[0023] Based on the range of bit counts and grouping granularity for each layer given by the quantization constraint template, the weights of each layer are divided into weighted groups;

[0024] Specify the number of bits, scaling factor, and grouping method for each weighted recombination, and register the association with the quantization constraint template in the configuration library;

[0025] The training node obtains the corresponding initial configuration for weight quantization through the model version number, scene identifier, and hardware identifier.

[0026] Furthermore, S3 includes:

[0027] When constructing a dual-path training topology, the initial configuration of the training nodes is based on the quantization constraint template and weight quantization. For parallel execution of mini-batch samples, a floating-point path with full-precision weights and activation and a quantization path with quantized weights are used.

[0028] In the quantization path, the task loss, the key layer representation difference, and the resource consumption penalty based on computational overhead and memory usage are calculated. The three are then combined into a composite loss according to the loss weight configuration registered in the configuration library. The composite loss drives the update of model weights and quantization-related parameters.

[0029] Furthermore, S4 includes:

[0030] During the dual-path training topology, a quantization behavior record is generated for each weight reorganization at fixed step intervals. The quantization behavior record includes the current training step, the weight difference statistics before and after quantization, the activation difference statistics, and the gradient norm statistics.

[0031] After smoothing and truncation, the quantized trajectory vectors are combined in chronological order within the trajectory observation window to form the weighted group. The quantized trajectory vectors are then stored in the trajectory library using the unique identifier of the weighted group, the starting training step number of the trajectory observation window, the ending training step number of the trajectory observation window, and the value summary as indexes.

[0032] Furthermore, S5 includes:

[0033] When updating the quantization strategy, at the end of the strategy update cycle, the quantized trajectory vector in the corresponding trajectory observation window is read from the trajectory library for each weighting.

[0034] The quantization difficulty score is calculated based on the weight difference statistics before and after quantization, the activation difference statistics, and the gradient norm statistics.

[0035] Calculate resource sensitivity scores based on hardware capability descriptions and quantitative constraint templates;

[0036] A quantitative difficulty graph is generated by using layer-based indexing and weighted reorganization indexing, and the quantitative difficulty score, resource sensitivity score, and graph version number are registered in the configuration database.

[0037] Furthermore, when adjusting the weight quantization configuration based on the quantization difficulty map, if the quantization difficulty score and resource sensitivity score of a certain weight group are both greater than the first threshold, the number of bits in that weight group is increased and the grouping granularity is reduced.

[0038] When all scores are less than the second threshold, the number of bits in the weighting group is reduced and the grouping granularity is increased. Based on this, the significant channel set is updated and a new version of the weight quantization configuration is generated.

[0039] The number of bits, scaling factor, grouping method, and significant channel markers recorded in the new version are associated with the quantization difficulty map version number and the previous weight quantization configuration version number and stored together.

[0040] After adjustment, resource requirements are assessed based on the quantitative constraint template. If the resource requirements exceed the upper limit of the resource budget, the system will revert to the previous weighted quantitative configuration version and record the resource over-limit error flag.

[0041] Furthermore, S6 includes:

[0042] Based on the target business scenario, construct a representative task set, and within a preset inspection interval, configure and run the representative task set with the current weight and record task indicators and resource indicators.

[0043] Compare the task metrics with the preset task performance lower limit, and compare the resource metrics with the resource budget upper limit given by the quantitative constraint template;

[0044] When the task indicators meet the preset task performance lower limit and the resource indicators meet the resource budget upper limit given by the quantitative constraint template within the continuous inspection period, the current weight quantitative configuration is marked as frozen quantitative configuration.

[0045] Based on the frozen quantization configuration, the corresponding model weights are quantized to generate a deployment model, and the deployment model and its frozen quantization configuration version number are registered in the model repository.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. By collecting task and resource features in the target business scenario to construct a quantization constraint template, and under the linkage control of dual-path training, quantization trajectory vector and quantization difficulty map, weighted recombination and dynamic adjustment of bit number, scaling factor and grouping method of significant channels, the model task performance is kept basically stable within the given latency, power consumption and storage constraints. This solves the core problem that the performance degradation of existing static weight quantization is uncontrollable after deployment and it is difficult to finely optimize for specific scenarios.

[0048] 2. By implementing version locking and idempotent management of quantization constraint templates, salient channel sets, quantization trajectory vectors, quantization difficulty maps, and frozen quantization configurations throughout the entire quantization perception training process, and by combining periodic on-site evaluations of representative task sets to drive configuration freezing and deployment rollback, the automatic evolution and traceable implementation of quantization strategies are achieved. This reduces the engineering burden of repeated manual parameter tuning and trial-and-error deployment, and improves the operational stability and compliance verifiability of the model in complex business environments. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating the AWQ model optimization algorithm that integrates weighted quantization perceptual training according to the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example: An AWQ model optimization method incorporating weighted quantization-aware training, including:

[0052] S1. Collect sample and task information based on the target business scenario, and generate quantitative constraint templates for latency, power consumption, and storage based on the computational and storage costs of the deployed hardware. The specific implementation is as follows:

[0053] The system preferably first collects operational data for the target business scenario of the planned deployment model. The target business scenario represents a combination of inference services under a fixed application domain, fixed access point, and fixed user group. Each scenario is uniquely identified by a scenario identifier. The system continuously collects call logs and task records generated within this scenario over a period of time. The call logs record the fact that the model service is accessed, and the task records provide a structured description of each inference task.

[0054] The task log preferably records at least the session identifier, request time, task type identifier, context length, generation length, user terminal category, region identifier, and latency and power obtained from on-site measurement. The session identifier is used to distinguish different session links, the request time is used to describe the time point when the task was initiated, the task type identifier is used to distinguish different task categories such as dialogue generation, code generation, and knowledge question answering, the context length and generation length can be set as length indicators in terms of the number of tags, the user terminal category is used to characterize the type of terminal from which the task comes, the region identifier is used to characterize the approximate geographical range of the task source, the latency is preferably in milliseconds, the time interval from the task arriving at the system to the result being returned is recorded by the monitoring module, and the power is preferably in watts, obtained by a power monitoring device near the deployment node sampling at a fixed rhythm.

[0055] To ensure consistency in the time dimension of the logs and records mentioned above, the system pre-defines an observation window for each type of scenario in the configuration. The observation window can be set to a continuous time interval of several hours, several days, or several weeks. Within the observation window, the call logs and task records are organized in chronological order, and timestamps from different sources are aligned to a unified time axis. When there are missing records, interpolation of adjacent time slices is preferred to fill in the gaps. Records outside the two ends of the observation window are not included in this statistical analysis.

[0056] For extreme values ​​that deviate significantly from the overall level within the observation window, such as records with latency much higher than normal service latency or sudden power spikes, the system can set several percentile positions as references. Records that exceed the upper percentile or fall below the lower percentile are considered outliers, and preferably they are reduced in weight for statistical participation or directly removed. At the same time, upper and lower limits are set for key indicators to ensure that subsequent calculations are carried out within a limited range, thereby avoiding a few extreme cases from dominating the resource budget.

[0057] After organizing the operational characteristics of the business side, the system generates a description of the computational and storage costs based on the capability boundaries of the target deployed hardware. The deployed hardware is uniquely identified by a hardware identifier, which can correspond to a specific type of server, accelerator card, or inference cluster configuration.

[0058] The system compiles estimates of the computational cost, available memory capacity, and peak power required for typical operators under different bit counts and memory widths, based on technical specifications provided by hardware manufacturers and results from pre-conducted stress tests. For example, for matrix multiplication, convolution, and self-attention operations, the system can provide the time overhead and upper bound of memory usage for each layer under different bit counts. These computational and storage costs are preferably stored in the hardware capability library as configuration records. Fields may include operator type, layer index, candidate bit count, estimated computational cost, estimated memory usage, and estimated peak power, along with a version number to ensure that the old and new versions can be distinguished should the hardware capability assessment method or measurement means change.

[0059] Within the observation window corresponding to the target scenario, the system statistically analyzes the latency and power in the task records according to a preset time scale. This can be set to statistically analyze the time series by day or week, obtaining the latency and power distribution at each time scale. Preferably, the mean, most percentile values, and extreme value ranges are calculated. Combined with the service level constraints promised by the service provider in this scenario, acceptable latency, power, and video memory limits are set for this scenario. The latency limit can be set to a certain percentile latency not exceeding a certain number of milliseconds; the power limit can be set to a peak power within the observation window not exceeding a certain number of watts; and the video memory limit can be set to ensure that the video memory used during inference does not exceed a certain percentage of the available hardware video memory.

[0060] After understanding the aforementioned business load characteristics and hardware capability boundaries, the system generates a quantization constraint template. This template provides unified constraints for subsequent weight quantization configuration and is preferably stored in a configuration database as a configuration table. The configuration database can be a centralized configuration management database. Each configuration table records at least the scenario identifier, hardware identifier, allowed bit range for each layer, allowed grouping granularity range for each layer, upper limit of resource budget for that combination, template version number, and effective date.

[0061] The bit count range is used to define the range of bit count values ​​that can be used in subsequent weight quantization. For example, it can be set as a set of discrete values ​​from low to medium bits. The grouping granularity range is used to define the upper and lower bounds of the number of parameters that each weighting set can include. The resource budget upper limit is used to comprehensively characterize the upper limit of average latency, power consumption, and video memory under this scenario and hardware combination. The template version number is used to lock the version of templates formed under different calculation methods, service strategies, or observation periods. The effective time is used to describe the applicable range of the template on the timeline. Preferably, an expiration time or alternative template version number can also be configured to achieve smooth switching during business evolution.

[0062] To ensure that only one effective template exists for the same scene and hardware combination within any given time period, the system uses the combination of scene identifier and hardware identifier as the idempotent key. In the configuration library, it limits that at most one record corresponding to this idempotent key has an effective state at any given time. When a new template version is generated and ready for use, the effective time interval of the old version is adjusted to switch it to the historical state, thereby avoiding parallel activation.

[0063] Regarding interaction with external modules, the system provides template query capabilities for scheduling units, which can be management components responsible for organizing training and deployment tasks. When initiating a query request, the scheduling unit carries a scenario identifier and a hardware identifier. Based on this combination, the system searches the configuration library for currently effective quantization constraint templates using an idempotent key. If a record matching the time condition is found, all constraint parameters and the template version number are returned for unified reference in subsequent weight quantization configuration, training scheduling, and deployment decisions. If no effective template with the idempotent key identifier is found in the configuration library, the system preferably returns a preset error identifier. The error identifier can be set as a set of enumerable numbers or text descriptions, and the requester's identifier, request initiation time, scenario identifier and hardware identifier carried in the request, and the fact that no template was found are recorded in the audit log.

[0064] Audit logs are used to provide a chain of evidence for subsequent problem investigation and accountability. They are preferably appended to read-only storage media and archived along with the system and template versions. To protect personal information security and meet compliance requirements, the business statistics and resource budgets involved in the quantitative constraint templates and audit logs are aggregated only at the scenario and hardware levels, without recording specific terminal identifiers or natural person-related identification information. Session identifiers can be saved in the logs using anonymization methods, such as irreversible hashing, to ensure that a single user's behavior trajectory cannot be recovered from the templates and logs.

[0065] The strategy rules and parameter configurations for template generation and updating are also version locked. The system assigns a version number to each rule and parameter set in the configuration library. The quantitative constraint template records the version information of the rules and parameters it references. This approach allows for a quick retrospective of the business observation window, statistical methods, filtering strategies, and service level constraints used at the time when the template generation logic is evaluated in the future, based on the template version number, and forms a complete chain of evidence. This facilitates explaining to the reviewing agency the construction capability and scope of application of the quantitative constraint template in this embodiment.

[0066] S2. Using calibration samples, statistically analyze the activation distribution of each layer on the model, identify significant channels, determine the number of bits, scaling factor, and grouping method for each weight group, and obtain the initial configuration for weight quantization. The specific implementation is as follows:

[0067] After obtaining the quantization constraint template based on the target business scenario, the system preferably organizes historical calls within the same business scenario to construct a calibration sample set and generate the initial configuration for weight quantization. Historical calls are used to indicate the inference task records actually carried by the model to be deployed within a certain time window. This time window can be set to several days or several weeks to ensure that it can cover business fluctuations without introducing outdated scenarios.

[0068] To ensure the representativeness and controllability of the calibration samples, the system performs stratified extraction based on task type, context length, and user terminal category. Task type distinguishes different uses such as dialogue generation, code generation, and knowledge-based question answering; context length characterizes the scale of the prompt information; and user terminal category characterizes the type of device from which the access originates, such as a mobile terminal, desktop terminal, or edge node. Within each stratum, the system extracts several corpora from historical calls as candidate samples. During the extraction process, it uses the session identifier as the primary key to identify duplicate requests within the same session, merging duplicate requests or retaining only one, ensuring that the same session generates only a limited number of representative records in the calibration set.

[0069] To reduce the risk of personal information leakage, the system performs an anonymization strategy before the corpus is stored on disk, retaining only fields directly related to model inference, such as anonymized text content, task type identifiers, and length indicators, and removing fields that are likely to point to natural persons, such as user identifiers, precise timestamps, and geographical locations, or performing irreversible hashing on them, so that subsequent stages cannot recover a single user's behavior from the calibration sample.

[0070] For records with severe field missingness, context length or generated length significantly exceeding the preset reasonable range, containing illegal characters, or deemed unsuitable for training by the compliance module, the system preferably removes them from the candidate set, thereby forming a calibration sample set with controllable quality within a limited batch size and time window. The calibration sample set refers to a representative set of samples used to estimate the internal statistical characteristics of the model in the target business scenario. Its capability lies in covering the main task types, length distributions, and terminal categories, and its scope is jointly defined by the stratified sampling strategy and filtering rules.

[0071] After the calibration sample set is prepared, the system calls the model to be deployed in the offline environment to perform multiple batches of inference on the set. The model to be deployed is uniquely identified in the model repository by its version number. Each inference records the activation values ​​within each layer of the model. The activation values ​​are used to characterize the intermediate representations generated by each layer under a given input, and can preferably be regarded as the numerical response of the corresponding layer in the channel dimension.

[0072] The system uses layer and channel indices as coordinates to perform amplitude distribution statistics on the activation values ​​of each channel in each layer across multiple batches. The amplitude distribution can be set as a frequency distribution over several fixed intervals, or as a combination of several percentile values ​​with the mean and variance. Based on the amplitude distribution, the system calculates the energy proportion of each channel in the overall activation energy of the layer. This energy proportion describes the importance of the channel in the layer's response. Simultaneously, it determines whether a channel needs priority protection based on a preset threshold. The preset threshold can be set to an energy proportion not lower than a certain percentage or the high percentile of the activation amplitude not lower than a certain value. Channels that meet the threshold conditions are marked as significant channels, forming a set of significant channels.

[0073] The salient channel set is used to centrally represent channels that have a significant impact on task performance under calibration samples. Its capability lies in guiding more conservative configurations for important channels during subsequent weight quantization. Its scope is jointly determined by the threshold rules and the calibration sample range. The salient channel set is stored in a configuration repository in tabular form. This repository is a persistent storage for centrally managing various configuration records. Each record contains at least a layer index, a channel index, a task type identifier, a salient marker, a threshold rule version number, and its own version number. The layer index uniquely identifies a layer in the model, the channel index uniquely identifies a channel within that layer, the salient marker distinguishes between salient and non-salient channels, the threshold rule version number indicates the definition of the energy percentage threshold or amplitude threshold used for this judgment, and the own version number distinguishes salient channel configurations generated under different calibration samples or different threshold rules. Through this method, the generation process and definition of the salient channel set can be fully traced during subsequent review or maintenance.

[0074] After obtaining the significant channel set, the system constructs an initial configuration for weight quantization of the model weights based on the quantization constraint template under the same scenario. The quantization constraint template provides the allowed bit range and grouping granularity range for each layer. The bit range is used to limit the set of bit widths that can be used in subsequent quantization, and the grouping granularity range is used to limit the upper and lower bounds of the scale of each weight regroup in the parameter dimension.

[0075] The system divides the weights of a layer into several weight reassemblies along a predetermined dimension. Each weight reassembly can be set as a subset consisting of several channels or several blocks. The weight reassembly serves as the configuration unit for the number of bits and scaling factor during subsequent quantization. Its capability lies in reducing storage and computational overhead by sharing scaling factors through grouping while ensuring numerical accuracy. Its range is jointly determined by the structure of the weight tensor, the grouping strategy, and the range of grouping granularity.

[0076] For each weighted reassembly, the system considers whether the channels covered by the group are salient channels, the weight of the layer in the hardware cost, and the upper and lower limits of the number of bits specified by the quantization constraint template. It then selects an initial number of bits from the candidate bit set and calculates a scaling factor for the group based on the amplitude statistics of the weights in the calibration samples. This scaling factor is used to map the original weights to the quantization integer domain. Simultaneously, the grouping method used for this weighted reassembly is recorded, including the grouping dimension and the number of elements within each group. All the above configurations for all weighted reassemblies together constitute the initial configuration for weight quantization.

[0077] The initial configuration for weight quantization is stored in a configuration database as a configuration table. Each record contains at least the layer index, weight reassembly index, number of bits, scaling factor, grouping method, significant channel markers associated with that weight reassembly, referenced quantization constraint template version number, and its own version number. The referenced quantization constraint template version number indicates the resource budget and bit count range constraints under which the initial configuration was generated, while the own version number distinguishes different configurations obtained under different calibration samples, different significant channel sets, or different grouping strategies. By storing both the template version number and the own version number in the initial configuration record, a clear version lock relationship can be established between the template version and the initial configuration version, ensuring that the original quantization constraint criteria can be accurately restored through version association during subsequent training and deployment phases.

[0078] To ensure that the initial weight quantization configuration and quantization constraint template used during training are consistent with the model structure, target scene, and hardware configuration, the training node sends a configuration retrieval request to the configuration library management module via remote call before startup. This request includes at least the model version number, scene identifier, and hardware identifier. The configuration library management module searches for the currently active template in the quantization constraint template set based on the scene identifier and hardware identifier, and then searches for a configuration record matching the template version number in the initial weight quantization configuration set.

[0079] If a matching initial weight quantization configuration can be found in the configuration library, matching the model version number, scene identifier, hardware identifier, and template version number, the system returns the initial configuration and the corresponding quantization constraint template to the training node for subsequent construction of the dual-path training topology and quantization path. If any step fails to find the configuration, such as the quantization constraint template not being generated under the combination of scene identifier and hardware identifier, or the corresponding template version number not finding a matching initial weight quantization configuration, or the model version number not matching the model structure registered in the initial configuration record, the configuration library management module preferably returns a preset error identifier and records the requester identifier, request time, input model version number, scene identifier, and hardware identifier, as well as the specific reason for the failure to establish the mapping in the log.

[0080] Log entries are preferably written in append-only mode, disallowing silent modification, serving as evidence for subsequent troubleshooting of missing configurations and version inconsistencies. Upon receiving an error flag, training nodes will not resume training for that round, thus preventing training on unlocked or inconsistent configurations, avoiding unreproducible quantization behaviors and strategies, and protecting the validity of subsequent quantization trajectory vectors and quantization difficulty maps. For repeated configuration retrieval requests with the same model version number, scene identifier, and hardware identifier combination, the system can set an idempotent strategy, using the model version number, scene identifier, and hardware identifier as idempotent keys. If the initial configuration has already been generated and the version matches, the existing configuration record is directly returned without re-triggering the calibration and configuration generation processes, ensuring configuration consistency while reducing additional resource consumption.

[0081] In a preferred embodiment, the size of the calibration sample set can be set to several thousand to several tens of thousands of requests, the energy percentage threshold can be set to a certain percentage of the total energy of each layer, the number of bits can be set to a discrete set from low bits to medium bits, and the grouping granularity can be set to each group cover several tens to several hundreds of parameters. Under this configuration, the initial configuration of weight quantization for a medium-sized language model can significantly reduce the storage and computational overhead in the subsequent quantization path while maintaining representative sample coverage, and provide a stable foundation for subsequent strategy adjustments based on quantization trajectory vectors.

[0082] S3. Construct a dual-path training graph under the quantization constraint template and initial weight quantization configuration. Execute both the floating-point path and the quantization path simultaneously for each batch. Construct a composite loss on the quantization path that includes task loss, layer representation difference, and resource regularization term. The specific implementation is as follows:

[0083] After completing the construction of the significant channel set and the initial configuration of weight quantization, the system establishes a dual-path training topology on the training nodes based on the quantization constraint template and the initial configuration of weight quantization under the same scenario. The dual-path training topology is used to execute two inference paths simultaneously on the same small batch of samples under unified scheduling. One is a floating-point path and the other is a quantization path. This allows for continuous comparison of the behavioral differences between the two precision forms within a controlled resource budget and provides a basis for subsequent quantization strategy adjustments.

[0084] The floating-point path refers to the path that uses full-precision weights and full-precision activations for inference at each layer of the model. Its capability lies in providing a performance benchmark close to the theoretical upper limit, but its range is limited by the target model structure and available hardware resources. The quantization path refers to the path that quantizes the weights at each layer of the model according to the number of bits, scaling factor, and grouping method registered in the initial configuration of the weight quantization before inference. Its capability lies in approximating the performance of the floating-point path while reducing storage and computational requirements. Its range is jointly limited by the range of the number of bits and the range of grouping granularity given by the quantization constraint template.

[0085] When constructing a dual-path training topology, it is preferable to assign the same batch size and sequence length to the two paths. The batch size can be set to the number of samples that can be accommodated within the memory budget of a single training node, and the sequence length can be set to a representative length range obtained by combining the context distribution statistics under the target scene, thereby ensuring that the two paths generate comparable hierarchical representations on the same batch of data.

[0086] For each mini-batch of samples, the training node first calculates the representations and final predictions of each layer along the floating-point path within the dual-path training topology, and then calculates the corresponding layer representations and prediction results along the quantization path under quantization weights. In the quantization path, the system calculates three types of quantities for this mini-batch of samples: one is the task-related loss, used to measure the deviation of the quantization path prediction results from the labeled answer or target output; its ability lies in directly reflecting the performance of the quantization path on the target task, and its range is determined by the selected loss form and task type; another is the representational difference quantity, used to measure the internal representation of the quantization path at several key layers relative to the corresponding floating-point path. The degree of difference in layer representation can be categorized into key layers, which can be set as higher layers close to the output, layers containing more significant channels, or layers with high sensitivity to the task. The ability of the difference quantity to represent the difference quantity lies in reflecting the impact of quantization on the distribution of intermediate representations, and its range is limited by the selected set of layers and the difference measurement method. Another type is the resource consumption penalty quantity, which is used to constrain the computational overhead and memory consumption generated under the current weight quantization configuration based on the range of bit numbers of each layer recorded in the quantization constraint template and the hardware cost estimation results. Its ability lies in explicitly introducing the resource budget pressure into the training objective, and its range is defined by the upper limit of the resource budget registered in the constraint template.

[0087] The system preferably registers the composite loss weight configuration in an independent configuration table in the configuration library. This configuration table includes at least the task-related loss weight, the representation difference weight, and the resource consumption penalty weight, as well as the corresponding version number. During training, the weight configuration is referenced according to the version number. The composite loss is formed by weighting and synthesizing the three types of quantities. The composite loss serves as the overall goal for updating the weights and quantization-related parameters during backpropagation, so as to achieve a balance between task performance, representation consistency, and resource budget.

[0088] The training scheduling unit, as the management component responsible for organizing training tasks, can be configured as a centralized scheduling service or a cluster control module. When preparing to start a training phase, it distributes the training configuration to the training nodes via remote calls. The training configuration includes at least the model version number, the initial configuration version number for weight quantization, the version number for quantization constraint template, the batch size, the maximum number of training steps, and resource budget indicators. The model version number is used to identify the model structure and parameter baseline involved in this training. The initial configuration version number for weight quantization and the version number for quantization constraint template are used to ensure that this training is strictly carried out according to a certain locked configuration. The batch size and the maximum number of training steps are used to define the scale and duration of this training phase. The resource budget indicators are used to constrain the allowable usage of the training phase on a single node or multiple nodes, such as the upper limit of the average latency per iteration, the upper limit of the average power consumption per unit time, or the upper limit of the proportion of GPU memory usage.

[0089] Upon receiving the training configuration, the training node preferably first performs a consistency check on the reference relationship between the model version number, the initial configuration version number of weight quantization, and the version number of quantization constraint template, either locally or through the configuration library management module. Simultaneously, it checks whether the resource budget given in the training configuration is not lower than the minimum resource requirements specified in the quantization constraint template for that scenario and hardware combination. If it finds that there is no valid mapping between versions, a missing version record, or the resource budget is lower than the constraint template requirements, the training node generates a corresponding error flag and feeds it back to the training scheduling unit. It also records the timestamp, node identifier, received version number combination, and judgment result in its local log, terminating the current round of training preparation. This is to avoid starting a dual-path training topology under inconsistent configuration or insufficient resource budget conditions, thereby preventing unreproducible quantization behavior and uncontrolled resource consumption.

[0090] To reduce resource waste caused by redundant configuration, the system can generate idempotent keys based on the training configuration content and the version number involved. These idempotent keys are recorded in the training session registration table. When the training scheduling unit issues the same training configuration multiple times within a certain period, the training node identifies it as the same training session through the idempotent key. Preferably, the established dual-path training topology and related caches are reused without reallocating resources or repeatedly building the computation graph. For completed or terminated training sessions, the training node can distinguish between them by combining the training status field when maintaining the idempotent key. For example, idempotent keys for terminated sessions are marked as non-reusable, while reconnection and attachment are allowed for ongoing sessions, ensuring sequential consistency and controllable resource utilization.

[0091] In a preferred embodiment, the batch size can be set to several tens to several hundred samples, the maximum training steps can be set to several thousand to several tens of thousands of steps, and the sum of the weights of task-related loss, representation difference, and resource consumption penalty can be set to a fixed constant. In this configuration, the weight of task-related loss is given a larger weight, followed by representation difference, and the weight of resource consumption penalty is relatively smaller. This prioritizes task performance in the early stages of training, and gradually uses the resource consumption penalty to drive the configuration towards resource efficiency as convergence approaches. Under this configuration, the dual-path training topology can fully collect the deviation information of the quantization path relative to the floating-point path while maintaining controllable resource consumption. This lays the foundation for the subsequent construction of quantization trajectory vectors and quantization difficulty maps, and ensures the traceability and repeatability of the entire training chain through versioned training configuration and idempotent control.

[0092] S4. During training, record the weight difference, activation difference, and gradient norm before and after quantization for each weight group at intervals. Update the quantized trajectory vector of the weight group based on the time series. Specifically, the implementation is as follows:

[0093] After the dual-path training topology is running stably, the system preferably registers quantization behavior records for each weight reassembly at fixed step intervals on the training nodes for each training iteration, in order to characterize the historical behavior trajectory of the weight reassembly in the quantized state. A weight reassembly refers to a group of weight parameter units aggregated in the same layer according to a predetermined grouping method. Its capability lies in serving as the basic unit for quantization configuration, undertaking the configuration of the number of bits and scaling factor. Its range is jointly limited by the aforementioned grouping method and grouping granularity range.

[0094] The quantization behavior record includes at least the current training step number, the statistic of the weight difference before and after quantization in this step, the activation difference statistic of the quantized path and the floating-point path in the layer where the weight reassembly is located, and the gradient norm statistic associated with the weight reassembly. The training step number is used to characterize the time position of the record in the whole training process, the weight difference before and after quantization is used to characterize the degree of numerical shift caused by quantizing the weights before and after quantization according to the initial weight quantization configuration and the current quantization strategy, the activation difference is used to characterize the degree of deviation between the quantized path and the floating-point path in the output representation of the layer on the same mini-batch of samples, and the gradient norm is used to characterize the sensitivity of the weight reassembly to the composite loss in the current training step.

[0095] The aforementioned statistics are preferably represented by a combination of the mean, several percentile values, and extreme values. For example, the mean, percentile, and maximum value of the weight difference, activation difference, and gradient norm can be calculated for the mini-batch samples, respectively, to take into account both the overall level and local peaks, thereby more fully reflecting the behavioral characteristics of the weighted group in the current step. The system can configure a recording interval parameter for each weighted group. The recording interval can be set to several training steps, and a new quantized behavior record is generated only when the recording interval is reached, so as to avoid excessive storage and computational overhead caused by overly dense registration of each step. The recording interval's capability lies in controlling the fineness of temporal sampling, and its range can be set according to the model size, training duration, and resource budget.

[0096] To mitigate the interference of short-term fluctuations on subsequent evaluations, the system applies smoothing and truncation strategies to the aforementioned statistics when generating quantitative behavior records. The smoothing strategy can be configured to perform a moving average of the same statistic over a range of recent records, thus diluting the impact of single-step outliers over time. The truncation strategy can be configured to assign an upper and lower limit value to each type of statistic; when the statistic calculated for a particular record exceeds this range, it is truncated to the most recent boundary value to prevent a few abnormally large values ​​from artificially inflating subsequent evaluation indicators.

[0097] To ensure continuity over time, when statistical values ​​are missing within several consecutive recording intervals, the system preferably uses the statistical value from the most recent valid record to replace the current value, thus avoiding gaps in the quantization trajectory on the time axis. Based on the aforementioned observation window, the system configures a trajectory observation window for each weighted group. This trajectory observation window can be set to cover a range of training steps corresponding to several recording intervals, used to aggregate quantization behavior records within this time period.

[0098] Within a trajectory observation window, the system continuously records quantization behavior multiple times for the same weighted reassembly, and combines the statistical sequence within the window into a quantized trajectory vector according to time sequence. The quantized trajectory vector is used to describe the quantization offset, representation deviation, and sensitivity change trend of a certain weighted reassembly over a training period. Its ability lies in providing time series evidence for subsequent difficulty assessment and strategy adjustment. Its scope is jointly determined by the length of the trajectory observation window, the recording interval, and the selection method of statistics.

[0099] The length of the trajectory observation window can be set to cover several hundred or several thousand training steps to achieve a balance between temporal resolution and storage overhead. For scenarios with shorter training processes, the window length can be shortened, while for scenarios with longer training processes and the goal of capturing medium- to long-term trends, the window length can be extended accordingly.

[0100] After the quantized trajectory vectors are generated, the system stores them in a dedicated trajectory library. This library centrally manages trajectory records for each weighted reorganization under different observation windows. Its capability lies in providing a queryable and version-manageable view of quantized behavior, covering all layers, all weighted reorganizations, and multiple consecutive observation windows within the same model version. The trajectory library sets a primary key field for each trajectory record. The primary key includes at least a unique identifier for the weighted reorganization, the starting training steps of the trajectory observation window, the ending training steps of the trajectory observation window, and a value summary. The unique identifier for the weighted reorganization can be composed of the model version number, layer index, and weighted reorganization index. The value summary is used to condense the statistical information in the trajectory vector, for example, by encoding it through a sequence of sampling points or by further aggregating indicators.

[0101] In the trajectory database, each trajectory record is also assigned a trajectory version number to distinguish trajectories generated under different training stages, configurations, or smoothing strategies. When smoothing or truncation rules are adjusted, the version number can be used to differentiate between the old and new rules. To avoid multiple semantically equivalent records within the same observation window due to duplicate registration, the system sets an idempotent key on the combination of the unique identifier of the weighted reassembly and the start and end training steps of the trajectory observation window. When a newly written record has the same idempotent key as an existing record, the system can be configured to overwrite or ignore the write. The specific strategy can be defined in the trajectory database configuration, thereby avoiding logical duplication within the same time range.

[0102] Considering that requests to write to the trajectory database may be out of order in distributed training or multi-node parallel scenarios, the system, in addition to checking idempotent keys, can also use the coverage of the trajectory observation window as a criterion to adjudicate trajectory records with the same weighting that fall within partially overlapping windows. For example, when there is a conflict between trajectory records with larger and smaller coverage, the system preferably retains the record with larger coverage and marks the record with smaller coverage as expired. Expired records are not returned by default during queries, but can still be used for backtracking analysis when needed.

[0103] To protect user privacy and meet compliance requirements, the trajectory database only records the aggregated statistics at the batch and window dimensions. It does not save any single sample-level activation values, gradient values, or original content. The weighted recombinant unique identifier corresponds to the task type and scene identifier, but does not involve specific session identifiers or terminal identifiers, thereby avoiding inferring individual user behavior from trajectory records.

[0104] In deployment scenarios with extremely limited resources or high latency sensitivity, the system can adopt a mode of reducing the monitoring scope, constructing quantized trajectory vectors only for a few key layers and key weight groups, while keeping the remaining layers and weight groups with fixed quantization configurations and not participating in trajectory registration. Key layers can be set as layers that have a significant impact on task performance, and key weight groups can be set as weight groups containing a large number of significant channels or weight groups that account for a significant proportion of hardware costs. The sets of key layers and key weight groups and their monitoring strategies are stored in the configuration library as independent configuration records, and are assigned monitoring strategy version numbers to distinguish between full-coverage monitoring schemes and reduced monitoring schemes during subsequent evaluations.

[0105] The reduced monitoring scheme is registered alongside the full coverage scheme in the evidence chain. Multiple trajectory scheme configurations can exist for the same model version and scene identifier. When generating the quantization difficulty map, the system selects the corresponding trajectory data source according to the current monitoring strategy version. This ensures that the basic technical concept of the method remains unchanged under the condition of limited resource budget. That is, by periodically recording the quantization behavior of the weight group during training, a quantization trajectory vector with temporal continuity is formed, which is used as the basis for subsequent quantization strategy adjustments.

[0106] In a preferred embodiment, the recording interval can be set to several tens of training steps, the trajectory observation window length can be set to several hundreds to several thousand training steps, the statistics adopt a combination of mean, percentile and maximum value, the smoothing strategy adopts fixed window moving average, and the truncation strategy adopts upper and lower limits set for different layers. Under this configuration, a stable quantized trajectory vector can be constructed for each key weight recombination within a controllable storage and computing overhead, providing a data foundation for the subsequent construction of quantization difficulty map, and forming a complete version management and evidence chain through trajectory version number, idempotent key and expiration mark.

[0107] S5. Calculate the difficulty score of the weight group based on the quantized trajectory vector to generate a quantized difficulty map. Adjust the number of bits, scaling factor, and grouping method of the weight group under the quantization constraint template, and update the salient channel set. Specifically, the implementation is as follows:

[0108] After constructing quantized trajectory vectors for each weighted recombination in the aforementioned manner during the training phase, the system, at the end of the preset policy update cycle, summarizes and evaluates the behavior of each weighted recombination under quantized conditions within the current phase based on the latest version of the quantized trajectory vectors registered in the trajectory library. The policy update cycle is used to limit the range of training steps covered by a single quantization policy adjustment. Its capability lies in controlling the frequency and granularity of policy adjustments, and its range can be set to cover several trajectory observation windows.

[0109] At each policy update, the system extracts the quantized trajectory vector of the corresponding weight group within the most recent trajectory observation window from the trajectory database according to the model version number, layer index, and weight group index. The recorded weight difference statistics, activation difference statistics, and gradient norm statistics are used as the evaluation basis. The magnitude and stability of these statistics within the observation window are comprehensively characterized to obtain the quantization difficulty score for that weight group. The quantization difficulty score describes the ease with which the weight group maintains task performance and representation consistency under given quantization configuration and training conditions. Its ability lies in providing a ranking basis for the differentiated allocation of subsequent quantization resources. Its range is determined by a combination window of the three types of statistics: weight difference, activation difference, and gradient norm.

[0110] The system can assign higher weights to certain statistical measures. For example, in scenarios sensitive to task performance, it can increase the contribution of activation difference to the difficulty score; in scenarios with high requirements for convergence stability, it can increase the weight of gradient norm stability. These weight parameters are registered in a versioned configuration library to ensure traceability between different evaluation rules. In addition to the difficulty score, the system can also calculate a resource sensitivity score for each weighted set based on the aforementioned hardware capability description and quantization constraint template. The resource sensitivity score describes the impact of bit number adjustments or changes in grouping granularity on the overall computational and storage costs of the weighted set. Its capability lies in helping to prioritize the allocation of limited budgets to weighted sets that are difficult to quantify but have a significant impact on resource consumption under resource constraints. Its range is jointly determined by the hardware cost model and the current quantization configuration.

[0111] After obtaining the quantitative difficulty score and resource sensitivity score of ownership restructuring in one round of evaluation, the system organizes these scores into a quantitative difficulty graph according to the layer index and the weight restructuring index. The quantitative difficulty graph can be understood as a two-dimensional arrangement with the layer index as one dimension and the weight restructuring index as the other dimension. Each position in this arrangement stores the quantitative difficulty score and resource sensitivity score of the corresponding weight restructuring in the current policy update cycle, as well as related metadata.

[0112] The capability of the quantization difficulty map lies in presenting a unified view of the vulnerability and resource contribution of each part of the model under quantization conditions, providing a structured basis for subsequent centralized decision-making. Its scope covers all layers and weight groups involved in monitoring under the current model version. When generating the quantization difficulty map, the system assigns it an independent map version number and records the set of trajectory version numbers and evaluation rule version numbers referenced in the map construction. This allows for future tracing back to the original evaluation criteria and trajectory sources based on the map version number. After completing the quantization difficulty map construction, the system adjusts the quantization strategy of each weight group according to the quantization difficulty map, within the bit count and grouping granularity range specified by the quantization constraint template.

[0113] Specifically, for weighted reassemblies with both high quantization difficulty scores and high resource sensitivity scores, the system preferably allocates a higher number of bits to them within the candidate bit set, or reduces the grouping granularity within the allowed range, so that these weighted reassemblies, which are difficult to quantize and have a significant impact on task performance, can obtain a more refined numerical representation. For weighted reassemblies with consistently low quantization difficulty scores over several consecutive policy update cycles, the system can appropriately reduce the number of bits or relax the grouping granularity without exceeding the lower limit specified by the quantization constraint template, in order to free up computing power and storage budget, which can then be used to support other weighted reassemblies with higher difficulty. The above adjustment process can adopt a hierarchical strategy, for example, first sorting the layers according to difficulty scores, and then selecting the subset to be adjusted first based on resource sensitivity, so as to reduce the range of policy changes in a single update and keep the training process smoothly converging.

[0114] While making local adjustments to the strategies of each weight group, the system also reconstructs the salient channel set based on the latest round of quantization difficulty scores. The salient channel set was originally determined by energy percentage or activation amplitude thresholds. By introducing quantization difficulty scores, channels that consistently exhibit high difficulty in quantization can be further identified at the channel level. For example, if a weight group containing a channel maintains a high difficulty score across multiple strategy update cycles, and this channel has been identified as an important channel under the aforementioned salient channel rules, the system can include this channel in the salient channel set with higher priority and mark it as a key protection channel. Conversely, for channels that were once considered significant in terms of energy percentage but have consistently maintained a low level in quantization difficulty scores, the system can mark them as channels with reduced priority and, if necessary, temporarily remove them from the salient channel set, allowing limited high-precision resources to be concentrated on channels that are truly difficult to quantize. The reconstructed salient channel set is written back to the configuration repository with a new set version number, and records the channel index list involved in this round of changes and the differences in salient marking before and after, to improve the evidence chain regarding channel-level strategy evolution.

[0115] After completing a round of strategy adjustments based on the quantization difficulty graph, the system summarizes the updated bit count, scaling factor, grouping method, and significant channel markers to form a new weight quantization configuration version. The new weight quantization configuration version is registered in the configuration database in the form of a configuration table. Each record, in addition to the original layer index, weight regrouping index, bit count, scaling factor, grouping method, and significant channel marker, also includes the quantization difficulty graph version number referenced in this round of update, the time window identifier for the corresponding training phase, and the configuration number of the previous version.

[0116] The system can simultaneously generate a configuration difference summary, which records which weight groups saw an increase or decrease in bit count, which weight groups experienced changes in grouping granularity, and which channels had their salient labels altered during the current update. The configuration difference summary is archived along with the configuration version, forming part of the policy evolution evidence chain. This chain allows operations personnel and review bodies to review at any point in time how the model quantization policy evolved during training, from the initial configuration version to the latest frozen version.

[0117] After adjusting the quantization strategy, the system assesses the resource requirements of the new weighted quantization configuration based on the resource budget ceiling and hardware cost model recorded in the quantization constraint template. This resource requirement assessment estimates the average latency, peak latency, average power consumption, and memory usage during inference under this configuration. If the estimation results indicate that any resource requirement exceeds the corresponding budget ceiling, the system classifies the current strategy update as a resource overrun event. Instead of taking effect immediately, it reverts to the previous version of the weighted quantization configuration. Simultaneously, the system logs the occurrence time of this resource overrun event, the involved model version number, the quantization difficulty map version number referenced during the strategy update, the name of the overrun metric, and the overrun magnitude, and assigns an error flag to the event.

[0118] To facilitate more targeted resource consumption balancing in subsequent policy updates, the system can identify layers with significant resource consumption based on resource demand assessment results. These layers are marked as "stressed layers," indicating that they are approaching or exceeding reasonable costs under the current resource budget. The system's capability is to guide subsequent policy updates to prioritize releasing budget in other layers, and its scope is limited to the set of layers that contributed significantly in the resource assessment. In the next round of policy updates, the system can restrict further increases in bit count or reductions in grouping granularity for stressed layers based on the stressed layer markings, and encourage appropriate downsizing of non-stressed layers to restore overall resource balance.

[0119] Considering that multiple update processes may attempt to modify the weight quantization configuration in parallel in a large-scale distributed training environment, the system employs a version number locking mechanism for the configuration version during the policy update cycle to prevent mutual overwriting or version overlap. When starting a new policy update, the system first reads the currently effective weight quantization configuration version number from the configuration library based on the model version number, scene identifier, and hardware identifier, and reserves a successor version number for this round of updates. Then, a locking flag is set for this version number combination.

[0120] During the locking period, when other concurrent update processes detect that the same model version number, scene, and hardware combination are already locked, they will generate an idempotent key based on the configuration content, quantization difficulty map version number, and update intent, and compare it with the currently ongoing update request. When the idempotent key matches, the system will include the subsequent request in the same update session instead of generating a new version separately. When the idempotent key does not match, the system can queue the subsequent request to the next policy update cycle or directly return a prompt indicating the current locking state. By combining version number locking with idempotent key recognition, it can be ensured that only one valid new configuration version is generated in each policy update cycle, thereby avoiding multiple conflicting quantization policy versions within the same training phase.

[0121] In a preferred embodiment, the policy update cycle can be set to several thousand training steps, with a hard limit of generating at most one new configuration version within each cycle; the quantization difficulty score can be a weighted combination of the mean of the weight difference, the percentile of the activation difference, and the variance of the gradient norm within the trajectory observation window; the resource sensitivity score can be the degree of impact of the unit bit change in the hardware cost model on latency and power consumption. Under this setting, the system can construct a clear policy evolution trajectory after dozens of rounds of training and policy updates, enabling the model to gradually converge the limited accuracy budget to the most difficult weight and channel to quantize within the resource budget constraint, laying the foundation for subsequent quantization configuration freezing and deployment.

[0122] S6. After training iterations and quantization difficulty graph updates, when task performance and resource metrics meet preset conditions, freeze the quantization configuration, quantize the model weights according to the quantization configuration, and export and deploy the model. Specifically, the implementation is as follows:

[0123] Based on the aforementioned alternating training iterations and quantization strategy updates, the system preferably sets up an independent on-site verification mechanism to evaluate the overall performance of the model under the current weight quantization configuration under near-real load conditions, thereby determining whether the configuration has long-term usability. To this end, the system constructs a representative task set in the target business scenario. This representative task set simulates the main online request patterns, aiming to reproduce the real business distribution as closely as possible with a limited sample size. Its scope is preferably defined by hierarchical extraction based on task type and context length: task type distinguishes different uses such as dialogue generation, code generation, knowledge question answering, and summary generation; context length divides short text, medium-length text, and long context intervals. Within each layer, the system randomly extracts a certain number of requests according to a preset ratio, merging them into a set containing several thousand requests. The content and version of the representative task set can be updated periodically according to business changes.

[0124] In each round of evaluation, the system runs all requests from the representative task set with the current quantified configuration. This process can be executed in an independent evaluation cluster or during low-traffic periods without affecting online service stability. Simultaneously, the monitoring module records task and resource metrics during the evaluation. Task metrics reflect the model's performance at the functional level, quantitatively measuring the degree to which the current quantified configuration achieves the expected business objectives. These metrics can include single or weighted metrics such as accuracy, recall, generation quality score, and manual quality inspection score. The system can select different combinations of task metrics based on different task types; for example, in classification tasks, it emphasizes accuracy and recall, while in generation tasks, it emphasizes automatic quality scoring and manual sampling results.

[0125] Resource metrics are used to characterize the model's consumption of computing and storage resources under the current quantization configuration. Their ability is to reflect whether the configuration meets the resource budget set by the quantization constraint template. Their range can include average latency, several percentile latency, average power, and memory usage ratio. Average latency and percentile latency are calculated by the monitoring module based on the request return time. Average power is sampled by the power monitoring device at a fixed rhythm and averaged within the evaluation window. Memory usage ratio is given by the ratio of the peak value of the deployed node during the inference process to the available memory in the hardware.

[0126] The above evaluation is repeated within a preset check interval. The check interval controls the evaluation frequency and is designed to capture phased changes in model performance without excessively interfering with training and deployment. Its range can be set to be divided into several training steps, several minutes, or several hours. The system compares the task and resource metrics of each round of evaluation with pre-set task performance lower limits and resource budget upper limits. The task performance lower limit defines the minimum acceptable business performance, and its range can be obtained by subtracting the allowable performance loss from the metrics of the full-precision model or a historically validated version model on the same representative task set. The resource budget upper limit directly references the latency, power, and memory limits given in the quantization constraint template for this scenario and hardware combination.

[0127] To avoid misjudgments caused by fluctuations in a single assessment, the system preferably requires that, in a series of consecutive checks, all task indicators remain stable above their corresponding lower limits, and all resource indicators do not exceed the upper limits specified by the quantification constraint template. Simultaneously, the variation range of each indicator (e.g., the relative difference between two adjacent indicators) must remain within a preset fluctuation range over several check cycles, which can be set to not exceed a certain percentage. Only when all of the above conditions are met simultaneously will the system determine that the current weight quantification configuration and its associated set of significant channels have reached the freezing condition.

[0128] Once the freezing condition is met, the system marks the current weight quantization configuration as a frozen quantization configuration. A frozen quantization configuration represents a quantization strategy that has been recognized as stable and usable after complete training, evaluation, and resource verification under the current scenario and hardware combination. Its capability lies in serving as a baseline configuration for subsequent deployment and rollback, and its scope is limited by the version number and applicable scenario identifier. The system updates the configuration's status to frozen in the configuration library and prevents subsequent policy update processes from modifying this configuration. New policy updates can only be performed on a copy of the frozen configuration or on configurations generated during subsequent training phases. Subsequently, based on the number of bits, scaling factor, and grouping method recorded in the frozen quantization configuration, the system performs a one-time quantization operation on the full-precision weights of the corresponding model version, completing the quantization mapping of all weights according to the parameters specified for each weight regroup in the frozen quantization configuration, generating a deployment model that can be directly run on the target hardware.

[0129] A deployed model refers to a model instance obtained under frozen quantization configuration constraints and intended for online inference services. Its capability lies in providing near-full-precision model performance while meeting resource budget constraints. Its scope is jointly defined by the model structure, frozen quantization configuration version, and applicable scenario identifier. The system registers the deployed model in the model repository, a centralized storage system for managing model versions and their associated metadata. Each model record includes at least the model structure identifier, the location of the deployed model file, the frozen quantization configuration version number, the version number of the referenced quantization constraint template, the version number of the training corpus used, and the effective time period.

[0130] The model repository uses a version locking mechanism to ensure that the same frozen quantization configuration corresponds to only one deployment model version under the same scenario and hardware combination. When a new deployment request carries the same frozen quantization configuration version number and scenario and hardware identifier, the system identifies it by generating an idempotent key. The idempotent key can be obtained by combining the frozen quantization configuration version number, scenario identifier, and hardware identifier. If a deployment model corresponding to the idempotent key already exists in the repository, the existing model file information is returned directly without regenerating it, thereby ensuring idempotency and reducing the overhead of duplicate quantization.

[0131] To meet security and compliance requirements, the metadata in the deployment model records only records statistical information at the scenario and task levels, such as task type coverage and representative task set versions, and does not record single user identifiers, original session identifiers, or information that can be traced back to natural persons; when necessary, version information related to the training corpus can be anonymized and an irreversible summary can be preserved.

[0132] When a deployment node receives a deployment instruction and attempts to load the frozen quantization configuration and deployment model files, if it encounters situations such as the specified version not being found in the model repository, the storage system being temporarily inaccessible, or the signature verification failing, the system preferably automatically selects the deployment model version that has recently passed acceptance and is still within its support period for replacement according to a predefined rollback strategy. The rollback action will record the rollback reason, the original target version number, the actual loaded version number, and the occurrence time in the log item by item, which will be used as the evidence chain for subsequent auditing and problem tracking. At the same time, the rollback event will be notified to the operation and maintenance personnel through the alarm channel for manual confirmation.

[0133] To avoid deployment sequence errors leading to online version rollback, the system sorts deployment requests by target deployment version number. When a deployment request with a target version number lower than the currently deployed version number is received, it can be directly rejected and an error flag can be returned. The error flag can include a message indicating that rollback to a lower version is not allowed without manual approval. For special cases where rollback is necessary, the corresponding permissions can be released after approval through a separate approval process.

[0134] Combining the aforementioned training and policy update settings, in a representative set of runs, the trajectory observation window length can be set to cover several hundred training steps, the policy update cycle can be set to cover several thousand training steps, and the weight bit range can be set to increase from low bits to medium bits. Some key layers that are in the high difficulty region for a long time in the quantization difficulty map are assigned higher bit counts and more granular groupings after two rounds of policy updates. The task indicators on the representative task set are improved by several percentage points compared to the initial quantization configuration, while the average latency only increases by a small percentage, and the resource indicators still meet the budget limit given by the quantization constraint template, thereby achieving a comprehensive balance between performance and cost under limited resource conditions.

[0135] For scenarios with stricter resource constraints, the system can also adopt an alternative solution that only performs trajectory monitoring and policy updates on the critical layers and maintains fixed quantitative configurations for non-critical layers. This alternative solution is registered as an independent policy version in the configuration library. Its ability is to reduce the overhead of monitoring and updating. Its scope is limited to a pre-selected set of critical layers. This solution is consistent with the full coverage solution in terms of methodological concept and is an equivalent form of this method. It will be included in the protection scope consideration in the following description.

[0136] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An AWQ model optimization method integrating weighted quantization-aware training, characterized in that, include: S1. Collect sample and task information based on the target business scenario, and generate quantitative constraint templates for constraint latency, power consumption and storage based on the computing cost and storage cost of the deployed hardware; S2. Using calibration samples, statistically analyze the activation distribution of each layer on the model, identify significant channels, determine the number of bits, scaling factor and grouping method of each weight group, and obtain the initial configuration of weight quantization. S3. Construct a dual-path training graph under the quantization constraint template and initial configuration of weight quantization. Simultaneously execute the floating-point path and quantization path for each batch, and construct a composite loss including task loss, layer representation difference and resource regularization term on the quantization path. S4. During the training process, record the weight difference, activation difference and gradient norm before and after quantization for each weight group at intervals, and update the quantized trajectory vector of the weight group based on the time series. S5. Calculate the difficulty score of the weight group based on the quantized trajectory vector to generate a quantized difficulty map. Adjust the number of bits, scaling factor and grouping method of the weight group under the quantization constraint template, and update the set of significant channels. S6. After training iteration and quantization difficulty graph update, when the task performance and resource indicators meet the preset conditions, freeze the quantization configuration, quantize the model weights according to the quantization configuration, and export the deployed model.

2. The AWQ model optimization method based on weighted quantization perceptual training according to claim 1, characterized in that, S1 includes: Assign scenario identifiers to target business scenarios, collect call logs and task records in the observation window, align task fields, latency, and power in the task records to a unified time axis, and statistically analyze latency and power according to the observation window and remove abnormal records. Based on the computational cost and video memory usage registered in the deployment hardware capability library, the latency limit, power limit, and video memory limit are determined according to service level constraints; Register quantized constraint templates containing bit ranges for each layer, grouping granularity ranges, and resource budget limits in the configuration library, using scene identifiers and hardware identifiers as keys. The template version number and effective time limit the existence of one effective template for the same scenario and hardware combination.

3. The AWQ model optimization method based on weighted quantization perceptual training according to claim 1, characterized in that, S2 include: In the target business scenario, extract anonymized historical calls in a hierarchical manner according to task type and context length to form a calibration sample set; Record the activation values ​​organized by layer index and channel index on the model to be deployed, generate a significant channel set according to the energy proportion threshold and write it into the configuration library; Based on the range of bit counts and grouping granularity for each layer given by the quantization constraint template, the weights of each layer are divided into weighted groups; Specify the number of bits, scaling factor, and grouping method for each weighted recombination, and register the association with the quantization constraint template in the configuration library; The training node obtains the corresponding initial configuration for weight quantization through the model version number, scene identifier, and hardware identifier.

4. The AWQ model optimization method based on weighted quantization-aware training according to claim 1, characterized in that, S3 includes: When constructing a dual-path training topology, the initial configuration of the training nodes is based on the quantization constraint template and weight quantization. For parallel execution of mini-batch samples, a floating-point path with full-precision weights and activation and a quantization path with quantized weights are used. In the quantization path, the task loss, the key layer representation difference, and the resource consumption penalty based on computational overhead and memory usage are calculated. The three are then combined into a composite loss according to the loss weight configuration registered in the configuration library. The composite loss drives the update of model weights and quantization-related parameters.

5. The AWQ model optimization method based on weighted quantization perceptual training according to claim 1, characterized in that, S4 includes: During the dual-path training topology, a quantization behavior record is generated for each weight reorganization at fixed step intervals. The quantization behavior record includes the current training step, the weight difference statistics before and after quantization, the activation difference statistics, and the gradient norm statistics. After smoothing and truncation, the quantized trajectory vectors are combined in chronological order within the trajectory observation window to form the weighted group. The quantized trajectory vectors are then stored in the trajectory library using the unique identifier of the weighted group, the starting training step number of the trajectory observation window, the ending training step number of the trajectory observation window, and the value summary as indexes.

6. The AWQ model optimization method based on weighted quantization perceptual training according to claim 1, characterized in that, S5 include: When updating the quantization strategy, at the end of the strategy update cycle, the quantized trajectory vector in the corresponding trajectory observation window is read from the trajectory library for each weighting. The quantization difficulty score is calculated based on the weight difference statistics before and after quantization, the activation difference statistics, and the gradient norm statistics. Calculate resource sensitivity scores based on hardware capability descriptions and quantitative constraint templates; A quantitative difficulty graph is generated by using layer-based indexing and weighted reorganization indexing, and the quantitative difficulty score, resource sensitivity score, and graph version number are registered in the configuration database.

7. The AWQ model optimization method based on weighted quantization-aware training according to claim 6, characterized in that: When adjusting the weight quantization configuration based on the quantization difficulty map, if the quantization difficulty score and resource sensitivity score of a certain weight group are both greater than the first threshold, the number of bits in that weight group is increased and the grouping granularity is reduced. When all scores are less than the second threshold, the number of bits in the weighting group is reduced and the grouping granularity is increased. Based on this, the significant channel set is updated and a new version of the weight quantization configuration is generated. The number of bits, scaling factor, grouping method, and significant channel markers recorded in the new version are associated with the quantization difficulty map version number and the previous weight quantization configuration version number and stored together. After adjustment, resource requirements are assessed based on the quantitative constraint template. If the resource requirements exceed the upper limit of the resource budget, the system will revert to the previous weighted quantitative configuration version and record the resource over-limit error flag.

8. The AWQ model optimization method based on weighted quantization perceptual training according to claim 1, characterized in that, S6 include: Based on the target business scenario, construct a representative task set, and within a preset inspection interval, configure and run the representative task set with the current weight and record task indicators and resource indicators. Compare the task metrics with the preset task performance lower limit, and compare the resource metrics with the resource budget upper limit given by the quantitative constraint template; When the task indicators meet the preset task performance lower limit and the resource indicators meet the resource budget upper limit given by the quantitative constraint template within the continuous inspection period, the current weight quantitative configuration is marked as frozen quantitative configuration. Based on the frozen quantization configuration, the corresponding model weights are quantized to generate a deployment model, and the deployment model and its frozen quantization configuration version number are registered in the model repository.

Images