Method and apparatus for model optimization with dynamic quantization

Abstract

The application provides a dynamic quantization model optimization method and device, and belongs to the technical field of model optimization. The dynamic quantization model optimization method comprises the following steps: obtaining input data to be processed; performing forward propagation calculation on the input data by using a model to obtain an initial probability distribution result; calculating a confidence score of the model for the input data based on the initial probability distribution result; determining a target quantization precision corresponding to the confidence score according to a preset mapping relationship between confidence and quantization level; and performing quantization processing on network parameters and / or intermediate activation values of the model according to the target quantization precision to obtain a dynamic quantization model. The dynamic matching of model calculation precision and data complexity can be realized, and therefore more intelligent and efficient model reasoning can be realized.

Description

Technical Field

[0001] This invention relates to the field of model optimization technology, and in particular to a dynamic quantization model optimization method and apparatus. Background Technology

[0002] As the parameter size of large language models grows, their enormous computational and memory consumption severely limits their deployment on resource-constrained devices such as mobile devices and edge computing nodes. Model quantization is one of the mainstream techniques to address this problem, reducing model size and accelerating computation by decreasing the numerical precision of model weights and activation values.

[0003] However, existing quantization techniques have significant limitations. A fixed quantization precision is typically determined after model training or before inference. This approach lacks flexibility, leading to unnecessary precision waste for simple samples and potentially causing a significant drop in output quality for complex samples due to insufficient precision. Other existing dynamic quantization methods primarily adjust based on the complexity of the input data or hardware conditions, resulting in coarse-grained optimization and failing to achieve a fine-grained trade-off between accuracy and efficiency.

[0004] Therefore, there is an urgent need in this field for a method that can adaptively adjust the quantization accuracy based on the model's own feedback in order to achieve more intelligent and efficient model inference. Summary of the Invention

[0005] This invention provides a dynamic quantization model optimization method and apparatus to solve the shortcomings of existing technologies that cannot adaptively adjust accuracy, and to achieve the effect of dynamic matching between model calculation accuracy and data complexity.

[0006] This invention provides a dynamic quantization model optimization method, comprising: Obtain the input data to be processed; The model is used to perform forward propagation calculations on the input data to obtain the initial probability distribution results; Calculate the confidence score of the model for the input data based on the initial probability distribution results; Based on the preset mapping relationship between confidence level and quantization level, the target quantization accuracy corresponding to the confidence score is determined; Based on the target quantization accuracy, the network parameters and / or intermediate activation values ​​of the model are quantized to obtain a dynamic quantization model.

[0007] According to a dynamic quantization model optimization method provided by the present invention, the step of calculating the confidence score of the model for the input data based on the initial probability distribution results includes at least one of the following methods: Calculate the information entropy of the initial probability distribution result, and use the negative correlation value of the information entropy as the confidence score; Extract the maximum probability value from the initial probability distribution results, and use the maximum probability value as the confidence score; Extract the maximum probability value and the second-highest probability value from the initial probability distribution results, calculate the difference or ratio between them, and use the difference or ratio as the confidence score.

[0008] According to a dynamic quantization model optimization method provided by the present invention, the step of determining the target quantization accuracy corresponding to the confidence score based on a preset mapping relationship between confidence score and quantization level includes: The confidence score is compared with at least one preset threshold; If the confidence score is within the first numerical range, then the first quantization precision is selected as the target quantization precision; If the confidence score is in the second numerical range, then the second quantization precision is selected as the target quantization precision; The number of bits for the first quantization precision is different from the number of bits for the second quantization precision.

[0009] According to the model optimization method for dynamic quantization provided by the present invention, the confidence score is positively or negatively correlated with the number of bits of quantization precision; When an aggressive efficiency strategy is adopted, if the confidence score is higher than the preset confidence threshold, the quantization precision with a lower number of bits is selected as the target quantization precision. When a conservative quality strategy is adopted, if the confidence score is higher than the preset confidence threshold, the quantization precision with a higher number of bits is selected as the target quantization precision.

[0010] According to a dynamic quantization model optimization method provided by the present invention, the network parameters and / or intermediate activation values ​​of the model are quantized, including: From multiple pre-stored model weight copies of different precisions, directly call the weight copy corresponding to the target quantization precision; Alternatively, the full-precision floating-point weights of the model can be obtained, and the corresponding quantization weights and quantization parameters can be calculated in real time according to the target quantization precision.

[0011] According to a dynamic quantization model optimization method provided by the present invention, the method further includes a feedback optimization step: The dynamic quantization model is used to perform inference calculations on the input data to generate the final output result. Obtain the confidence score, the selected target quantization precision, and user feedback or quality score of the generated final output during this inference process; Based on the user feedback or quality score, the threshold in the mapping relationship between the confidence level and the quantization level is dynamically adjusted.

[0012] According to a dynamic quantization model optimization method provided by the present invention, the step of determining the target quantization accuracy corresponding to the confidence score based on a preset mapping relationship between confidence score and quantization level further includes: Obtain hardware resource status data of the current operating environment, wherein the hardware resource status data includes at least one of memory utilization and CPU / GPU load; The target quantization accuracy is determined based on the confidence score and the hardware resource status data.

[0013] According to the present invention, a dynamic quantization model optimization method is provided, wherein the model is a large language model based on the Transformer architecture, and the quantization processing of the network parameters and / or intermediate activation values ​​of the model is performed on a specific level or module of the model, and different levels or modules are assigned different target quantization accuracies according to their respective local confidence.

[0014] The present invention also provides a model optimization device for dynamic quantization, comprising: The acquisition module is used to acquire the input data to be processed. The first processing module is used to perform forward propagation calculations on the input data using the model to obtain the initial probability distribution result; The second processing module is used to calculate the confidence score of the model for the input data based on the initial probability distribution result; The third processing module is used to determine the target quantization accuracy corresponding to the confidence score based on the preset mapping relationship between confidence score and quantization level. The fourth processing module is used to quantize the network parameters and / or intermediate activation values ​​of the model according to the target quantization accuracy to obtain a dynamic quantization model.

[0015] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the dynamic quantization model optimization method as described above.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the model optimization method of dynamic quantization as described above.

[0017] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the model optimization method of dynamic quantization as described above.

[0018] The dynamic quantization model optimization method and apparatus provided by this invention calculates the confidence level as the decision variable for quantization accuracy, rather than relying on the uniform accuracy of the entire model or adjusting the accuracy solely based on hardware load in traditional quantization techniques. This enables dynamic matching between model calculation accuracy and data complexity, thereby achieving more intelligent and efficient model inference. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the dynamic quantization model optimization method provided by the present invention; Figure 2 is a schematic diagram of the dynamic quantization model optimization device provided by the present invention; Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] The following is combined with Figure 1 Figure 3 illustrates the dynamic quantization model optimization method and apparatus of the present invention.

[0023] This embodiment provides a dynamic quantization model optimization method, which can be applied to computing devices such as servers, edge computing devices, or mobile terminals. Figure 1 The overall flowchart of the method is shown.

[0024] like Figure 1 As shown, the method mainly includes steps S1, S2, S3, S4 and S5.

[0025] S1: Obtain the input data to be processed.

[0026] In large-scale language model application systems, the input data is typically a sequence of text input by the user. For example, if a user inputs "Please help me write a poem about spring," the system running the model first converts this text sequence into a sequence of token IDs that the model can process using a token segmenter.

[0027] S2: Use the model to perform forward propagation calculations on the input data to obtain the initial probability distribution results.

[0028] To assess the model's confidence in the current input, a preliminary prediction can be made.

[0029] Understandably, you can choose one of the following two methods.

[0030] In one embodiment, a complete forward propagation is performed using the complete model to obtain the Logits of the output layer, which are then normalized by the Softmax layer to obtain the probability distribution for the prediction of the next Token.

[0031] In another embodiment, to reduce runtime overhead, only the first few layers of the model, such as the first 6 layers of a Transformer, can be used as a probe or lightweight prediction head to quickly generate an approximate probability distribution result.

[0032] S3: Calculate the confidence score of the model for the input data based on the initial probability distribution results.

[0033] The confidence score reflects the model's certainty about its output. This embodiment provides several specific calculation logics.

[0034] In one example, the confidence score can be calculated based on the maximum probability value. This can be achieved by directly scanning the probability distribution vector and locating the element with the highest probability value. The maximum value directly reflects the model's confidence in the optimal candidate outcome, and this value can be normalized and used as the confidence score.

[0035] Specifically, you can obtain the value with the highest probability in the probability distribution. For example, if the probability of predicting the next word as "wind" is 0.92, then the confidence score is 0.92. This method is the fastest to calculate.

[0036] In another example, the calculation can be based on probability differences. The probability distribution vector can be sorted to extract the highest and second-highest probability values. Then, the difference or ratio between these two values ​​is calculated. The difference or ratio reflects the discriminatory power between competing candidates. If the difference is small, it indicates that the model hesitates between the two options, resulting in a lower confidence score; conversely, a larger difference generates a higher confidence score.

[0037] Specifically, we can obtain the highest probability value (Top-1) and the second highest probability value (Top-2). For example, Top-1 is 0.6 and Top-2 is 0.55, with a difference of 0.05. The smaller the difference, the more indecisive the model is between the two options, and the lower the confidence level.

[0038] In another example, information entropy can be used for calculation. The probability distribution vector can be traversed to assess the dispersion of the numerical distribution, thereby calculating the information entropy. Information entropy, in a physical sense, characterizes the uncertainty of a system. If the probability distribution exhibits a flat characteristic, meaning the probabilities of each category are similar, the calculated entropy value is high, and the system maps it to a lower confidence score. If the probability distribution exhibits a sharp characteristic, meaning the probability of a certain category is significantly prominent, the calculated entropy value is low, and the system maps it to a higher confidence score. Specifically, the information entropy of the probability distribution can be calculated. The lower the entropy value, the sharper the distribution, and the more deterministic the model.

[0039] In some embodiments, the reciprocal of the entropy value or the negatively correlated linear transformation value can be used as the confidence score, and there is no limitation here.

[0040] It should be noted that diverse confidence metrics can be employed in the above embodiments, enabling the system to capture uncertainties at different levels of the model. Calculations based on probability differences can also effectively address the limitation of a single maximum probability value failing to identify ambiguous states. For example, when the model predicts a 49% probability for both options, the maximum probability method might misclassify it as having moderate confidence, while the difference method can accurately identify extremely low confidence, thus triggering high-precision calculations and significantly improving the model's robustness in handling ambiguous inputs.

[0041] S4: Determine the target quantization precision corresponding to the confidence score based on the preset mapping relationship between confidence level and quantization level.

[0042] The system can predefine a mapping table between confidence intervals and quantization precision.

[0043] For example, by setting the first threshold T1=0.4 and the second threshold T2=0.8, we can obtain the following ranges of threshold T.

[0044] Low confidence interval (T<0.4): corresponds to the first quantization precision, such as INT8 or FP16.

[0045] High confidence interval (T>0.8): corresponds to the second quantization precision, such as INT4 or INT2.

[0046] Medium confidence interval (0.4≤T≤0.8): corresponds to the third quantization precision, such as INT4-groupwise.

[0047] In some embodiments, determining the target quantization precision corresponding to the confidence score based on a preset mapping relationship between confidence score and quantization level includes: comparing the confidence score with at least one preset threshold; if the confidence score is in a first numerical range, selecting a first quantization precision as the target quantization precision; if the confidence score is in a second numerical range, selecting a second quantization precision as the target quantization precision; wherein the number of bits in the first quantization precision is different from the number of bits in the second quantization precision.

[0048] Because a threshold is preset during the configuration phase, the threshold can divide a continuous confidence score range into multiple non-overlapping numerical intervals.

[0049] During runtime, the real-time calculated confidence score is compared to these thresholds: when the confidence score falls into the first numerical range (low confidence range), a first control instruction is triggered to select the first quantization precision (high bit width precision); when the confidence score falls into the second numerical range (high confidence range), a second control instruction is triggered to select the second quantization precision (low bit width precision). The first and second quantization precisions differ significantly in the number of bits used to represent the data. For example, the first quantization precision uses 16-bit floating-point representation, while the second quantization precision uses 4-bit integer representation.

[0050] By discretizing the threshold interval, complex continuous variable mapping problems can be transformed into simple logical judgment problems, greatly reducing the computational overhead and latency of the decision-making process. Simultaneously, the hierarchical mechanism allows developers to flexibly define confidence levels according to business needs, providing highly configurable system elasticity and enabling the same system to adapt to business scenarios with varying fault tolerance levels.

[0051] S5: Based on the target quantization accuracy, quantize the network parameters and / or intermediate activation values ​​of the model to obtain a dynamic quantization model.

[0052] Understandably, quantization can be either the dynamic loading of weights or the dynamic truncation of the calculation process.

[0053] Specifically, the network parameters and / or intermediate activation values ​​of the model are quantized, including: directly calling the weight copy corresponding to the target quantization precision from multiple pre-stored model weight copies of different precisions; or, obtaining the full-precision floating-point weights of the model and calculating the corresponding quantization weights and quantization parameters in real time according to the target quantization precision.

[0054] In one example, a multi-copy switching method can be used. Weight copies of the same model with different precisions can be pre-stored in the storage medium. For example, one weight file in INT8 format and one weight file in FP16 format can be stored separately. Once the target quantization precision is determined, the corresponding precision weight copy can be directly loaded into the computation unit via memory pointer redirection.

[0055] Specifically, multiple sets of weight copies of the model can be stored in memory in advance. In this case, the inference speed is the fastest, but the storage space required is slightly larger.

[0056] In another example, real-time quantization can be used. The model's full-precision floating-point weights can be stored in video memory. Once the target quantization precision is determined, an online quantization kernel is activated. During the data transfer from video memory to the computation registers, the corresponding quantization weight values ​​and quantization parameters such as scaling factors and zero-point offsets are calculated in real time using a vectorized instruction set.

[0057] For example, the model can retain full-precision (FP16 / FP32) weights in GPU memory. Once the target precision is determined, such as INT8, the FP16 values ​​can be scaled and rounded to INT8 format in real-time as the data is read from GPU memory to the computation unit, and the corresponding Scale and Zero-point parameters can be calculated. In this case, GPU memory can be saved, and flexibility is high.

[0058] Understandably, the input data to be processed can be received through the input interface. In a natural language processing scenario, this input data is a token sequence after word segmentation; in a computer vision scenario, this input data is a preprocessed image tensor.

[0059] The model can be a large language model based on the Transformer architecture. The quantization of the network parameters and / or intermediate activation values ​​of the model is performed on specific levels or modules of the model. Different levels or modules are assigned different target quantization precisions according to their respective local confidence.

[0060] For large language models based on the Transformer architecture, finer-grained control can be performed. For example, the model can be divided into multiple logical layers, such as Attention layers, Feed-Forward layers, or modules.

[0061] During inference, not only the overall confidence level can be calculated, but also the local confidence level can be calculated for the output features of specific layers. Based on the local confidence level of each layer, different target quantization precision can be independently assigned to each layer. For example, for the Attention layer, which is sensitive to precision and has low local confidence, high bit precision is assigned; for the Feed-Forward layer, which has a large number of parameters but high redundancy, if its local confidence is high, low bit precision is assigned.

[0062] Understandably, the granularity of quantization optimization can be refined from the entire model to the layer / module level, enabling precise on-demand adjustment of accuracy. By configuring differentiated hybrid precision strategies, it is possible to deeply compress the most redundant parts of the model with the largest number of parameters without losing almost the overall prediction accuracy, thereby achieving a higher compression ratio and acceleration effect than uniform quantization.

[0063] Understandably, by invoking the model, forward propagation computation can be performed on the input data. In this step, to balance overhead, either full model inference can be performed, or only shallow partial inference of the model can be executed. The computation terminates at the model's classification head or output layer, thereby obtaining the initial probability distribution result for the current task objective. This result is a normalized numerical vector, where each element represents the predicted probability of the corresponding class.

[0064] Understandably, by reading the initial probability distribution results mentioned above, a quantitative numerical indicator, namely the confidence score, is calculated based on the characteristics of the distribution. The confidence score characterizes the model's degree of confidence or cognitive clarity regarding the current input data.

[0065] Furthermore, by calling a pre-defined mapping table in memory, the mapping table defines the logical correspondence between confidence scores and quantization levels, such as bit width depth. The calculated confidence score can be used as an index key to search and match the corresponding target quantization precision in the mapping table. The target quantization precision defines the bit width that the model parameters or activation values ​​should maintain in subsequent calculations, such as 8 bits, 4 bits, or 16 bits of floating-point.

[0066] It should be noted that, based on the determined target quantization accuracy, the dynamic quantization executor quantizes the network parameters of the model and / or the intermediate activation values ​​generated during inference, constructing a dynamic quantization model. Subsequently, this dynamic quantization model can be used to complete the final inference calculation on the input data, and the calculation result can be used as the final output.

[0067] The dynamic quantization model optimization method provided by the present invention calculates the confidence level as the decision variable for quantization accuracy, rather than relying on the uniform accuracy of the entire model or adjusting the accuracy solely based on hardware load in traditional quantization techniques. This enables dynamic matching between model calculation accuracy and data complexity, thereby achieving more intelligent and efficient model inference.

[0068] This invention allows for flexible configuration of the mapping relationship. Depending on the application scenario, the confidence score and the number of bits in the quantization precision are positively or negatively correlated. When using an aggressive efficiency strategy, if the confidence score is higher than a preset confidence threshold, a quantization precision with a lower number of bits is selected as the target quantization precision. When using a conservative quality strategy, if the confidence score is higher than a preset confidence threshold, a quantization precision with a higher number of bits is selected as the target quantization precision. The preset confidence threshold can be a relatively high value.

[0069] Strategy 1: Aggressive efficiency strategy, corresponding to a negative correlation. In this mode, it is assumed that high confidence means the task is simple and does not require high precision. When the confidence score is detected to be higher than the preset high confidence threshold, the current task can be determined to be a simple task, and a lower quantization precision, such as 2-bit, can be actively selected to pursue the ultimate processing speed; conversely, it only falls back to high bit precision when the confidence is low.

[0070] Strategy Two: Conservative Quality Strategy, Corresponding to Positive Correlation. In this mode, it is assumed that high confidence means the model has captured key details, requiring high precision to preserve them. When the confidence score is detected to be higher than a preset high confidence threshold, to prevent quantization noise from damaging the accurate features already established by the model, a higher quantization precision, such as 16-bit, can be selected. Conversely, when the confidence is extremely low, it is determined that the model output itself has limited value, thus reducing precision to save unnecessary computational power.

[0071] In this implementation, by providing configurable strategy directions, the contradiction between rapid response and precise calculation that a single quantization strategy cannot simultaneously be achieved is resolved. Aggressive strategies are suitable for latency-sensitive scenarios such as instant messaging and instant translation; conservative strategies are suitable for scenarios with extremely high accuracy requirements, such as medical assistance and financial analysis, greatly expanding the application scope and improving scenario adaptability.

[0072] In some embodiments, determining the target quantization accuracy corresponding to the confidence score based on a preset mapping relationship between confidence score and quantization level further includes: acquiring hardware resource status data of the current operating environment, wherein the hardware resource status data includes at least one of memory occupancy and CPU / GPU load; and determining the target quantization accuracy based on the confidence score and the hardware resource status data.

[0073] Understandably, the model's deployment system can integrate hardware monitoring interfaces to periodically poll the resource status data of the current operating environment, including but not limited to the system's real-time memory usage, CPU / GPU computing load, and device temperature.

[0074] When determining the target quantization precision, in addition to considering the confidence score, hardware status can also be introduced as a correction factor. For example, when memory usage is detected to be close to the physical limit, regardless of the confidence score, the system will forcibly overwrite the original mapping logic and prioritize the selection of low-bit quantization precision to free up memory space; when hardware resources are detected to be idle, the restrictions on high-precision calculations will be appropriately relaxed.

[0075] In this context, by combining software-level confidence with hardware-level resource load, a more robust system-level protection mechanism can be built. This effectively prevents program crashes or device lag caused by forcibly performing high-precision calculations when hardware resources are scarce, ensuring the availability and stability of the system under extreme conditions.

[0076] In some embodiments, the dynamic quantization model optimization method of the present invention further includes a feedback optimization step: using the dynamic quantization model to perform inference calculations on the input data to generate the final output result; obtaining the confidence score, the selected target quantization precision, and user feedback or quality score of the generated final output result during the inference process; and dynamically adjusting the threshold in the mapping relationship between confidence and quantization level based on the user feedback or quality score.

[0077] After completing the inference and outputting the results, a background logging process can be started to record the confidence score of this inference, the quantization precision used, and subsequent user feedback or quality scores. User feedback can be explicit rating signals or implicit behavioral signals such as whether to regenerate or adopt the results.

[0078] The optimization algorithm can be run periodically to analyze the historical data. If it is found that using a specific quantization precision within a certain confidence interval causes the negative feedback rate to exceed a preset warning threshold, the thresholds in the mapping relationship can be automatically adjusted. For example, the confidence threshold for triggering low-bit quantization can be increased, making the system more cautious when facing similar situations in the future.

[0079] In this embodiment, by introducing a closed-loop feedback mechanism, the quantization strategy is no longer a static rule that is fixed at the factory, but has the ability to learn online and self-correct. This allows the model optimization system to gradually adapt to the behavioral habits and quality requirements of specific user groups over time, thereby continuously improving the user experience.

[0080] The dynamic quantization model optimization apparatus provided by the present invention will be described below. The dynamic quantization model optimization apparatus described below can be referred to in correspondence with the dynamic quantization model optimization method described above.

[0081] like Figure 2 As shown, the dynamic quantization model optimization device of this embodiment includes an acquisition module 210, a first processing module 220, a second processing module 230, a third processing module 240, and a fourth processing module 250.

[0082] The acquisition module 210 is used to acquire the input data to be processed; The first processing module 220 is used to perform forward propagation calculations on the input data using the model to obtain the initial probability distribution results; The second processing module 230 is used to calculate the confidence score of the model for the input data based on the initial probability distribution results; The third processing module 240 is used to determine the target quantization accuracy corresponding to the confidence score based on the preset mapping relationship between confidence score and quantization level. The fourth processing module 250 is used to quantize the network parameters and / or intermediate activation values ​​of the model according to the target quantization accuracy to obtain a dynamic quantization model.

[0083] The dynamic quantization model optimization device provided in this embodiment of the invention calculates the confidence level as the decision variable for quantization accuracy, rather than relying on the uniform accuracy of the entire model or adjusting the accuracy solely based on hardware load in traditional quantization techniques. This enables dynamic matching between model calculation accuracy and data complexity, thereby achieving more intelligent and efficient model inference.

[0084] Figure 3 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 3, the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a dynamic quantization model optimization method. This method includes: acquiring input data to be processed; performing forward propagation calculations on the input data using a model to obtain an initial probability distribution result; calculating the model's confidence score for the input data based on the initial probability distribution result; determining the target quantization accuracy corresponding to the confidence score according to a preset mapping relationship between confidence and quantization level; and quantizing the model's network parameters and / or intermediate activation values ​​according to the target quantization accuracy to obtain a dynamic quantization model.

[0085] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the dynamic quantization model optimization method provided by the above methods. The method includes: acquiring input data to be processed; performing forward propagation calculation on the input data using a model to obtain an initial probability distribution result; calculating the confidence score of the model for the input data based on the initial probability distribution result; determining the target quantization accuracy corresponding to the confidence score according to a preset mapping relationship between confidence and quantization level; and quantizing the network parameters and / or intermediate activation values ​​of the model according to the target quantization accuracy to obtain a dynamic quantization model.

[0087] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a model optimization method for dynamic quantization provided by the methods described above. The method includes: acquiring input data to be processed; performing forward propagation calculation on the input data using a model to obtain an initial probability distribution result; calculating a confidence score of the model for the input data based on the initial probability distribution result; determining a target quantization precision corresponding to the confidence score according to a preset mapping relationship between confidence and quantization level; and quantizing the network parameters and / or intermediate activation values ​​of the model according to the target quantization precision to obtain a dynamic quantization model.

[0088] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0089] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dynamic quantization model optimization method, characterized in that, include: Obtain the input data to be processed; The model is used to perform forward propagation calculations on the input data to obtain the initial probability distribution results; Calculate the confidence score of the model for the input data based on the initial probability distribution results; Based on the preset mapping relationship between confidence level and quantization level, the target quantization accuracy corresponding to the confidence score is determined; Based on the target quantization accuracy, the network parameters and / or intermediate activation values ​​of the model are quantized to obtain a dynamic quantization model.

2. The model optimization method for dynamic quantization according to claim 1, characterized in that, The calculation of the confidence score of the model for the input data based on the initial probability distribution results includes at least one of the following methods: Calculate the information entropy of the initial probability distribution result, and use the negative correlation value of the information entropy as the confidence score; Extract the maximum probability value from the initial probability distribution results, and use the maximum probability value as the confidence score; Extract the maximum probability value and the second-highest probability value from the initial probability distribution results, calculate the difference or ratio between them, and use the difference or ratio as the confidence score.

3. The model optimization method for dynamic quantization according to claim 1, characterized in that, The step of determining the target quantization accuracy corresponding to the confidence score based on the preset mapping relationship between confidence score and quantization level includes: The confidence score is compared with at least one preset threshold; If the confidence score is within the first numerical range, then the first quantization precision is selected as the target quantization precision; If the confidence score is in the second numerical range, then the second quantization precision is selected as the target quantization precision; The number of bits for the first quantization precision is different from the number of bits for the second quantization precision.

4. The model optimization method for dynamic quantization according to claim 3, characterized in that, The confidence score is positively or negatively correlated with the number of bits of quantization precision; When an aggressive efficiency strategy is adopted, if the confidence score is higher than the preset confidence threshold, the quantization precision with a lower number of bits is selected as the target quantization precision. When a conservative quality strategy is adopted, if the confidence score is higher than the preset confidence threshold, the quantization precision with a higher number of bits is selected as the target quantization precision.

5. The model optimization method for dynamic quantization according to claim 1, characterized in that, Quantization of the model's network parameters and / or intermediate activation values ​​includes: From multiple pre-stored model weight copies of different precisions, directly call the weight copy corresponding to the target quantization precision; Alternatively, the full-precision floating-point weights of the model can be obtained, and the corresponding quantization weights and quantization parameters can be calculated in real time according to the target quantization precision.

6. The model optimization method for dynamic quantization according to claim 1, characterized in that, The method also includes a feedback optimization step: The dynamic quantization model is used to perform inference calculations on the input data to generate the final output result. Obtain the confidence score, the selected target quantization precision, and user feedback or quality score of the generated final output during this inference process; Based on the user feedback or quality score, the threshold in the mapping relationship between the confidence level and the quantization level is dynamically adjusted.

7. The model optimization method for dynamic quantization according to claim 1, characterized in that, The step of determining the target quantization accuracy corresponding to the confidence score based on the preset mapping relationship between confidence score and quantization level further includes: Obtain hardware resource status data of the current operating environment, wherein the hardware resource status data includes at least one of memory utilization and CPU / GPU load; The target quantization accuracy is determined based on the confidence score and the hardware resource status data.

8. The model optimization method for dynamic quantization according to claim 1, characterized in that, The model is a large language model based on the Transformer architecture. The quantization processing of the model's network parameters and / or intermediate activation values ​​is performed on specific levels or modules of the model. Different levels or modules are assigned different target quantization precisions according to their respective local confidence levels.

9. A dynamic quantization model optimization device, characterized in that, include: The acquisition module is used to acquire the input data to be processed. The first processing module is used to perform forward propagation calculations on the input data using the model to obtain the initial probability distribution result; The second processing module is used to calculate the confidence score of the model for the input data based on the initial probability distribution result; The third processing module is used to determine the target quantization accuracy corresponding to the confidence score based on the preset mapping relationship between confidence score and quantization level. The fourth processing module is used to quantize the network parameters and / or intermediate activation values ​​of the model according to the target quantization accuracy to obtain a dynamic quantization model.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it implements the model optimization method of dynamic quantization as described in any one of claims 1 to 8.

Images