A model effect evaluation system and method based on quantitative reasoning

Abstract

The invention discloses a model effect evaluation system and method based on quantitative reasoning, and relates to the technical field of quantitative reasoning; the evaluation system comprises functional modules: a model loading and preprocessing module, a quantitative processing module, a reasoning process simulation module, a performance evaluation module and a result feedback module; the technical key points are: implementing a dynamic adjustment optimization strategy according to the situation, simulating various test sets for multiple times, and accurately judging whether the model performance is stable after calculation; once it is detected that the comprehensive effect estimation exceeds the error range, the adjustment optimization process is triggered, and the weight distribution is optimized to ensure the rationality of the quantitative parameters; if the problem still exists, the model accuracy is further adjusted to gradually approach the optimal performance point; in this process, a dynamic adjustment formula is also introduced to realize automatic adjustment of the accuracy adjustment range, which not only ensures the timeliness of the adjustment, but also avoids the instability caused by excessive adjustment.

Description

Technical Field

[0001] The present invention relates to the field of quantitative reasoning technology, and in particular to a model effect evaluation system and method based on quantitative reasoning. Background Art

[0002] With the widespread application of large language models (LLMs), reasoning efficiency and resource consumption have become the main bottlenecks restricting their expansion in practical applications, especially in complex tasks such as text and image prompt word generation, optimizing memory usage and reasoning speed has become particularly important; among them, quantized reasoning is an important technology, which mainly involves using lower-precision data for calculations and memory access, aiming to improve reasoning efficiency; according to public information, quantized reasoning can significantly reduce the storage space of the model and reduce computing requirements through specific quantization techniques, such as converting the parameters of the model from floating-point numbers to finite integer representations; this technology can accelerate the reasoning process while maintaining a certain accuracy, especially on devices with limited resources, such as embedded systems, edge computing and mobile devices, the advantages of quantized reasoning are particularly obvious; quantized reasoning plays an important role in the application of current deep learning models and is a key means to optimize model performance and improve reasoning efficiency.

[0003] The existing application number is 202111426152.4, and the technical solution pointed out in the document named A flexible riser risk assessment method based on quantitative analysis and probabilistic reasoning is: the method includes the following steps: Step 1, construct a fault tree with "flexible riser failure" as the top event; Step 2, improve the risk matrix method based on the two-dimensional cloud model theory to obtain a "comment cloud matrix"; Step 3, expert evaluation and evaluation result aggregation: evaluate the bottom events, aggregate the evaluation results, obtain a comprehensive cloud of evaluation results, compare it with the "comment cloud matrix", and judge the risk level: Step 4, calculate the fuzzy failure rate of the bottom event; Step 5, Bayesian network reasoning, judge the probability of the top event under the specific state, and realize fault tracing; However, its evaluation method is relatively single and cannot adapt to different model situations, which leads to the accuracy of its evaluation results needs to be further improved.

[0004] Combining the above documents and existing technologies:

[0005] When traditional quantitative reasoning technology is used in combination with large language models, how to build an inference framework that can flexibly integrate and evaluate various quantized large language models; how to comprehensively and objectively evaluate the impact of different quantization strategies on model performance; and how to quantify the overall performance of different quantization strategies through comprehensive effect estimation so as to make the best choice in practical applications are issues that need to be addressed urgently. As for whether the model's operating performance is stable and how to adjust and optimize it when the performance is unstable, the traditional method can only ensure the timeliness of the adjustment. During the adjustment process, there may be over-adjustments, which may cause the model to be unstable in subsequent use. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In view of the deficiencies in the prior art, the present invention provides a model effect evaluation system and method based on quantitative reasoning, which solves the problems raised in the background technology by running various functional modules configured in the system.

[0008] (II) Technical solution

[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0010] A model effect evaluation system based on quantitative reasoning, the evaluation system comprising:

[0011] Model loading and preprocessing module, loads the original large language model and performs preprocessing;

[0012] The quantization processing module analyzes and determines the weight distribution of the large language model, and matches the quantization method based on the determination results; it simultaneously runs the pre-built quantization method library and allocates high- and low-precision quantization methods based on the task characteristics.

[0013] The reasoning process simulation module builds the reasoning framework, integrates the quantized large language model, runs the predefined test set for reasoning, and records the reasoning indicator data set;

[0014] The performance evaluation module builds a linear aggregation calculation model based on the inference indicator data set obtained after inference simulation for each type of test set, and generates a comprehensive effect estimate corresponding to each type of test set;

[0015] The result feedback module obtains N comprehensive effect estimates after simulating various test sets N times. When the comprehensive effect estimate is outside the error range, an adjustment optimization strategy is executed.

[0016] Furthermore, the original large language model is loaded using a model loading library, and preprocessing includes model structure analysis and weight format conversion.

[0017] Furthermore, the process of weight distribution analysis is as follows:

[0018] Perform statistical analysis on the weights of the large language model to obtain the distribution characteristic parameters of the weights, including the mean, standard deviation, maximum value, and minimum value; extract the mean and standard deviation from the distribution characteristic parameters;

[0019] Compare the ratio of the standard deviation to the mean with a preset cutoff value:

[0020] If the ratio exceeds the defined threshold, the weight distribution is determined to be uneven; otherwise, the weight distribution is determined to be uniform.

[0021] Furthermore, the process of matching the quantization method according to the determination result is as follows:

[0022] If the weight distribution is determined to be uniform, the uniform quantization method is matched;

[0023] If the weight distribution is determined to be non-uniform, the non-uniform quantization method is matched.

[0024] Furthermore, the process of assigning high and low precision quantization methods based on task characteristics is as follows:

[0025] For a type of task, a high-precision quantization method is selected, and the high precision is float16;

[0026] For the second type of tasks, a low-precision quantization method is selected, and the low precision is int8;

[0027] For three types of tasks, a hybrid quantization mechanism is implemented;

[0028] Among them, the task characteristics of the first category of tasks are: high accuracy requirements, and the first category of tasks includes image recognition and natural language processing; the task characteristics of the second category of tasks are: fast response is required, and the second category of tasks includes online reasoning and real-time detection; the task characteristics of the third category of tasks are: meeting the task characteristics of the first and second categories of tasks at the same time, and the third category of tasks includes real-time image analysis;

[0029] The process of implementing the hybrid quantization mechanism is as follows:

[0030] Identify the key parts: Identify the last Q convolutional layers and fully connected layers near the output layer as the key parts;

[0031] Select quantization accuracy: use high-precision quantization for key parts and low-precision quantization for non-key parts;

[0032] The value of Q is a positive integer and is at least 1.

[0033] Furthermore, the predefined test sets include: uniform quantization + high precision, uniform quantization + low precision, uniform quantization + high and low precision mixed, non-uniform quantization + high precision, non-uniform quantization + low precision, non-uniform quantization + high and low precision mixed;

[0034] Recorded inference metrics datasets:

[0035] When low precision exists in the test set, the inference indicator data set includes: inference time, percentage of accuracy drop, memory usage, and energy consumption; when low precision does not exist in the test set, the inference indicator data set includes: inference time, percentage of accuracy drop, and memory usage.

[0036] Furthermore, when there is low precision in the corresponding class test set, the formula running in the constructed linear aggregation calculation model is as follows:

[0037] ;

[0038] Where, low_pre represents the comprehensive effect estimation, α, β, γ, δ, ε are weight coefficients, and their value ranges are all [0, 1]. T represents the inference time, ΔA represents the percentage of accuracy decrease, Mj represents the baseline value, M represents the memory usage, and E represents the energy consumption.

[0039] When there is no low precision in the corresponding class test set, the formula running in the constructed linear aggregation calculation model is as follows:

[0040] ;

[0041] Wherein, α', β', γ', δ' are another set of weight coefficients, and their value ranges are all [0, 1].

[0042] Furthermore, when N simulations are performed on various test sets, where N is a positive integer and the value of N is at least 3, the error range is set as follows:

[0043] The average value of N comprehensive effect estimates is calculated, and the error is 0.1% of the average value. Then the error range is: [Lp_avg -0.1%Lp_avg, Lp_avg+0.1%Lp_avg], where Lp_avg represents the average value of N comprehensive effect estimates.

[0044] Further, the process of adjusting the optimization strategy is as follows:

[0045] Get the corresponding test set type and optimize the weight distribution;

[0046] If the weight distribution is uniform, the midpoint quantization algorithm is used for optimization;

[0047] If the weight distribution is uneven, a clustering algorithm is used for optimization;

[0048] If the comprehensive effect estimate is still outside the error range after optimization, the model accuracy is adjusted;

[0049] If it is high precision, the step-by-step precision increase instruction is triggered;

[0050] If the precision is low, the step-by-step precision reduction instruction is triggered;

[0051] If the precision is mixed with high and low precision, the step-by-step precision increase instruction and the step-by-step precision decrease instruction are triggered synchronously;

[0052] Until after N simulations, the comprehensive effect estimate is always within the error range;

[0053] Among them, when the step-by-step increase precision instruction or the step-by-step decrease precision instruction is triggered, the precision value increased or decreased each time is the same. According to the original step precision adjustment value and the maximum difference value of the comprehensive effect estimate exceeding the error range, a dynamic adjustment formula is constructed to obtain the step precision value after each adjustment:

[0054] ;

[0055] Where A represents the step accuracy value after each adjustment, P represents the original step accuracy adjustment value, k represents the adjustment coefficient, and its value range is [0, 1], Δmax is the maximum difference of the comprehensive effect estimate outside the error range, and oi represents the predetermined error limit.

[0056] A model effect evaluation method based on quantitative reasoning includes the following steps:

[0057] S1, load the original large language model and perform preprocessing;

[0058] S2. Analyze and determine the weight distribution of the large language model, and match the quantization method based on the determination results; synchronously run the pre-built quantization method library, and assign high- and low-precision quantization methods based on task characteristics;

[0059] S3. Build an inference framework, integrate the quantized large language model, run the predefined test set for inference, and record the inference indicator data set;

[0060] S4. Based on the inference indicator data set obtained after the inference simulation of each type of test set, a linear aggregation calculation model is built to generate a comprehensive effect estimation corresponding to each type of test set;

[0061] S5. After simulating various test sets N times, N comprehensive effect estimates are obtained. When the comprehensive effect estimate is outside the error range, an adjustment optimization strategy is executed.

[0062] (III) Beneficial effects

[0063] The present invention provides a model effect evaluation system and method based on quantitative reasoning, which has the following beneficial effects:

[0064] (1) In the quantization processing stage, this solution can intelligently analyze the weight distribution characteristics and match the most suitable quantization method to reduce the quantization error. At the same time, according to the requirements of task characteristics, it can flexibly allocate high-precision and low-precision quantization methods. For tasks that meet the requirements of both high precision and fast response, a hybrid quantization mechanism is implemented. By identifying the key parts and assigning high-precision quantization, and using low-precision quantization for the non-critical parts, the model accuracy and inference speed are effectively balanced.

[0065] (2) This solution uses predefined test sets to comprehensively evaluate the performance of the model under different quantization strategies, records key inference indicator data sets, and builds a linear aggregation calculation model to generate a comprehensive effect estimate for each type of test set based on the actual type of model's demand for inference indicator data. This estimate not only considers the individual performance of each inference indicator, but also reflects the interaction between them. At the same time, it can make the performance comparison of different quantization strategies more comprehensive, objective, and accurate.

[0066] (3) This solution implements dynamic adjustment and optimization strategies according to the situation, simulates various test sets multiple times, and calculates the average value and error range of the comprehensive effect estimate, so as to further accurately judge whether the model performance is stable; once it is detected that the comprehensive effect estimate exceeds the error range, the adjustment and optimization process is triggered to optimize the weight distribution to ensure the rationality of the quantitative parameters; if the problem still exists, the model accuracy is further adjusted by step-by-step increasing or decreasing the accuracy instructions to gradually approach the optimal performance point; in this process, a dynamic adjustment formula is also introduced to realize the automatic adjustment of the accuracy adjustment range, which not only ensures the timeliness of the adjustment, but also avoids the instability caused by excessive adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a modular schematic diagram of the evaluation system of the present invention;

[0068] Figure 2 A flow chart of the implementation of the adjustment and optimization strategy in the present invention;

[0069] Figure 3 Schematic diagram of the evaluation method steps of the present invention. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] Example 1

[0072] See also Figure 1 to Figure 2 , this embodiment provides a model effect evaluation system based on quantitative reasoning;

[0073] System Overview

[0074] The model effect evaluation system based on quantitative reasoning aims to evaluate the efficiency and accuracy of large language models under specific tasks by quantizing model weights and reasoning processes. The system combines quantitative reasoning technology to perform low-precision quantization on the model and evaluate the impact of quantization on model performance. The system consists of multiple functional modules that run in sequence to ensure the comprehensiveness and accuracy of the evaluation process.

[0075] Functional modules and technical solutions

[0076] The evaluation system includes the following functional modules: model loading and preprocessing module, quantization processing module, reasoning process simulation module, and performance evaluation module and result feedback module as the core of the evaluation system;

[0077] Model loading and preprocessing module, loads the original large language model and performs preprocessing;

[0078] Among them, the model loading library (such as PyTorch, TensorFlow) is used to load the model, and the preprocessing content includes model structure analysis and weight format conversion; the model structure is analyzed using tools to ensure the accuracy of subsequent quantization processing;

[0079] Function details:

[0080] Load the original large language model:

[0081] The first task of this module is to load pre-trained large language models stored on disk. These models are usually saved in a specific format, such as .pt or .pth files for PyTorch, .h5 or SavedModel format for TensorFlow, etc. The loading process needs to ensure the integrity and correctness of the model to avoid any corruption or inconsistency during the transmission or saving process;

[0082] Preprocessing:

[0083] Model structure analysis: After loading the model, the module needs to perform a detailed analysis of the model's structure, including identifying the number of layers in the model, the type of each layer (such as convolutional layer, fully connected layer, activation layer, etc.), the parameters of the layer (such as weights, biases, etc.), and the connection between them; structural analysis is the basis for subsequent quantization processing, because different layer types and structures may require different quantization strategies;

[0084] Weight format conversion: In some cases, the weight format of the original model may not be suitable for direct quantization; therefore, the module may need to convert the weights to a format that is more suitable for quantization, such as converting floating-point weights to fixed-point representation; weight format conversion also needs to consider the value range and precision loss to ensure that the performance of the converted model is not greatly affected;

[0085] The specific implementation process is described as follows:

[0086] Select the model loading library: Select the appropriate model loading library according to the original model saving format and subsequent processing requirements; for example, if the model is trained with PyTorch, then select PyTorch as the loading library; if the model is trained with TensorFlow, then select TensorFlow;

[0087] Load the model: Use the API provided by the selected model loading library to load the model; this usually involves calling the loading function in the library and passing in the path to the model file as a parameter;

[0088] Analyze the model structure: Use the tools or APIs provided by the model loading library to traverse the model structure; for example, in PyTorch, you can use model.named_parameters() or model.modules() to traverse the parameters and layers of the model; record the information of each layer, including type, parameters, input and output shapes, etc., for subsequent processing;

[0089] Through the above steps, the model loading and preprocessing module can ensure that the original large language model is correctly loaded, the structure is analyzed in detail, and the weight format (if necessary) is converted to a format suitable for subsequent quantization processing; this lays a solid foundation for subsequent model quantization, compression or optimization work.

[0090] Quantization processing module:

[0091] Perform weight distribution analysis on large language models to determine whether the weight distribution is uniform, and match quantization methods based on the determination results; simultaneously run the pre-built quantization method library to assign high- and low-precision quantization methods based on task characteristics;

[0092] The process of weight distribution analysis is as follows:

[0093] Perform statistical analysis on the weights of the large language model to obtain the distribution characteristic parameters of the weights, including the mean, standard deviation, maximum value, and minimum value; extract the mean and standard deviation from the distribution characteristic parameters;

[0094] Compare the ratio of the standard deviation to the mean with a preset cutoff value:

[0095] If the ratio exceeds the defined threshold, it means that the standard deviation is large relative to the mean, indicating that the weight distribution is relatively dispersed, and the weight distribution is determined to be non-uniform; if the ratio does not exceed the defined threshold, it means that the standard deviation is small relative to the mean, indicating that the weight distribution is relatively concentrated, and the weight distribution is determined to be uniform;

[0096] It should be noted that for the use of maximum and minimum values, the difference between the maximum and minimum values ​​can be used as verification. This difference (range) can reflect the degree of dispersion of the weights. If the range is large, it may indicate that the weight distribution is uneven; otherwise, the distribution is uniform, which plays a role in verifying the judgment results and can be selectively adopted according to actual needs;

[0097] The process of matching the quantization method according to the determination result is as follows:

[0098] If the weight distribution is determined to be uniform, the uniform quantization method is matched;

[0099] If the weight distribution is determined to be non-uniform, the non-uniform quantization method is matched;

[0100] Explanation: Uniform quantization and non-uniform quantization are two commonly used methods for quantizing model weights;

[0101] The uniform quantization method is applicable to the case where the weight distribution is uniform; this method divides the weight value range into several small intervals, and each interval is represented by a fixed value (usually the midpoint or average value of the interval), thereby realizing the discretization of the weight; this method is simple and easy to implement, and can effectively compress the model size and reduce the amount of calculation, and is particularly suitable for scenarios where the weight distribution is relatively concentrated; the non-uniform quantization method is aimed at the case where the weight distribution is non-uniform; due to the unevenness of the weight distribution, the method of dividing the interval equally may cause large quantization errors; therefore, the non-uniform quantization method adaptively adjusts the size and position of the quantization interval according to the actual distribution of the weight, so that the high-frequency weight values ​​are more finely quantized, while the low-frequency weight values ​​are quantized coarser; this method can more accurately retain the important information of the model and improve the performance of the quantized model, and is particularly suitable for scenarios with complex weight distribution or long tails.

[0102] Before running the quantization method library, analyze the accuracy requirements of the current task on the large language model;

[0103] The process of assigning high and low precision quantization methods based on task characteristics is as follows:

[0104] For the first type of tasks, that is, the task characteristics are: high precision requirements, a high-precision quantization method is selected, and the high precision is float16; for the second type of tasks, that is, the task characteristics are: fast response is required, a low-precision quantization method is selected, and the low precision is int8; for the third type of tasks, that is, the task characteristics are: meeting the task characteristics of both the first and second types of tasks, a hybrid quantization mechanism is implemented;

[0105] Among them, one type of tasks includes at least image recognition and natural language processing;

[0106] The second category of tasks includes at least online reasoning and real-time detection;

[0107] The three categories of tasks include at least real-time image parsing;

[0108] The process of implementing the hybrid quantization mechanism is as follows:

[0109] Identify the key parts: Identify the last Q convolutional layers and fully connected layers near the output layer as the key parts;

[0110] Select quantization accuracy: use high-precision quantization for key parts and low-precision quantization for non-key parts;

[0111] Among them, the value of Q is a positive integer and is at least 1. The confirmation of the key part can also be defined based on feature sensitivity and gradient size; Feature sensitivity: In the convolution layer, feature maps can be analyzed to determine which channels or filters have a greater impact on the final output, and these channels or filters are regarded as key parts; Gradient size: During back propagation, it can be observed which weights have larger gradients (i.e., exceeding the expected value) and give these weights higher precision during quantization; In practical applications, multiple methods can be combined to obtain more comprehensive key part identification;

[0112] Extended description: Train and test the quantized large language model to ensure that its accuracy and inference speed meet the requirements; adjust the hybrid quantization mechanism as needed until the best balance is found; try different key part identification methods and quantization accuracy combinations to find the best balance; this part is not the focus of this embodiment, so it will not be repeated here;

[0113] In addition, for tasks other than those in the quantitative method library, an independent selection button is provided to record the user's choice.

[0114] The solution can efficiently load and preprocess the original large language model to ensure the integrity and correctness of the model, and provide a solid foundation for subsequent quantization processing through detailed model structure analysis. In the quantization processing stage, it can intelligently analyze the weight distribution characteristics and match the most suitable quantization method (uniform quantization or non-uniform quantization) to reduce quantization errors. At the same time, according to the requirements of task characteristics, it can flexibly allocate high-precision and low-precision quantization methods, and even implement a hybrid quantization mechanism for tasks that meet both high-precision and fast response requirements. This hybrid quantization mechanism effectively balances model accuracy and inference speed by identifying key parts and assigning high-precision quantization, while using low-precision quantization for non-critical parts. In addition, the solution also provides an autonomous selection button to meet users' personalized quantization needs for specific tasks.

[0115] In summary, this technical solution solves the key technical problems in the quantization process of large language models, including the accuracy of model loading and preprocessing, the intelligent selection of quantization methods, and the balance between high precision and fast response requirements, providing a comprehensive and effective solution for model quantization, compression or optimization.

[0116] Reasoning process simulation module:

[0117] Build an inference framework, integrate the quantized large language model, run the predefined test set for inference, and record the inference indicator data set;

[0118] Among them, the construction of the inference framework: it means that a flexible and efficient inference framework needs to be built, which can easily integrate various quantized large language models; the inference framework should support multiple hardware platforms to ensure the wide applicability of the evaluation results; model integration: import the quantized model into the inference framework and ensure that the model can be loaded and initialized correctly; verify the integrity of the model to ensure that no damage or inconsistency is introduced during the quantization process;

[0119] Predefined test sets: These are representative test sets to comprehensively evaluate the performance of the model in different scenarios.

[0120] The predefined test suites include:

[0121] (1) Uniform quantization + high precision;

[0122] (2) Uniform quantization + low precision;

[0123] (3) Uniform quantization + high and low precision mixing;

[0124] (4) Non-uniform quantization + high precision;

[0125] (5) Non-uniform quantization + low precision;

[0126] (6) Non-uniform quantization + high and low precision mixing;

[0127] Among them, these six categories are representative test sets to ensure that subsequent evaluations are more comprehensive and accurate;

[0128] Recorded inference metrics datasets:

[0129] When there is low precision in the test set, the inference indicator data set includes: inference time (i.e., inference duration), percentage of accuracy drop, memory usage, and energy consumption (i.e., using energy consumption monitoring tools to measure the energy consumption of the model when it is running);

[0130] When there is no low precision in the test set, the inference indicator data set includes: inference time, accuracy drop percentage, and memory usage;

[0131] It should be noted that for inference time, percentage drop in accuracy, and memory usage, whether it is uniform quantization or non-uniform quantization, whether it is high precision, low precision or mixed precision, these three types of indicators are key indicators and are important indicators for evaluating the quantization effect; and energy consumption (i.e. energy usage) is only considered when low precision exists in the test set; when the test set contains low-precision quantization, energy usage becomes a particularly important indicator; low-precision quantization can usually significantly reduce the energy consumption of the model, which is especially important for resource-constrained environments such as mobile devices or embedded systems; therefore, in this case, special attention should be paid to energy usage to evaluate the energy efficiency improvement brought by quantization.

[0132] Performance evaluation module:

[0133] Based on the inference indicator data set obtained after inference simulation for each type of test set, a linear aggregation calculation model is built to generate the comprehensive effect estimation corresponding to each type of test set;

[0134] When there is low precision in the corresponding class test set, the formula running in the constructed linear aggregation calculation model is as follows:

[0135] ;

[0136] Where, low_pre represents the comprehensive effect estimation, α, β, γ, δ, ε are weight coefficients, and their value ranges are all [0, 1]. T represents the inference time, ΔA represents the percentage of accuracy decrease, Mj represents the baseline value, represents the expected maximum possible memory usage, M represents the memory usage, and E represents the energy consumption.

[0137] When there is no low precision in the corresponding class test set, the formula running in the constructed linear aggregation calculation model is as follows:

[0138] ;

[0139] In the formula, α', β', γ', δ' are another set of weight coefficients, and their value ranges are all [0, 1];

[0140] Logic design description:

[0141] Reasoning time T: Logarithmic transformation is used for standardization, because reasoning time usually spans multiple orders of magnitude, and logarithmic transformation can better handle this situation; Accuracy drop percentage ΔA: Linear transformation is used to convert it into a value between 0 and 1, indicating the degree of accuracy preservation; Memory usage M: Normalized by the ratio to the maximum possible memory usage; Energy consumption E (considered only when low precision exists): Also logarithmic transformation is used for standardization;

[0142] Considering that different indicators may have mutual influences (for example, faster inference time may lead to higher energy consumption), we can introduce some mutual influence factors (such as γ) to adjust the weights of each indicator. In fact, these factors can also be explained as weights; the weight coefficient is determined by the coefficient of variation method, which is a method of weighting each indicator according to the degree of variation between the current value and the target value of each evaluation indicator; if the numerical difference of an indicator is large, it can clearly distinguish the evaluated objects, indicating that the indicator has rich discrimination information, so the indicator should be given a larger weight; on the contrary, if the numerical difference of each evaluated object on a certain indicator is small, then the ability of this indicator to distinguish the evaluated objects is weak, so the indicator should be given a smaller weight; this method directly uses the information contained in each indicator to obtain the weight of the indicator through calculation, so it is objective;

[0143] In order to obtain a comprehensive evaluation value (i.e., comprehensive effect estimation), we can use nonlinear functions (such as exponential functions, logarithmic functions, or quadratic functions) to aggregate standardized indicators; in this way, the comprehensive evaluation value can not only reflect the individual performance of each indicator, but also reflect the interaction between them.

[0144] In this example table:

[0145] Please note that the weight coefficients (α, β, γ, δ, ε and α', β', γ', δ') here are assumed and used for example purposes only; in actual applications, these parameters can also be determined by experiments or expert experience.

[0146] The solution builds a flexible and efficient reasoning framework that can easily integrate and evaluate various large language models after quantization, ensuring the integrity and consistency of the model during the quantization process; using predefined test sets, the solution comprehensively evaluates the performance of the model under different quantization strategies (uniform quantization and non-uniform quantization, high-precision and low-precision hybrid), and records key reasoning indicator data sets, including reasoning time, percentage of accuracy drop, memory usage, and energy consumption (especially when quantized at low precision); by building a linear aggregation calculation model and combining the weight coefficient determined by the coefficient of variation method, a comprehensive effect estimate corresponding to each type of test set is generated. This estimate not only takes into account the individual performance of each indicator, but also reflects the interaction between them; this comprehensive evaluation method makes the performance comparison of different quantization strategies more comprehensive, objective and accurate.

[0147] Result feedback module:

[0148] After N simulations of various test sets, N comprehensive effect estimates are obtained. If the comprehensive effect estimate is always within the error range, no response action is taken; if the comprehensive effect estimate is outside the error range, an adjustment optimization strategy is executed;

[0149] Where N is a positive integer, and the value of N is at least 3 times;

[0150] The process of setting the error range is as follows:

[0151] Calculate the average value of N comprehensive effect estimates, the error is 0.1% of the average value, then the error range is: [Lp_avg -0.1%Lp_avg, Lp_avg+0.1%Lp_avg], where Lp_avg represents the average value of N comprehensive effect estimates;

[0152] The content of adjusting the optimization strategy is as follows:

[0153] Get the corresponding test set type and optimize the weight distribution;

[0154] If the weight distribution is uniform, the symmetric distribution characteristics are used to optimize the parameters using the midpoint quantization algorithm to reduce the error;

[0155] If the weight distribution is non-uniform, a clustering algorithm (or density assessment method) is used to analyze the weight distribution and determine the quantization points and quantization parameters;

[0156] If the comprehensive effect estimate is still outside the error range after optimization, the model accuracy will be adjusted;

[0157] If it is high precision, the step-by-step precision increase instruction is triggered;

[0158] If the precision is low, the step-by-step precision reduction instruction is triggered;

[0159] If the precision is mixed with high and low precision, the step-by-step precision increase instruction and the step-by-step precision decrease instruction are triggered synchronously;

[0160] Until after N simulations, the comprehensive effect estimate is always within the error range;

[0161] When the running time of the result feedback module exceeds the set threshold and has not yet terminated, the evaluation system is re-run;

[0162] Among them, no matter whether the step-by-step increase precision instruction or the step-by-step decrease precision instruction is triggered, the precision value of each increase or decrease (i.e., the adjustment amplitude) is the same. According to the original step precision adjustment value and the maximum difference value outside the error range of the comprehensive effect estimate, a dynamic adjustment formula is constructed to obtain the step precision value after each adjustment:

[0163] ;

[0164] In the formula, A represents the step accuracy value after each adjustment, P represents the original step accuracy adjustment value (this is a fixed accuracy adjustment step), k represents the adjustment coefficient, which is used to control the amplitude of the accuracy adjustment, and its value range is: [0, 1], Δmax is the maximum difference value of the comprehensive effect estimate outside the error range, and oi represents the predetermined error limit;

[0165] The logic is as follows:

[0166] When the maximum difference Δmax of the estimation is less than or equal to the predetermined error limit oi, it indicates that the current accuracy estimation is stable and does not need to be adjusted, so the adjusted step accuracy value A remains the original step accuracy adjustment value P;

[0167] When the maximum difference Δmax of the estimation exceeds the predetermined error limit oi, it indicates that the current accuracy estimation is unstable or biased and needs to be adjusted. At this time, the adjusted step accuracy value A will be calculated based on the original step accuracy adjustment value P and the maximum difference Δmax-oi exceeding the error. The adjustment coefficient k is used to control the adjustment range to ensure that the accuracy adjustment is neither too sensitive nor too slow.

[0168] The set threshold value in "When the running time of the result feedback module exceeds the set threshold value but is not terminated" is a fixed value set in advance. For example, if the running time of the result feedback module is 3 hours and the set threshold value is 2 hours, the maximum running time of the result feedback module is 2 hours. If it exceeds 2 hours, it means that the running time of the overall system is too long and there is a problem with the system itself, and the system needs to be restarted or run.

[0169] The solution implements a rigorous and dynamic adjustment and optimization strategy in the result feedback module. By simulating various test sets multiple times and calculating the average value and error range of the comprehensive effect estimate, the solution can accurately determine whether the model performance is stable. Once it is detected that the comprehensive effect estimate exceeds the error range, the solution will immediately start the adjustment and optimization process, first optimizing the weight distribution to ensure the rationality of the quantization parameters. If the problem still exists, the model accuracy will be further adjusted, and the accuracy instructions will be increased or decreased in a step-by-step manner to gradually approach the optimal performance point.

[0170] During this process, the solution also introduced a dynamic adjustment formula, which intelligently adjusted the accuracy adjustment range based on the maximum difference between the comprehensive effect estimate and the predetermined error limit, ensuring the timeliness of the adjustment while avoiding the instability caused by excessive adjustment. In addition, the solution also set a running time threshold for the result feedback module to ensure that the system can complete the evaluation and optimization within a reasonable time, avoiding system resource consumption and potential problems that may be caused by long-term operation.

[0171] Example 2

[0172] See also Figure 3 Based on Example 1, this embodiment further provides a model effect evaluation method based on quantitative reasoning, including the following specific steps:

[0173] S1, load the original large language model and perform preprocessing;

[0174] S2. Analyze and determine the weight distribution of the large language model, and match the quantization method based on the determination results; synchronously run the pre-built quantization method library, and assign high- and low-precision quantization methods based on task characteristics;

[0175] S3. Build an inference framework, integrate the quantized large language model, run the predefined test set for inference, and record the inference indicator data set;

[0176] S4. Based on the inference indicator data set obtained after the inference simulation of each type of test set, a linear aggregation calculation model is built to generate a comprehensive effect estimation corresponding to each type of test set;

[0177] S5. After simulating various test sets N times, N comprehensive effect estimates are obtained. When the comprehensive effect estimate is outside the error range, an adjustment optimization strategy is executed.

[0178] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. A person of ordinary skill in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A model effect evaluation system based on quantitative reasoning, characterized in that: The assessment system includes: Model loading and preprocessing module, loads the original large language model and performs preprocessing; The quantization processing module analyzes and determines the weight distribution of the large language model, and matches the quantization method based on the determination results; it simultaneously runs the pre-built quantization method library and allocates high- and low-precision quantization methods based on the task characteristics. The reasoning process simulation module builds the reasoning framework, integrates the quantized large language model, runs the predefined test set for reasoning, and records the reasoning indicator data set; The performance evaluation module builds a linear aggregation calculation model based on the inference indicator data set obtained after inference simulation for each type of test set, and generates a comprehensive effect estimate corresponding to each type of test set; The result feedback module obtains N comprehensive effect estimates after N simulations of various test sets. When the comprehensive effect estimates are outside the error range, the optimization strategy is adjusted. When N simulations are performed on various test sets, N is a positive integer and the value of N is at least 3; The process of setting the error range is as follows: Calculate the average value of N comprehensive effect estimates, and the error is 0.1% of the average value. The error range is: [Lp_avg -0.1%*Lp_avg, Lp_avg+0.1%*Lp_avg], where Lp_avg represents the average value of N comprehensive effect estimates. The process of adjusting the optimization strategy is as follows: Get the corresponding test set type and optimize the weight distribution; If the weight distribution is uniform, the midpoint quantization algorithm is used for optimization; If the weight distribution is uneven, a clustering algorithm is used for optimization; If the comprehensive effect estimate is still outside the error range after optimization, the model accuracy is adjusted; If it is high precision, the step-by-step precision increase instruction is triggered; If the precision is low, the step-by-step precision reduction instruction is triggered; If the precision is mixed with high and low precision, the step-by-step precision increase instruction and the step-by-step precision decrease instruction are triggered synchronously; Until after N simulations, the comprehensive effect estimate is always within the error range; Among them, when the step-by-step increase precision instruction or the step-by-step decrease precision instruction is triggered, the precision value increased or decreased each time is the same. According to the original step precision adjustment value and the maximum difference value of the comprehensive effect estimate exceeding the error range, a dynamic adjustment formula is constructed to obtain the step precision value after each adjustment: Where A represents the step accuracy value after each adjustment, P represents the original step accuracy adjustment value, k represents the adjustment coefficient, and its value range is [0, 1], Δmax is the maximum difference of the comprehensive effect estimate outside the error range, and oi represents the predetermined error limit.

2. A model effect evaluation system based on quantitative reasoning according to claim 1, characterized in that: The original large language model is loaded using the model loading library, and the preprocessing includes model structure analysis and weight format conversion.

3. The model effect evaluation system based on quantitative reasoning according to claim 1, characterized in that: The process of weight distribution analysis is as follows: Perform statistical analysis on the weights of the large language model to obtain the distribution characteristic parameters of the weights, including the mean, standard deviation, maximum value, and minimum value; extract the mean and standard deviation from the distribution characteristic parameters; Compare the ratio of the standard deviation to the mean with a preset cutoff value: If the ratio exceeds the defined threshold, the weight distribution is determined to be uneven; otherwise, the weight distribution is determined to be uniform.

4. A model effect evaluation system based on quantitative reasoning according to claim 3, characterized in that: The process of matching the quantization method according to the determination result is as follows: If the weight distribution is determined to be uniform, the uniform quantization method is matched; If the weight distribution is determined to be non-uniform, the non-uniform quantization method is matched.

5. The model effect evaluation system based on quantitative reasoning according to claim 1, characterized in that: The process of assigning high and low precision quantization methods based on task characteristics is as follows: For a type of task, a high-precision quantization method is selected, and the high precision is float16; For the second type of tasks, a low-precision quantization method is selected, and the low precision is int8; For three types of tasks, a hybrid quantization mechanism is implemented; Among them, the task characteristics of the first category of tasks are: high accuracy requirements, and the first category of tasks includes image recognition and natural language processing; the task characteristics of the second category of tasks are: fast response is required, and the second category of tasks includes online reasoning and real-time detection; the task characteristics of the third category of tasks are: meeting the task characteristics of the first and second categories of tasks at the same time, and the third category of tasks includes real-time image analysis; The process of implementing the hybrid quantization mechanism is as follows: Identify the key parts: Identify the last Q convolutional layers and fully connected layers near the output layer as the key parts; Select quantization accuracy: use high-precision quantization for key parts and low-precision quantization for non-key parts; The value of Q is a positive integer and is at least 1.

6. The model effect evaluation system based on quantitative reasoning according to claim 4 is characterized in that: The predefined test sets include: uniform quantization + high precision, uniform quantization + low precision, uniform quantization + high and low precision mixture, non-uniform quantization + high precision, non-uniform quantization + low precision, non-uniform quantization + high and low precision mixture; Recorded inference metrics datasets: When low precision exists in the test set, the inference indicator data set includes: inference time, percentage of accuracy drop, memory usage, and energy consumption; when low precision does not exist in the test set, the inference indicator data set includes: inference time, percentage of accuracy drop, and memory usage.

7. A model effect evaluation system based on quantitative reasoning according to claim 6, characterized in that: When there is low precision in the corresponding class test set, the formula running in the constructed linear aggregation calculation model is as follows: Where, low_pre represents the comprehensive effect estimation, α, β, γ, δ, ε are weight coefficients, and their value ranges are all [0, 1]. T represents the inference time, ΔA represents the percentage of accuracy decrease, Mj represents the baseline value, M represents the memory usage, and E represents the energy consumption. When there is no low precision in the corresponding class test set, the formula running in the constructed linear aggregation calculation model is as follows: Wherein, α', β', γ', δ' are another set of weight coefficients, and their value ranges are all [0, 1].

8. A model effect evaluation method based on quantitative reasoning, using any system described in claims 1 to 7, characterized in that: The steps include: S1, load the original large language model and perform preprocessing; S2. Analyze and determine the weight distribution of the large language model, and match the quantization method according to the determination result; Run the pre-built quantization method library simultaneously, and assign high- and low-precision quantization methods based on task characteristics; S3. Build an inference framework, integrate the quantized large language model, run the predefined test set for inference, and record the inference indicator data set; S4. Based on the inference indicator data set obtained after the inference simulation of each type of test set, a linear aggregation calculation model is built to generate a comprehensive effect estimation corresponding to each type of test set; S5. After N simulations are performed on various test sets, N comprehensive effect estimates are obtained. When the comprehensive effect estimates are outside the error range, an adjustment optimization strategy is executed; When N simulations are performed on various test sets, N is a positive integer and the value of N is at least 3; The process of setting the error range is as follows: Calculate the average value of N comprehensive effect estimates, and the error is 0.1% of the average value. The error range is: [Lp_avg -0.1%*Lp_avg, Lp_avg+0.1%*Lp_avg], where Lp_avg represents the average value of N comprehensive effect estimates. The process of adjusting the optimization strategy is as follows: Get the corresponding test set type and optimize the weight distribution; If the weight distribution is uniform, the midpoint quantization algorithm is used for optimization; If the weight distribution is uneven, a clustering algorithm is used for optimization; If the comprehensive effect estimate is still outside the error range after optimization, the model accuracy is adjusted; If it is high precision, the step-by-step precision increase instruction is triggered; If the precision is low, the step-by-step precision reduction instruction is triggered; If the precision is mixed with high and low precision, the step-by-step precision increase instruction and the step-by-step precision decrease instruction are triggered synchronously; Until after N simulations, the comprehensive effect estimate is always within the error range; Among them, when the step-by-step increase precision instruction or the step-by-step decrease precision instruction is triggered, the precision value increased or decreased each time is the same. According to the original step precision adjustment value and the maximum difference value of the comprehensive effect estimate exceeding the error range, a dynamic adjustment formula is constructed to obtain the step precision value after each adjustment: Where A represents the step accuracy value after each adjustment, P represents the original step accuracy adjustment value, k represents the adjustment coefficient, and its value range is [0, 1], Δmax is the maximum difference of the comprehensive effect estimate outside the error range, and oi represents the predetermined error limit.

Images