Efficiency evaluation method and device of model reasoning system and electronic equipment

Abstract

The invention discloses an efficiency evaluation method and device for a model reasoning system and electronic equipment, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a model parameter corresponding to a target model, a frame parameter of an operation environment of the target model, and a calculation parameter of a calculation node of the target model, and determining a target application scene of the target model; determining a weight set according to the target application scene; based on the standard reference values corresponding to the model parameters, the frame parameters and the calculation parameters, standard values corresponding to the model parameters, the frame parameters and the calculation parameters are determined; the weight set and standard values corresponding to the model parameters, the frame parameters and the calculation parameters are utilized to calculate a first score of the target model, and the first score is used for indicating the efficiency corresponding to the target model reasoning system. The technical problem that a large model inference system in the related technology cannot realize global performance evaluation is solved, and the technical effect of improving the performance evaluation accuracy of the inference system is achieved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an efficiency evaluation method, apparatus, and electronic device for a model reasoning system. Background Technology

[0002] In practical applications, the overall performance of a model inference system is often limited by the scheduling efficiency of the software framework, the rational allocation of hardware resources, and the synergistic optimization among these three factors. Many evaluations in related technologies are limited to measuring the model's output speed, such as the number of tokens generated per second. While high generation speed is important in some scenarios, it cannot fully reflect the system's performance under high concurrency. For example, in scenarios with multiple users accessing the system simultaneously, although the model generation speed may be fast, if the software framework's dynamic batching mechanism is inadequate, causing a large number of user requests to wait in the queue, the system's actual response speed and user experience will be significantly reduced. This highlights the problem that large model inference systems in related technologies cannot achieve global performance evaluation. Summary of the Invention

[0003] This application provides an efficiency evaluation method, apparatus, and electronic device for model inference systems, to at least solve the problem that large model inference systems in related technologies cannot achieve global performance evaluation.

[0004] This application provides an efficiency evaluation method for a model inference system, comprising: obtaining model parameters corresponding to a target model, framework parameters of the target model's operating environment, and computational parameters of the target model's computational nodes, and determining the target application scenario of the target model, wherein the target application scenario is used to indicate the task performed by the target model; determining a weight set based on the target application scenario, wherein the weight set is used to indicate the weight values ​​corresponding to the model parameters, framework parameters, and computational parameters respectively; determining standard values ​​corresponding to the model parameters, framework parameters, and computational parameters based on the benchmark reference values ​​corresponding to the model parameters, framework parameters, and computational parameters respectively; and calculating a first score for the target model using the weight set and the standard values ​​corresponding to the model parameters, framework parameters, and computational parameters respectively, wherein the first score is used to indicate the efficiency of the target model inference system.

[0005] This application also provides an efficiency evaluation device for a model inference system, comprising: a scenario determination module, used to acquire model parameters corresponding to the target model, framework parameters of the target model's operating environment, and computational parameters of the target model's computational nodes, and determine the target application scenario of the target model, wherein the target application scenario is used to indicate the task performed by the target model; a weight determination module, used to determine a weight set according to the target application scenario, wherein the weight set is used to indicate the weight values ​​corresponding to the model parameters, framework parameters, and computational parameters respectively; a normalization module, used to determine the standard values ​​corresponding to the model parameters, framework parameters, and computational parameters respectively based on the benchmark reference values ​​corresponding to the model parameters, framework parameters, and computational parameters respectively; and a model evaluation module, used to calculate a first score of the target model using the weight set and the standard values ​​corresponding to the model parameters, framework parameters, and computational parameters respectively, wherein the first score is used to indicate the efficiency of the target model inference system.

[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the efficiency evaluation method of any of the above-described model inference systems.

[0007] This application also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, it implements the steps of the efficiency evaluation method for any of the above-described model inference systems.

[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described model inference system efficiency evaluation methods.

[0009] This application obtains the model parameters corresponding to the target model, the framework parameters of the target model's runtime environment, and the computational parameters of the target model's computational nodes, and determines the target application scenario of the target model. Based on the target application scenario, a weight set is determined. Based on the benchmark reference values ​​corresponding to the model parameters, framework parameters, and computational parameters, standard values ​​corresponding to the model parameters, framework parameters, and computational parameters are determined. Using the weight set and the standard values ​​corresponding to the model parameters, framework parameters, and computational parameters, the first score of the target model is calculated. At the computing platform level, the weight mechanism can guide the effective allocation of resources. For example, if the target scenario has high requirements for memory bandwidth, the weight set will emphasize MemoryBandwidth, prompting system optimizers to prioritize improving memory performance or optimizing memory access patterns, thereby improving resource utilization efficiency. The scenario-based weight set can generate customized performance optimization strategies for each target model. For example, if memory usage at the model level becomes a bottleneck, techniques such as model pruning and quantization can be used to reduce memory consumption; if the dynamic batch processing capability of the software framework is insufficient, batch processing strategies can be optimized or a more efficient framework version can be adopted. Therefore, this solves the problem in related technologies where large model inference systems cannot achieve global performance evaluation. Attached Figure Description

[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of the hardware environment for an optional efficiency evaluation method for a model inference system according to an embodiment of this application;

[0012] Figure 2 This is a flowchart of an optional efficiency evaluation method for a model inference system according to an embodiment of this application;

[0013] Figure 3 This is a schematic diagram of an optional model inference system according to an embodiment of this application;

[0014] Figure 4 This is a schematic diagram illustrating data acquisition in an optional model inference system according to an embodiment of this application;

[0015] Figure 5 This is a schematic diagram of an optional display interface according to an embodiment of this application;

[0016] Figure 6This is a structural block diagram of an optional efficiency evaluation device for a model inference system according to an embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0018] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0019] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] According to one aspect of the embodiments of this application, an efficiency evaluation method for a model inference system is provided. As an optional implementation, the above-described efficiency evaluation method for a model inference system can be applied, but is not limited to, to systems such as... Figure 1 The efficiency evaluation system for the model inference system in the hardware environment shown. This efficiency evaluation system may include, but is not limited to, terminal device 102, network 110, server 112, and database 114. Terminal device 102 runs a target client (e.g., ...). Figure 1 As shown, taking a target client as an example capable of running large models. The aforementioned terminal device 102 includes a display 108, a processor 106, and a memory 104. The display 108 can be used to display the system interface, and also to provide a human-computer interaction interface to receive human-computer interaction operations on the interface and touch operations on different controls. The processor is used to generate interaction instructions in response to the aforementioned human-computer interaction operations and send the interaction instructions to the server. The memory is used to store system files.

[0021] In addition, server 112 includes a processing engine 116, which performs store or read operations on database 114. Specifically, processing engine 116 reads reference questions from database 114.

[0022] Assumption Figure 1The terminal device 102 runs a client for generating text responses (implemented through a large language model). The specific process in this embodiment is as follows: As in step S102, the terminal device 102 sends model parameters, framework parameters, and computational parameters to the server 112 via network 110. The server 112 executes steps S104-S110, obtaining the model parameters corresponding to the target model, the framework parameters of the target model's operating environment, and the computational parameters of the target model's computational nodes, and determines the target application scenario of the target model; determines the weight set based on the target application scenario; determines the standard values ​​corresponding to the model parameters, framework parameters, and computational parameters based on their respective benchmark reference values; and calculates the first score of the target model using the weight set and the standard values ​​corresponding to the model parameters, framework parameters, and computational parameters.

[0023] The application scenarios of the solution in this application are not limited to the examples above. The method of this application can be used on any computing platform that runs large models.

[0024] Embodiments of this application provide an efficiency evaluation method for a model inference system. Figure 2 This is a flowchart of an optional efficiency evaluation method for a model inference system according to an embodiment of this application; as follows: Figure 2 As shown, the efficiency evaluation method for this model inference system includes:

[0025] Step S202: Obtain the model parameters corresponding to the target model, the framework parameters of the target model's runtime environment, and the computation parameters of the target model's computation nodes, and determine the target application scenario of the target model, wherein the target application scenario is used to indicate the task executed by the target model;

[0026] It's important to clarify that the target model refers to the object of this evaluation and optimization, namely a specific large-scale pre-trained model. Its characteristics include a large number of parameters, making it suitable for various AI tasks such as natural language processing, image recognition, and speech recognition. Model parameters refer to all variable values ​​that constitute the mathematical structure of the target model, including but not limited to weights, biases, and activation functions. These parameters are learned from the dataset during the training phase and directly determine the model's predictive ability and performance. Framework parameters refer to the configuration information required for the software framework to run or deploy the target model, including but not limited to batch size, optimizer settings, learning rate, dynamic scheduling strategy, parallel processing options, and quantization support. These parameters affect the model's operating efficiency and resource consumption. Computational parameters involve the hardware attributes of computing nodes (such as GPUs, CPUs, TPUs, etc.), such as peak computing power, memory capacity, memory bandwidth, network I / O speed, and power consumption characteristics. These parameters limit the physical conditions for model operation and affect the speed, stability, and cost-effectiveness of model inference. The target application scenario specifically refers to the usage scenarios of the target model in practical applications, such as real-time dialogue systems, offline batch generation tasks, and edge computing deployments. Different application scenarios have different requirements for model performance. For example, real-time dialogue emphasizes response speed and latency, while batch generation tasks may focus more on processing efficiency and cost.

[0027] Step S204: Determine the weight set according to the target application scenario, wherein the weight set is used to indicate the weight values ​​corresponding to the model parameters, framework parameters and calculation parameters respectively;

[0028] It should be noted that the weight set is a set of numerical values ​​used to represent the relative importance of model parameters, framework parameters, and computational parameters in a specific target application scenario. The larger the weight value, the greater the influence of that parameter in the overall evaluation.

[0029] In traditional evaluation systems, each metric is often assigned a fixed weight, which may not accurately reflect the true performance requirements of a specific scenario. For example, real-time dialogue systems may place greater emphasis on latency and stability; therefore, latency-related metrics (such as latency and TP99) should be given higher weights in the weight set. On the other hand, offline batch processing scenarios may prioritize computational efficiency and cost, making the model generation efficiency (OTPS) and the energy efficiency ratio of the computing platform particularly important.

[0030] In an optional implementation, weights can be directly assigned to different parameters based on the knowledge and experience of experts in the field. Experts can determine which parameters have a greater impact on system efficiency based on their experience in previous projects or their understanding of the target application scenario, thus providing a more intuitive weight allocation.

[0031] The Analytic Hierarchy Process (AHP) can be used to first construct a hierarchical model, categorizing and arranging model parameters, framework parameters, and computational parameters. Then, through questionnaires, workshops, or other methods, experts from multiple fields can score the relative importance of each parameter. Finally, AHP is used to calculate the specific weight value for each parameter. This method can systematically evaluate the interrelationships and priorities among parameters, resulting in a more scientific set of weights.

[0032] In some cases, the setting of weight sets may be influenced by individual needs. For example, a company may be particularly concerned with energy efficiency and want to place greater emphasis on the energy efficiency ratio when evaluating the system. In this case, a custom interface or configuration file can be used to allow users or system administrators to manually adjust the weights of each parameter to meet specific optimization objectives.

[0033] Step S206: Based on the benchmark reference values ​​corresponding to the model parameters, framework parameters, and calculation parameters, determine the standard values ​​corresponding to the model parameters, framework parameters, and calculation parameters respectively.

[0034] It should be noted that the benchmark reference value is a performance metric set for the model parameters, framework parameters, and computational parameters. The standard value is a performance index with a unified scale obtained by comparing the measured values ​​with the benchmark reference value, used to intuitively reflect the performance of the parameters relative to the benchmark.

[0035] In optional implementations, for model parameters, baselines for OTPS and Latency are set based on industry best practices or resource constraints. For framework parameters, baselines for QPS and TP99 are set based on business needs and model characteristics. For computational parameters, baselines for peak computing power, memory bandwidth, and energy efficiency ratio are set directly based on hardware specifications. All calculated standard values ​​are further dimensionless to ensure they fall within the range of 0 to 1. Values ​​closer to 1 indicate performance that is close to or exceeds the baseline, while values ​​closer to 0 indicate performance that is significantly below the baseline. The standard values ​​for each calculated parameter are recorded as the basis for subsequent weighted summation calculations of the system efficiency comprehensive score. Simultaneously, the distribution of standard values ​​is analyzed to identify the strengths and weaknesses of system performance.

[0036] Step S208: Calculate the first score of the target model using the standard values ​​corresponding to the weight set, model parameters, frame parameters, and computational parameters. The first score is used to indicate the efficiency of the target model's inference system.

[0037] It should be noted that the first score is a comprehensive efficiency score of the target model calculated by combining performance indicators from three levels: the model, software framework, and computing platform, along with a weighted set. In an optional implementation, ensure that all key performance indicators for the model, framework, and computing platform have been obtained and appropriately normalized to ensure direct comparison. Clearly define the weighted set, adjusting weight values ​​based on the characteristics of the application scenario and the focus of the evaluation objective to ensure all key indicators are reasonably considered. Apply the corresponding weight to each indicator, then perform a weighted sum to obtain the first score of the target model.

[0038] Example 1:

[0039] S202, Data Acquisition: Run tests on the target system to collect data on OTPS, Latency, and Memory usage at the model level; QPS, TP99, and quantization support at the software framework level; and peak computing power, memory bandwidth, and performance per unit power consumption at the computing platform level.

[0040] S204 and S206, Weighting and Normalization: Assuming that in real-time dialogue system applications, latency and TP99 have relatively high weights, at 0.25 and 0.2 respectively; OTPS and QPS have a weight of 0.15; and the remaining indicators have a weight of 0.05. All collected indicator values ​​are normalized according to the rules.

[0041] S208, First score calculation: Based on the weights set in S204 and the normalized index values ​​(standard values) collected in S206, a weighted summation calculation is performed.

[0042] For example, by comparing and analyzing the system efficiency of companies A, B, and C in the field of real-time reasoning of large-scale language models, and by implementing a multi-dimensional weighted scoring method, we can evaluate and compare the actual performance of different companies' solutions in specific application scenarios.

[0043] Choose language models of the same scale and type (e.g., dialogue models using billions of parameters) and set the target application scenario as real-time multi-turn interactive dialogue to ensure consistency in evaluation dimensions and standards. Based on model parameters, framework parameters, and computational parameters, select OTPS (generation speed), Latency (response latency), Memory (GPU memory usage), QPS (throughput), TP99 (tail latency), peak computing power, memory bandwidth, and energy efficiency ratio as key performance indicators. Referencing industry standards, set a common benchmark value for each indicator. For example, OTPS is 1500 tokens / s, Latency is 200ms, Memory is 10GB, QPS is 8 queries / s, TP99 is 200ms, peak computing power is 78 TFLOPS (FP16), memory bandwidth is 2000GB / s, and energy efficiency ratio is 0.030 TFLOPS / W.

[0044] Based on the requirements of real-time dialogue scenarios, a weight set is defined. For example, assume OTPS weight is 0.15, Latency weight is 0.35, Memory weight is 0.1, QPS weight is 0.15, TP99 weight is 0.15, peak computing power weight is 0.05, memory bandwidth weight is 0.02, and energy efficiency ratio weight is 0.03. Note that the sum of all weights is 1. Real-world data is collected from each enterprise, including OTPS, Latency, Memory, and other metrics under specified models and scenarios, and a standard value for each metric is calculated. For each enterprise, a first score is calculated based on its standard value and weight set. The target model efficiency scores of each enterprise are compared to identify the enterprise solution that performs best in real-time multi-turn interactive dialogue applications. Simultaneously, by analyzing the contribution of each metric, performance bottlenecks in the system are assessed, providing guidance for system optimization and future technology selection.

[0045] If, after calculation, Company A scores 1.05, Company B scores 0.95, and Company C scores 1.07, this means that in real-time multi-turn interactive dialogue scenarios, Company C's large model inference system has the highest efficiency score, indicating that it performs best in terms of model processing speed, response latency, throughput, and energy efficiency. Company B, on the other hand, has the lowest score, possibly indicating that it fails to meet business requirements in some key performance indicators.

[0046] This approach not only allows for a direct comparison of system efficiency across different companies in similar applications, but also identifies performance bottlenecks, providing targeted guidance for system optimization and helping companies make better technology selection decisions and investment plans.

[0047] In an optional implementation, Company D has model usage requirements, such as wanting to implement functions like financial transactions and e-commerce queries through the model. It selects a suitable model from those provided by Companies A, B, and C. Company D can then determine the specific application scenario, such as financial transactions or e-commerce queries, and determine the corresponding weight set based on this scenario. Regardless of whether Companies A, B, and C provide the same type of model (i.e., A might provide a language model, while B and C might provide multimodal models), in this horizontal comparison, all three model inference systems use the weight set determined by the scenario; that is, the weight sets corresponding to the three model inference systems are the same during the horizontal comparison.

[0048] This application obtains the model parameters corresponding to the target model, the framework parameters of the target model's runtime environment, and the computational parameters of the target model's computational nodes, and determines the target application scenario of the target model. Based on the target application scenario, a weight set is determined. Based on the benchmark reference values ​​corresponding to the model parameters, framework parameters, and computational parameters, standard values ​​corresponding to the model parameters, framework parameters, and computational parameters are determined. Using the weight set and the standard values ​​corresponding to the model parameters, framework parameters, and computational parameters, the first score of the target model is calculated. At the computing platform level, the weight mechanism can guide the effective allocation of resources. For example, if the target scenario has high requirements for memory bandwidth, the weight set will emphasize MemoryBandwidth, prompting system optimizers to prioritize improving memory performance or optimizing memory access patterns, thereby improving resource utilization efficiency. The scenario-based weight set can generate customized performance optimization strategies for each target model. For example, if memory usage at the model level becomes a bottleneck, techniques such as model pruning and quantization can be used to reduce memory consumption; if the dynamic batch processing capability of the software framework is insufficient, batch processing strategies can be optimized or a more efficient framework version can be adopted. Therefore, this solves the problem in related technologies where large model inference systems cannot achieve global performance evaluation.

[0049] In an optional implementation, the model parameters corresponding to the target model, the framework parameters of the target model's runtime environment, and the computation parameters of the target model's computing nodes are obtained, including: determining the generation efficiency, response latency, and video memory usage corresponding to the target model as model parameters; determining the throughput, tail latency, and quantization support corresponding to the runtime environment as framework parameters; and determining the focused peak computing power, memory bandwidth, and energy efficiency ratio corresponding to the computing nodes as computation parameters.

[0050] It's important to note that model parameters are metrics directly related to the target model's performance, including generation efficiency (OTPS, the number of tokens generated per second), response latency (latency, the time to complete one inference operation), and memory usage (memory consumption during model runtime). Framework parameters are performance metrics related to the software framework used by the target model, including throughput (QPS, the number of concurrent queries processed per second), tail latency (TP99, the 99th percentile latency), and quantization support (quantization accuracy supported by the framework). Computational parameters are metrics related to the hardware computing nodes on which the target model depends, including peak performance (the theoretical maximum computing power of the hardware), memory bandwidth (the speed at which data is transferred between memory and the processor), and energy efficiency ratio (computing power per unit of power consumption).

[0051] Perform various inference tasks on the target model, measuring its generation efficiency (OTPS), response latency, and memory usage. This may include scenarios with input text of varying lengths and batch processing of different task types to ensure the comprehensiveness of model parameters. Under the target model's runtime environment, record the software framework's throughput (QPS), tail latency (TP99), and quantization support to evaluate its scheduling and optimization capabilities in high-concurrency, low-latency scenarios. Collect the target computing node's peak performance, memory bandwidth, and energy efficiency ratio to understand its hardware performance bottlenecks and resource utilization efficiency in the target application scenario.

[0052] Obtaining nine core quantitative indicators at the model, software framework, and computing platform levels is fundamental to evaluating system efficiency. The following details how to acquire this key data through experiments, data analysis, and software tools:

[0053] Model-level data acquisition:

[0054] OTPS (Output Time Per Second): This measures the number of tokens generated by the model within a given time period by running the model on a representative computing platform. This typically involves using the model inference API or directly calling the model code, statistically analyzing the generation speed, and ensuring that load conditions (such as input sequence length and concurrent queries) match the actual application scenario.

[0055] Latency: Also in a representative computing environment, using standard benchmarking tools such as MLPerf Inference Benchmark, the model's inference latency is tested. The time from when the input is accepted until the output is fully returned is recorded to evaluate the model's real-time responsiveness.

[0056] Memory Usage: During model execution, monitor memory usage using system performance monitoring tools (such as NVIDIA's NVidia-SMI, TensorBoard). Record the highest memory usage during model loading and execution to understand its hardware storage resource requirements.

[0057] Data acquisition at the software framework level:

[0058] Throughput (QPS): By constructing a high-concurrency test environment that simulates real-world applications, a large number of requests are sent to the inference framework, and the number of requests that can be processed per second is recorded. This requires setting reasonable request types, frequencies, and loads to reflect real-world service pressure.

[0059] Tail Latency (TP99): In high-concurrency testing, response time data for each request is collected, and then the upper limit of the 99% request completion time is calculated, which is the TP99 metric. This typically involves analyzing log files or using the statistical functions of performance monitoring tools.

[0060] Quantization Support: By analyzing the framework documentation and source code, this section identifies whether the framework supports different quantization algorithms (INT8, INT4, FP16, etc.) and scores the degree of support. Experiments can also be conducted to test quantization models and evaluate the framework's compatibility and performance changes before and after quantization, thus quantizing the framework's quantization support.

[0061] Data acquisition at the computing platform level:

[0062] Peak Performance: Consult the specifications provided by the hardware manufacturer to obtain the theoretical peak performance data of computing units such as GPUs and CPUs. Sometimes it is also necessary to verify the actual peak computing capabilities of the hardware under ideal conditions by running specific computing power test programs (such as Linpack and Dhrystone).

[0063] Memory bandwidth: Theoretical memory bandwidth data can also be obtained from hardware specifications. In actual measurements, memory bandwidth testing tools (such as Stream benchmark or Bandwidth Test) should be used to conduct tests under conditions free from other load interference to ensure the accuracy of the results.

[0064] Energy Efficiency Ratio (EER): Based on the known peak computing power of the hardware, power monitoring tools (such as IPMI, GPU power sensors) are used to record the actual power consumption of the system under different loads, and then the computing power output per unit power consumption is calculated, i.e., the energy efficiency ratio. This needs to be performed under stable environmental conditions to eliminate external interference and ensure data reliability.

[0065] It is important to note that during data collection, consistency and repeatability of all test conditions should be ensured. The same input dataset and runtime environment should be used to avoid data deviations caused by fluctuations in the test environment. For comprehensive evaluation, tests should be conducted under multiple load conditions (such as different concurrency levels and different input sequence lengths) to collect a series of data points, ensuring that the evaluation results cover different use cases. Industry-recognized standardized testing tools and methods should be used whenever possible to ensure the accuracy and comparability of data collection, avoiding errors or biases that may arise from self-designed methods.

[0066] Another method is to collect and measure key metrics at the software framework level during the actual operation of the model:

[0067] Utilize existing performance monitoring tools, such as Prometheus in conjunction with Grafana for data visualization, or develop a dedicated performance monitoring agent that can integrate into the framework's runtime process to monitor and record key performance parameters in real time. Configure the monitoring agent to track the framework's concurrent processing capacity (QPS), latency distribution, and model quantization level. Ensure the agent can seamlessly interact with the running model and framework without significantly increasing additional computational burden. While the model is running normally, the performance monitoring agent continuously records key metric data, which can be captured at fixed time intervals or when specific events are triggered.

[0068] Enable detailed logging within the software framework to capture processing time for each request, model inference time, and resource usage information. Regularly analyze log data to extract QPS, latency statistics, and memory usage. For latency, calculate the response time for each request using timestamps in the logs and calculate the TP99 value. Analyze the framework's ability to process quantization models through log analysis, including the proportion of requests running under quantization models and performance data, to evaluate the framework's actual quantization support.

[0069] When performance bottlenecks are detected (such as excessively high TP99 latency), adjust the framework's dynamic batch size, thread count, or other configuration parameters in real time to attempt to improve performance. Configure a real-time performance evaluation module that can immediately calculate the software framework's performance metrics and overall score based on currently collected data, providing immediate feedback to decision-makers. Utilize machine learning techniques, such as online learning algorithms, to automatically adjust the framework's configuration based on real-time performance data, seeking the optimal parameter combination to improve operational efficiency.

[0070] Figure 3 This is a schematic diagram of an optional model inference system according to an embodiment of this application; as shown... Figure 3 As shown, multiple models (such as model 1 to model n) can be run on computing node 306. Computing node 306 can run multiple models through software framework 304. For one of the models, it can be determined as the target model and its efficiency can be evaluated. For example, model 2 is the target model 302.

[0071] Figure 4 This is a schematic diagram illustrating data acquisition in an optional model inference system according to an embodiment of this application; as shown... Figure 4 As shown, evaluation node 402 can obtain data from three levels in the above manner, and evaluation node 402 can perform efficiency evaluation using the efficiency evaluation method of the built-in model inference system to obtain a first score. Moreover, evaluation node 402 can display the first score in various forms, such as through messages or displaying a radar chart on a monitor.

[0072] The above-described implementation methods of this application enable the collection of detailed performance data for each evaluation dimension (model, framework, computing platform) and standardization processing, ensuring data accuracy and comparability in subsequent evaluation processes. This provides a solid data foundation for system optimization, hardware resource selection, and performance tuning, helping to improve the overall efficiency of the target model in the target application scenario.

[0073] In an optional implementation, the first score of the target model is calculated using the weight set and the standard values ​​corresponding to the model parameters, framework parameters and computational parameters, respectively. This includes: determining the target weights in the weight set that correspond to at least one model parameter, framework parameter and computational parameter respectively; and performing a weighted summation based on the target weights corresponding to at least one model parameter, framework parameter and computational parameter and the standard values ​​corresponding to at least one model parameter, framework parameter and computational parameter respectively to obtain the first score.

[0074] It should be noted that the first score refers to the comprehensive efficiency score obtained by weighted summation based on model parameters, framework parameters, and computational parameters, combined with the weight set. It reflects the overall performance of the target model in a given application scenario.

[0075] In an optional implementation, based on the target application scenario, a weight set is retrieved from a scenario weight mapping table, and a target weight is assigned to each model parameter, framework parameter, and computation parameter. The normalized index value is multiplied by its corresponding target weight, and then all products are summed to obtain a weighted average, which is the first score of the target model.

[0076] In an optional implementation, determining the standard value corresponding to each of the model parameters, framework parameters, and calculation parameters based on their respective benchmark reference values ​​includes: determining at least one benchmark reference value corresponding to each of the model parameters, framework parameters, and calculation parameters; and obtaining the standard value corresponding to at least one model parameter, framework parameter, and calculation parameter based on the at least one model parameter, framework parameter, and calculation parameter and their respective benchmark reference values.

[0077] It should be noted that benchmark values ​​can be used as reference values ​​for comparison when evaluating system efficiency. These values ​​are usually set based on industry averages, best practices, or specific hardware specifications, serving as a starting point for quantitative evaluation. Standard values ​​are generated from benchmark values ​​after specific processing or adjustment, and are used to normalize the results in weighted summation calculations, ensuring that all performance indicators are compared on a uniform scale.

[0078] In an optional implementation, benchmark values ​​for each performance metric are established based on industry data, best practices, and hardware specifications. These benchmark values ​​reflect the ideal or average performance level of each level of performance metric in the target application scenario. Next, standard values ​​are determined based on the relationship between the actual values ​​of each metric and the benchmark values. Then, normalization formulas for positive metrics (higher values ​​are better) and negative metrics (lower values ​​are better) are applied to convert the actual performance metric values ​​into dimensionless scores relative to the standard values. Finally, a weighted sum is performed using a defined set of weights and the normalized performance metric values ​​to obtain the first score of the target model in the specific application scenario.

[0079] In an optional implementation, for model parameters (such as OTPS), framework parameters (such as QPS), and computing platform parameters (such as peak computing power), consult industry reports, technical white papers, or experimental data to find performance index values ​​that perform well or on average in specific application scenarios as benchmark reference values.

[0080] The above-described implementation methods of this application ensure the scientific and systematic nature of the evaluation process. They not only comprehensively consider the model's performance at different levels but also dynamically adjust weights according to the characteristics of specific application scenarios, obtaining evaluation results that best meet actual needs. This provides a solid data foundation for subsequent system optimization and resource allocation.

[0081] In an optional implementation, a standard value corresponding to at least one model parameter, framework parameter, and calculation parameter is obtained based on at least one model parameter, framework parameter, and calculation parameter and their respective corresponding benchmark reference values. This includes: when the model parameter, framework parameter, and calculation parameter are first parameters, determining the ratio of the model parameter, framework parameter, and calculation parameter to the corresponding benchmark reference value as the standard value, wherein the first parameter is a positive indicator; when the model parameter, framework parameter, and calculation parameter are second parameters, determining the ratio of the corresponding benchmark reference value to the model parameter, framework parameter, and calculation parameter as the standard value, wherein the second parameter is a negative indicator.

[0082] It should be noted that the standard value is a dimensionless value calculated according to certain rules based on the comparison between the model parameters, framework parameters, and calculation parameters and the benchmark reference value. It is used to quantitatively evaluate the relative performance of these parameters in the target application scenario.

[0083] A higher value for a positive metric indicates better system performance, such as the model's OTPS (total tokens output per second), the framework's QPS (queries per second), and the hardware's peak computing power. When calculating benchmark values, the ratio of the actual value of a positive metric to a benchmark reference value is used for normalization.

[0084] The smaller the value of a negative metric, the better the system performance. Examples include model latency, framework TP99 (99th percentile latency), and hardware memory usage. When calculating benchmark values, the ratio of the baseline reference value to the actual value of the negative metric is used for normalization.

[0085] In an optional implementation, benchmark reference values ​​are set for each model parameter, framework parameter, and computing platform parameter based on reasonable performance index values ​​determined through market research, industry standards, or experiments. For positive indicators (such as OTPS, QPS, peak computing power), the standard value is calculated by dividing the actual value by the benchmark reference value; for negative indicators (such as latency, memory usage, TP99 latency), the standard value is calculated by dividing the benchmark reference value by the actual value. Ensure that all indicators are converted to dimensionless values ​​to facilitate subsequent weighted summation calculations.

[0086] In the embodiments described above, the calculation of standard values ​​is a crucial preprocessing step. By comparing actual performance indicators with preset industry benchmarks and eliminating the influence of dimensions through normalization, a unified scale is provided for the comprehensive evaluation of system efficiency. This step makes the evaluation results more comparable and objective, providing strong data support for system optimization and resource allocation.

[0087] In an optional implementation, a standard value corresponding to at least one model parameter, framework parameter, and calculation parameter is obtained based on at least one model parameter, framework parameter, and calculation parameter and their respective corresponding benchmark reference values. This includes: when the model parameter, framework parameter, and calculation parameter are qualitative parameters, determining at least one benchmark reference value corresponding to the model parameter, framework parameter, and calculation parameter; and determining the standard value based on the model parameter, framework parameter, and calculation parameter at at least one benchmark reference value.

[0088] It should be noted that qualitative metrics, such as Quantization support, cannot be simply represented by numbers. Instead, they assess the extent to which a framework supports specific techniques (such as quantization).

[0089] In optional implementations, a grading method is used for metrics like Quantization support that cannot be directly quantified into numbers. For example, a score of 0 is assigned to no quantization support, indicating that the framework lacks the ability to quantize models; a score of 0.7 (or 70%) is assigned to INT8 quantization support, meaning it can only handle 8-bit quantization models to a certain extent; and a score of 1.0 is assigned to INT4 quantization support, indicating that the software framework can efficiently support the inference and training of 4-bit quantization models. Grading and assigning scores is the process of converting qualitative capabilities into quantitative scores, facilitating unified processing with other metrics. For the qualitative components of model parameters, framework parameters, or computational parameters, a benchmark reference value needs to be set first. This value is usually derived from industry standards, best practices, or expected values ​​in specific application scenarios. For example, the benchmark value for quantization support can be set as the framework perfectly supporting INT4 quantization models, allowing the quantization support of other frameworks or models to be evaluated and compared based on this standard. After determining the benchmark reference value, the standard value relative to the benchmark value is calculated based on the current state and grading assignment of the model parameters, framework parameters, or computational parameters, thus converting the qualitative metrics into quantitative scores related to actual performance.

[0090] In an optional implementation, a metric ranking system is defined to clarify the specific meaning of different support levels, such as "no quantization support," "basic quantization support (e.g., INT8)," and "advanced quantization support (e.g., INT4)," with a numerical score assigned to each level. The actual quantization support of the software framework is assessed through its official documentation, experimental testing, or community feedback, including whether it supports INT8 or INT4 quantization, the degree of support, and whether it provides automatic quantization tools and optimizes post-quantization performance. Based on the collected information, the software framework is scored in a pre-defined quantization support rating table. For example, if the framework supports INT8 but not INT4 quantization, the quantization support score is 0.7; if the framework does not support any quantization, the score is 0. A baseline value for Quantization support is set based on industry standards or expectations for specific scenarios. For example, if most advanced frameworks support INT8 quantization well, INT8 quantization support (score 0.7) can be set as the baseline value. For qualitative metrics like Quantization support, a rating is used instead of the actual numerical value, and then the previously mentioned normalization formula is applied. If the current framework's quantization support score is 1.0 (meaning it supports INT4 quantization), and the baseline value is set to 0.7, then its standard value is 0.7 / 1.0=0.7. This actually transforms the quantization support into an inverted value, indicating how many times more efficient the model is in quantization than the baseline.

[0091] Through the above-described embodiments of this application, the performance of the software framework on the quantitative support index can be quantified, and it can also be combined with other performance indicators to conduct system-level efficiency evaluation, thereby guiding the optimization selection of software frameworks and computing platforms to meet the needs of specific models and applications.

[0092] In an optional implementation, determining the benchmark reference value corresponding to at least one model parameter, framework parameter, and computation parameter includes: obtaining a reference data set corresponding to at least one reference model; and determining the median of the reference data set as the benchmark reference value corresponding to at least one model parameter and framework parameter.

[0093] In an optional implementation, a representative set of data is collected, derived from the actual operating results of similar models on mainstream inference systems, including key performance indicators such as model generation efficiency (OTPS), response latency, and memory usage. From the collected reference dataset, the median (or P50 value, i.e., the 50th percentile value) of each indicator is selected as the benchmark reference value for model parameters, framework parameters, and computational parameters. This is because the median can better reflect the central tendency of the data, avoid the influence of extreme values, and make the established baseline more reliable and representative.

[0094] Collect extensive performance data of similar models on mainstream inference systems. For example, collect OTPS, latency, and memory usage data of different versions of large language models on GPU clusters. Clean the collected raw data, removing outliers and irrelevant data to ensure data quality. Then, categorize the data by metrics, creating a separate data subset for each metric. For each performance metric's data subset, calculate the median (P50 value), which serves as the industry best practice baseline for that metric. If the data distribution is uneven, consider using the weighted median or other statistics. Store the median baseline values ​​for each metric in a database for subsequent performance evaluation and comparison. The database should include detailed information such as model name, framework version, computing platform configuration, and sampling environment to ensure the traceability and reference value of the baseline data. As industry technologies evolve, model and framework performance will improve accordingly. Therefore, it is necessary to regularly update the benchmark reference values ​​to reflect the latest technological advancements and performance standards.

[0095] The embodiments described above in this application enable the establishment of a reliable performance baseline for evaluating and optimizing the efficiency of the target model. This method not only provides an objective evaluation criterion but also helps identify the gap between the target model and the industry average, guiding subsequent system optimization and hardware selection.

[0096] In an optional implementation, determining a reference value corresponding to at least one model parameter, framework parameter, and computation parameter includes: determining the hardware specifications corresponding to the computation nodes of the target model; and determining the hardware specifications as the reference value corresponding to at least one computation parameter.

[0097] It's important to note that resource constraint baselines are hard boundaries set for various metrics of the model, software framework, and computing platform based on hardware specifications or performance requirements of specific business scenarios. These baseline values ​​reflect the expected minimum or maximum performance standards of the system under specific conditions. Hardware specifications refer to the technical parameters of the hardware devices in the computing nodes, such as processor speed, memory capacity, and graphics card performance, and are the foundation for determining the performance of the computing platform.

[0098] For latency-related metrics, such as response latency and tail latency (TP99), a baseline is set based on the maximum acceptable latency time for users (e.g., 300ms). This baseline reflects the minimum requirements for system response speed in high-concurrency, real-time interactive business scenarios to ensure user experience.

[0099] In optional implementations, a comprehensive survey of the target computing node's hardware is required to clarify its technical parameters, such as GPU model, CPU core count, memory size, and network bandwidth. This can be achieved by consulting the hardware's technical manual, using hardware testing tools (such as lshw, GPU-Z), or obtaining information directly from the equipment vendor. Based on the business scenario, define the performance standards that the model, software framework, and computing platform need to achieve. For example, for a real-time dialogue system, the maximum acceptable latency can be set to 300ms. Based on the hardware survey results and business performance requirements, set resource constraint baselines for each parameter. For hardware specifications, such as peak performance, the baseline value can be directly set to the theoretical maximum computing power of the GPU; for the tail latency (TP99) in the framework parameters, it should be set to the maximum latency requirement under the business scenario, such as 300ms. In the final system design, integrate the hardware, software framework, and model to ensure that all parameters meet the set resource constraint baselines. Through actual operation testing, collect performance data of the system under typical workloads to verify whether the baseline requirements are met. If the test results fail to meet the resource constraint baseline, it is necessary to adjust the model parameters, optimize the framework configuration, or upgrade the hardware specifications until the system performance reaches or exceeds the baseline standard.

[0100] Through the above-described embodiments of this application, resource constraint baselines can be set based on hardware specifications and business requirements to ensure that the system can achieve the expected performance standards during design and deployment, thereby providing the best user experience and service quality in actual operation.

[0101] In an optional implementation, the model parameters corresponding to the target model, the framework parameters of the target model's runtime environment, and the computation parameters of the target model's computation nodes are obtained, and the target application scenario of the target model is determined, including: determining the target threshold range that the model parameters, framework parameters, and computation parameters conform to; and determining the reference application scenario corresponding to the target threshold range as the target application scenario.

[0102] It's important to note that the target threshold range represents a reasonable range for the performance and resource consumption of model parameters, framework parameters, and computational parameters. These thresholds reflect the expected level or acceptable lower limit of parameter performance in a specific application scenario. Reference application scenarios can be a predefined set of typical application scenarios, each corresponding to a set of expected performance thresholds for models, frameworks, and computational parameters. When the performance of a set of parameters matches the threshold range of a reference application scenario, that scenario is considered the target application scenario. The performance data of the target model at each level is compared with the parameter threshold ranges of the reference application scenarios. If most of the key parameters of the target model fall within the threshold range of a reference application scenario, then that scenario is the potential optimal application scenario for the target model.

[0103] Example 2:

[0104] Assume the target model is a large language model based on the Transformer architecture, used in a real-time dialogue system. For the software framework, we use vLLM, and for the computing platform, we configure 4×NVIDIA A100 GPUs. At the model level, the tests yielded OTPS of 1581.302 tokens / s, latency of 161.892ms, and VRAM usage of 9GB. At the framework level, the QPS was 6.177 queries / s, TP99 was 163.198ms, and quantization support was 1. At the computing platform level, the peak computing power was 65 TFLOPS, the memory bandwidth was 1625.356 GB / s, and the energy efficiency ratio was 0.025 TFLOPS / W.

[0105] Threshold settings: Real-time dialogue systems require model latency <200ms, OTPS >1500 tokens / s, and GPU memory usage <10GB; QPS >8 queries / s, TP99 <200ms; peak computing power of at least 60 TFLOPS, memory bandwidth >1600GB / s, and energy efficiency ratio >0.02 TFLOPS / W. Scenario determination: Comparing the measured data with the threshold range, it was found that the target model's performance at the model level, framework level, and computing platform level all meet the parameter requirements of a real-time dialogue system. Therefore, a real-time dialogue system can be determined as the optimal application scenario for the target model.

[0106] The above-described implementation methods of this application verify the applicability of the target model in real-time dialogue scenarios and provide direction for subsequent performance optimization and resource allocation. For example, since the QPS is slightly below the threshold, we may consider further optimizing the dynamic batch processing strategy at the software framework layer to improve the concurrent request processing capability.

[0107] In an optional implementation, determining the target threshold range that the model parameters, framework parameters, and computation parameters meet includes: determining a first indicator of the target model based on the model parameters, wherein the first indicator is used to indicate the computational load of a single request of the target model; determining a second indicator of the target model's operating environment based on the framework parameters, wherein the second indicator is used to indicate the number of requests completed by the operating environment per unit time; determining a third indicator of the computing node based on the computation parameters, wherein the third indicator is used to indicate the resource constraints of the computing node; and determining the target threshold range based on the first, second, and third indicators.

[0108] It's important to note that the first metric is related to model parameters, indicating the computational cost of a single request for the target model. This typically reflects model complexity and processing speed, such as OTPS (Output Tokens Per Second)—the number of tokens generated per second by the model, which is particularly important for text generation models. The second metric, based on framework parameters, indicates the number of requests completed by the runtime environment per unit of time. This reflects the parallel processing capability and resource scheduling efficiency of the software framework, such as QPS (Queries Per Second)—the number of requests processed per second by the system, a key parameter for evaluating system responsiveness in high-concurrency scenarios. The third metric, through analysis of computational parameters, indicates the resource constraints of computing nodes. This includes hardware computing power, memory, and energy efficiency, such as peak performance and memory bandwidth, which directly determine the physical limitations and costs of model operation.

[0109] In optional implementations, industry performance data and best practices are collected to determine industry-recognized standard performance metrics such as OTPS of the model, QPS of the software framework, and peak computing power of computing nodes as initial settings. Performance requirements are then refined for specific application scenarios. For example, in real-time dialogue systems, high response speed and low latency are primary goals, while offline generation tasks may prioritize processing efficiency and cost control. Available computing resources, including GPU and CPU computing power, memory capacity, and network I / O speed, are evaluated to determine the resource constraints of the computing nodes—a third metric.

[0110] In an optional implementation, the reference application scenario corresponding to the target threshold range is determined as the target application scenario, including: when the target threshold range is a first range, a first reference application scenario is determined as the target application scenario, wherein the first reference application scenario is a high-concurrency short text scenario; when the target threshold range is a second range, a second reference application scenario is determined as the target application scenario, wherein the second reference application scenario is a high-concurrency long text scenario; when the target threshold range is a third range, a third reference application scenario is determined as the target application scenario, wherein the third reference application scenario is an edge low-power scenario.

[0111] It should be noted that the first range specifically refers to the target threshold range for performance metrics of models, frameworks, and computing platforms in high-concurrency short text processing scenarios. For example, metrics such as OTPS of the model and QPS of the framework should reach high levels, while controlling latency and memory usage. The second range is the target threshold range defined for high-concurrency long text processing scenarios. Compared with short text scenarios, this range may place more emphasis on the model's long sequence processing capabilities and the memory bandwidth of the computing platform, while also having high requirements for latency and concurrent processing capabilities. The third range applies to the target threshold range for low-power edge scenarios. Here, the evaluation of system efficiency focuses more on performance per unit power consumption, quantization support, and memory efficiency to ensure that a reasonable service level can still be maintained on resource-constrained edge devices.

[0112] In the optional implementation, the application scenarios of the model are inexhaustible. Therefore, three meta-scenarios can be set. Users can specify a meta-scenarios as the target application scenario, or determine a meta-scenarios as the target application scenario based on the actual data from the model's operation. After determining the target application scenario, a preset set of weights can be determined. Users can fine-tune and refine the weight set according to their needs to satisfy the determination of weights in all scenarios.

[0113] In an optional implementation, before determining the target threshold range that the model parameters, framework parameters, and computational parameters meet, the method includes: obtaining a reference data set corresponding to at least one reference model, wherein the reference data set is used to indicate the reference model parameters, reference framework parameters, and reference computational parameters; performing cluster analysis on the reference data set to obtain at least one cluster point; and determining the reference threshold range based on the cluster point.

[0114] It should be noted that the reference model can be an AI model that is known to perform well or is representative in the target domain, used as a benchmark to evaluate the performance of the target model. The reference dataset contains performance metrics data of the reference model, framework, and computing platform in different application scenarios.

[0115] In optional implementations, performance metrics data for multiple reference models are acquired, including OTPS, latency, and memory usage. QPS, TP99, and quantization support of different software frameworks when processing each reference model are recorded. Peak computing power, memory bandwidth, and energy efficiency data for various computing platforms (e.g., different GPU and CPU configurations) when running the aforementioned reference models are compiled. The collected reference data set is cleaned and normalized to ensure data consistency and comparability. The preprocessed data is clustered. Based on the clustering analysis results, the clusters with the best performance metrics are identified, representing the typical efficiency level of the model, framework, and computing platform working collaboratively in this application scenario. Based on the selected clusters, reference thresholds are set for each performance metric, such as a model OTPS of at least X tokens / s and a framework QPS not lower than Y queries / s. The reference threshold range is ensured to reflect not only performance requirements but also constraints of actual resources, such as the peak computing power and memory bandwidth of the computing platform.

[0116] In an optional implementation, determining the reference threshold range based on cluster points includes: determining a first index range, a second index range, and a third index range corresponding to the cluster points; and using the cluster points as the midpoint, determining the first index range, the second index range, and the third index range as the reference threshold range.

[0117] In an optional implementation, the performance metric value of each cluster point is extracted from the clustering analysis results. Using this metric value as the center, upper and lower thresholds are set based on business needs and scenario characteristics. For example, if the latency metric cluster point value at the model level is 150ms, considering the latency sensitivity of real-time applications, a first metric range of 120ms to 180ms can be set. All first, second, and third metric ranges are then integrated to form a comprehensive set of reference threshold ranges for subsequent target application scenario matching.

[0118] The embodiments described above in this application allow for setting reasonable reference threshold ranges for different scenarios, serving as a benchmark for evaluating the performance of the target model. In practical applications, if the performance indicators of the target model fall within a certain reference threshold range, the model can be considered suitable for the corresponding application scenario, providing scientific guidance for model selection, system configuration, and performance optimization.

[0119] In an optional implementation, determining the weight set based on the target application scenario includes: obtaining a scenario weight correspondence table, wherein the scenario weight correspondence table is used to indicate the correspondence between at least one reference application scenario and a reference weight set; and determining the weight set by searching the scenario weight correspondence table based on the target application scenario.

[0120] In an optional implementation, the scenario weight mapping table is created during the initial design phase of the evaluation method by pre-setting a set of weight values ​​for each scenario through research and analysis of various typical application scenarios. These weights aim to reflect the relative importance of each performance indicator in a specific scenario. Once the target application scenario is determined, the evaluator should consult the scenario weight mapping table to find the set of weights that match the scenario for use in the next step of score calculation.

[0121] In an optional implementation, before obtaining the scene weight correspondence table, one of the following is included: obtaining a reference weight set corresponding to the reference data set based on at least one reference model through a weight model; obtaining the weight set generation rules, and determining the reference weight set corresponding to the reference application scenario according to the weight set generation rules.

[0122] Using a weighted model: Performance metrics data for multiple reference models across various software frameworks and computing platforms are collected and organized to form a reference dataset. This dataset is used to train a weighted model, which can be a linear regression, neural network, or other machine learning model. The goal is to learn the relative importance of model parameters, framework parameters, and computing platform parameters in different application scenarios. For a new target application scenario, its features and requirements are input into the weighted model, which outputs a set of weight values, i.e., the reference weight set. By applying the weight set to a small number of test samples, its ability to reasonably reflect the performance requirements of the application scenario is verified, and fine-tuning is performed as necessary.

[0123] Application Weight Set Generation Rules: Based on industry experience, technical literature, and expert opinions, a set of weight set generation rules is established for different application scenarios. These rules should clearly define how weights are allocated according to scenario characteristics. Based on the characteristics of the target application scenario, such as high concurrency, real-time interaction, or low power consumption, the corresponding weight set is found or calculated. In practical applications, the weight set is adjusted based on preliminary evaluation results to more accurately reflect the performance requirements of the target application scenario. As technology advances and application scenario needs change, the weight set generation rules are updated regularly to ensure their continued effectiveness.

[0124] In an optional implementation, for a real-time dialogue system, the weight set generation rules are defined as follows: the weight of model latency should be higher than that of model generation efficiency (OTPS); the weights of framework high-concurrency processing capability (QPS) and stability (TP99) should be higher than those of other indicators; the weight of the computing platform's energy efficiency ratio should not be lower than 0.05 to ensure economical resource utilization. Based on these rules, a reference weight set is generated for the real-time dialogue system, for example: the weight of model parameters (OTPS) is set to 0.15, the weight of model parameters (Latency) is set to 0.3; the weight of framework parameters (QPS) is set to 0.2, the weight of framework parameters (TP99) is set to 0.25; the weight of computing platform parameters (peak computing power) is set to 0.03, the weight of computing platform parameters (memory bandwidth) is set to 0.02, and the weight of computing platform parameters (energy efficiency ratio) is set to 0.05. The reference weight set is initially applied to evaluate system efficiency. If a significant deviation is found between the evaluation results and the actual system performance, such as a low score for the latency indicator, it may be because the set weights are too high, causing the scores of other indicators to be compressed. At this point, the weight of Latency can be appropriately reduced, while the weights of other metrics (such as OTPS and QPS) can be increased to achieve a more balanced evaluation effect. Regularly review the applicability of the weight set generation rules and adjust them based on the latest market trends and technological advancements to ensure that the rules can continuously guide the generation of weight sets to adapt to ever-changing application scenario requirements.

[0125] Through the above-described implementation methods of this application, whether using a weight model or directly applying weight set generation rules, a reasonable reference weight set can be generated for real-time dialogue system application scenarios. This allows for accurate reflection of the key performance requirements of the application scenarios in subsequent system efficiency evaluations, guiding system optimization and the rational allocation of hardware resources.

[0126] In an optional implementation, the first score of the target model is calculated using the standard values ​​corresponding to the weight set and the model parameters, frame parameters and calculation parameters, respectively. This includes: determining the sub-score set calculated between the standard values ​​corresponding to the weight set and the model parameters, frame parameters and calculation parameters, respectively; summing the sub-score sets to obtain the first score; and displaying the sub-score set and the first score on the display interface.

[0127] It should be noted that the sub-score set is a set of scores for each dimension obtained by weighted summation during the calculation of the first score, based on the relationship between the weight set, model parameters, framework parameters, and computational parameters. The sub-score set includes the degree of contribution of each performance indicator to the overall score.

[0128] Example 3:

[0129] Assuming that model parameters, frame parameters, and computational parameters have been standardized in a real-time dialogue system scenario, the following standardized values ​​are available: OOPS (Output of Power): 1.0667; Latency: 0.9 (corrected value); Memory: 0.9; QPS (Queries Per Second): 0.9375; TP99 (Frame Tail Latency): 0.95; Peak Performance: 0.9286; Memory Bandwidth: 1.0294; Energy Efficiency: 1.1. The following weight set has been determined: Model Generation Capability: 0.15, Model Latency: 0.25, Memory Usage: 0.1, Frame Processing Capability: 0.15, Frame Tail Latency: 0.2, Hardware Performance: 0.05, Memory Bandwidth: 0.05, Energy Efficiency: 0.05. Calculate the sub-scores for each standardized value: Model generation capability sub-score: 0.16, Model latency sub-score: 0.225, GPU memory usage sub-score: 0.09, Framework processing capability sub-score: 0.140625, Framework tail latency sub-score: 0.19, Hardware computing power sub-score: 0.04643, Memory bandwidth sub-score: 0.05147, Energy efficiency ratio sub-score: 0.055. Sum all sub-scores to obtain the first score.

[0130] 0.16+0.225+0.09+0.140625+0.19+0.04643+0.05147+0.055=0.958525.

[0131] Finally, the display interface presents the set of sub-scores and the first score. For example, a bar chart can be used to show the sub-scores for each performance metric, and the total value of the first score can be displayed below the chart: "First Score: 0.9585". Additionally, information on the percentage of the sub-scores relative to the first score can be provided to help users understand which performance metrics have a greater impact on the overall score, thus providing direction for optimization efforts.

[0132] Figure 5 This is a schematic diagram of an optional display interface according to an embodiment of this application; as shown... Figure 5 As shown, after determining the standard values ​​and scores for each data point, these values ​​or scores can be displayed on the interface for easy viewing. The system comprises nine dimensions: model-level metrics such as OTPS (Output Time Per Second), Latency (Response Latency), and Memory (GPU Memory Usage); software framework-level metrics such as QPS (Throughput), TP99 (Tail Latency), and Quantization Support; and computing platform-level metrics such as Peak Computing Power, Memory Bandwidth, and Energy Efficiency. Different colors are used to mark the metrics at different levels for easy comparison and viewing by users.

[0133] Through the above-described implementation methods of this application, not only is the comprehensive efficiency score of the target model in the real-time dialogue system scenario obtained, but the contribution of different performance indicators to the overall score is also clarified. This provides strong data support for the subsequent optimization and upgrading of the system, and also facilitates decision-makers to make quick judgments based on the displayed score information.

[0134] Example 4:

[0135] Overall, when facing large-scale model inference systems, the following indicator system is established: Model level: Select indicators such as OTPS, Latency, and Memory. Software framework level: Establish indicators such as QPS, TP99, and Quantization Support. Computing platform level: Focus on indicators such as Peak Performance, Memory Bandwidth, and Energy Efficiency. Baseline setting: Industry best practice baseline: Collect industry data to determine the median or P50 value of each indicator. Resource constraint baseline: Set performance boundaries based on hardware specifications and business requirements. Indicator normalization: Normalize positive indicators (such as OTPS and QPS) by dividing the actual value by the baseline value. Normalize negative indicators (such as Latency and Memory) by dividing the baseline value by the actual value. Normalize qualitative indicators (such as Quantization Support) after assigning levels. Weight configuration: Determine indicator weights for different application scenarios to ensure matching with the target scenario.

[0136] Under typical loads, a data acquisition module records performance data for the model, framework, and platform. Normalization formulas are applied to process the collected performance data, yielding dimensionless standardized scores. A dynamic weighting engine is used to configure weights based on the user-input application scenario. Sub-scores are calculated by multiplying each performance metric by its weight. These sub-scores are then weighted and summed to obtain a comprehensive score. Scores at each level are analyzed to identify efficiency bottlenecks. The comprehensive score is analyzed to understand the overall system efficiency and the relative performance of each level. Based on the scores at each level, performance bottlenecks and optimization suggestions are identified and proposed. The comprehensive score calculation and output module visualizes the scoring results for easy and intuitive understanding. Weight configurations are flexibly adjusted according to different application scenarios to ensure the evaluation is targeted.

[0137] When a user explicitly states their intention to deploy a model inference system to achieve a specific goal, and clearly defines the specific application scenario (e.g., financial trading), the application scenario can be defined as a financial trading scenario, and a corresponding weight set can be determined based on this scenario. A financial trading scenario may not be within the preset application scenarios. In this case, the user can either set their own weight set or determine a weight set based on the application scenario. Setting a weight set independently is straightforward and will not be elaborated upon here. For determining a weight set based on the application scenario, a meta-scenario can be identified based on the user's needs or the actual operational data of the deployed model, and a preset weight set can be determined based on this meta-scenario. The preset weight set can then be fine-tuned according to the user's actual needs. For example, if lower latency is required to ensure smooth transactions, the weights corresponding to the latency parameter can be fine-tuned, and other parameters can be updated to obtain the weight set corresponding to the current specific application scenario. If the user's specific application scenario is within the preset application scenarios, then the corresponding weight set can be used directly.

[0138] After determining the weight set, users can evaluate multiple model inference systems they wish to assess, obtaining a first score for each system. Because the parameters of the model inference systems are standardized and the weight sets are identical, horizontal comparisons are possible. Finally, users can refer to the scores of each model inference system to select the desired model inference system for deployment. Furthermore, the scores can be displayed using a radar chart for easy viewing and comparison.

[0139] Normalized radar charts are a tool for visually displaying multidimensional data. They present key metrics across three levels—model, software framework, and computing platform—in radar chart format, with each metric representing a dimension, thus helping to analyze bottlenecks in system efficiency. Below is a solution for using normalized radar charts to diagnose efficiency issues and propose targeted improvement suggestions: Compare the target system's normalized radar chart with a reference radar chart of industry best practices to observe which dimensions' metric scores are below average or industry standards. Examine the shortest "tentacles" on the radar chart; these are usually the most inefficient or in need of improvement. Analyze the relationships between different dimensions; for example, high model memory usage may cause the computing platform's memory bandwidth to become a bottleneck. Examine whether the software framework's QPS matches the model's OTPS; low QPS may indicate a framework scheduling problem rather than a model or hardware limitation. Check whether the radar chart reflects the degree to which specific application scenario requirements are met; for example, whether latency metrics are low enough in real-time services. Compare the matching degree between key metrics and weight configurations to ensure that the evaluation results are consistent with actual needs.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0141] Embodiments of this application also provide an efficiency evaluation device for a model inference system. Figure 6 This is a structural block diagram of an optional efficiency evaluation device for a model inference system according to an embodiment of this application, such as... Figure 6 As shown, the device includes:

[0142] The scenario determination module 602 is used to obtain the model parameters corresponding to the target model, the framework parameters of the target model's operating environment, and the computation parameters of the target model's computation nodes, and to determine the target application scenario of the target model, wherein the target application scenario is used to indicate the task performed by the target model; the weight determination module 604 is used to determine the weight set according to the target application scenario, wherein the weight set is used to indicate the weight values ​​corresponding to the model parameters, framework parameters, and computation parameters respectively; the normalization module 606 is used to determine the standard values ​​corresponding to the model parameters, framework parameters, and computation parameters respectively based on the benchmark reference values ​​corresponding to the model parameters, framework parameters, and computation parameters respectively; the model evaluation module 608 is used to calculate the first score of the target model using the weight set and the standard values ​​corresponding to the model parameters, framework parameters, and computation parameters respectively, wherein the first score is used to indicate the efficiency of the target model.

[0143] Optionally, the scenario determination module 602 is used to determine the target threshold range that the model parameters, framework parameters, and calculation parameters meet; and to determine the reference application scenario corresponding to the target threshold range as the target application scenario.

[0144] Optionally, the scenario determination module 602 is used to determine a first indicator of the target model based on model parameters, wherein the first indicator is used to indicate the computational load of a single request of the target model; determine a second indicator of the target model's operating environment based on framework parameters, wherein the second indicator is used to indicate the number of completed requests in the operating environment per unit time; determine a third indicator of the computing node based on computational parameters, wherein the third indicator is used to indicate the resource constraints of the computing node; and determine a target threshold range based on the first, second, and third indicators.

[0145] Optionally, the scenario determination module 602 is used to determine a first reference application scenario as the target application scenario when the target threshold range is a first range, wherein the first reference application scenario is a high-concurrency short text scenario; to determine a second reference application scenario as the target application scenario when the target threshold range is a second range, wherein the second reference application scenario is a high-concurrency long text scenario; and to determine a third reference application scenario as the target application scenario when the target threshold range is a third range, wherein the third reference application scenario is an edge low-power scenario.

[0146] Optionally, the scenario determination module 602 is used to obtain a reference data set corresponding to at least one reference model, wherein the reference data set is used to indicate the reference model parameters, reference frame parameters and reference calculation parameters; perform cluster analysis on the reference data set to obtain at least one cluster point; and determine a reference threshold range based on the cluster point.

[0147] Optionally, the scenario determination module 602 is used to determine the first index range, the second index range, and the third index range corresponding to the cluster point; and to determine the first index range, the second index range, and the third index range as reference threshold ranges with the cluster point as the midpoint.

[0148] Optionally, the scenario determination module 602 is used to obtain a scenario weight correspondence table, wherein the scenario weight correspondence table is used to indicate the correspondence between at least one reference application scenario and a set of reference weights; and to determine the weight set by searching the scenario weight correspondence table according to the target application scenario.

[0149] Optionally, the weight determination module 604 is used to obtain a reference weight set corresponding to the reference data set based on the reference data set of at least one reference model through a weight model; obtain the weight set generation rule; and determine the reference weight set corresponding to the reference application scenario according to the weight set generation rule.

[0150] Optionally, the weight determination module 604 is used to determine the target weights in the weight set corresponding to at least one model parameter, frame parameter, and calculation parameter respectively; and to obtain a first score by weighted summation of the target weights corresponding to at least one model parameter, frame parameter, and calculation parameter and the standard values ​​corresponding to at least one model parameter, frame parameter, and calculation parameter respectively.

[0151] Optionally, the normalization module 606 is used to determine the reference values ​​corresponding to at least one model parameter, framework parameter, and calculation parameter; and to obtain the standard values ​​corresponding to at least one model parameter, framework parameter, and calculation parameter based on the reference values ​​corresponding to the at least one model parameter, framework parameter, and calculation parameter.

[0152] Optionally, the normalization module 606 is used to determine the ratio of the model parameters, framework parameters, and calculation parameters to the corresponding benchmark reference value as a standard value when the model parameters, framework parameters, and calculation parameters are the first parameters, wherein the first parameter is a positive indicator; and to determine the ratio of the corresponding benchmark reference value to the model parameters, framework parameters, and calculation parameters as a standard value when the model parameters, framework parameters, and calculation parameters are the second parameters, wherein the second parameter is a negative indicator.

[0153] Optionally, the normalization module 606 is used to determine at least one benchmark reference value corresponding to the model parameters, framework parameters, and calculation parameters when the model parameters, framework parameters, and calculation parameters are qualitative parameters; and to determine a standard value based on the model parameters, framework parameters, and calculation parameters at at least one benchmark reference value.

[0154] Optionally, the normalization module 606 is used to obtain a reference data set corresponding to at least one reference model; and to determine the median of the reference data set as the benchmark reference value corresponding to at least one model parameter and frame parameter.

[0155] Optionally, the normalization module 606 is used to determine the hardware specifications corresponding to the computing nodes of the target model; and to determine the hardware specifications as the reference values ​​corresponding to at least one computing parameter.

[0156] Optionally, the normalization module 606 is used to determine the generation efficiency, response latency, and video memory usage corresponding to the target model as model parameters; to determine the throughput, tail latency, and quantization support corresponding to the operating environment as framework parameters; and to determine the focused peak computing power, memory bandwidth, and energy efficiency ratio corresponding to the computing node as computing parameters.

[0157] Optionally, the model evaluation module 608 is used to determine the sub-score set calculated between the standard values ​​corresponding to the weight set and the model parameters, framework parameters and calculation parameters, respectively; to sum the sub-score sets to obtain the first score; and to display the sub-score set and the first score on the display interface.

[0158] For a description of the features in the embodiment corresponding to the efficiency evaluation device of the model reasoning system, please refer to the relevant description of the embodiment corresponding to the efficiency evaluation method of the model reasoning system, which will not be repeated here.

[0159] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the efficiency evaluation method for model inference systems.

[0160] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described embodiments of the efficiency evaluation method for model inference systems at runtime.

[0161] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0162] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the efficiency evaluation method for model inference systems.

[0163] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the efficiency evaluation method for model inference systems.

[0164] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0165] The above provides a detailed description of the efficiency evaluation method, apparatus, storage medium, and electronic device for a model inference system provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. An efficiency evaluation method of a model inference system, comprising: obtaining model parameters corresponding to a target model, framework parameters of a running environment of the target model, and computing parameters of a computing node of the target model, and determining a target application scenario of the target model, wherein the target application scenario is used to indicate a task performed by the target model; determining a weight set according to the target application scenario, wherein the weight set is used to indicate weight values corresponding to the model parameters, the framework parameters, and the computing parameters respectively; determining standard values corresponding to the model parameters, the framework parameters, and the computing parameters respectively based on benchmark reference values corresponding to the model parameters, the framework parameters, and the computing parameters respectively; and calculating a first score of the target model by using the weight set and the standard values corresponding to the model parameters, the framework parameters, and the computing parameters respectively, wherein the first score is used to indicate an efficiency corresponding to a target model inference system.

2. The method of claim 1, wherein the obtaining the model parameters corresponding to the target model, the framework parameters of the running environment of the target model, and the computing parameters of the computing node of the target model, and determining the target application scenario of the target model comprises: determining a target threshold range to which the model parameters, the framework parameters, and the computing parameters conform; and determining a reference application scenario corresponding to the target threshold range as the target application scenario.

3. The method of claim 2, wherein the determining the target threshold range to which the model parameters, the framework parameters, and the computing parameters conform comprises: determining a first index of the target model based on the model parameters, wherein the first index is used to indicate a computing amount of a single request of the target model; determining a second index of a running environment of the target model based on the framework parameters, wherein the second index is used to indicate a number of completed requests per unit time of the running environment; determining a third index of the computing node based on the computing parameters, wherein the third index is used to indicate a resource constraint of the computing node; and determining the target threshold range according to the first index, the second index, and the third index.

4. The method of claim 2, wherein the determining the reference application scenario corresponding to the target threshold range as the target application scenario comprises: in a case where the target threshold range is a first range, determining a first reference application scenario as the target application scenario, wherein the first reference application scenario is a high-concurrency short-text scenario; in a case where the target threshold range is a second range, determining a second reference application scenario as the target application scenario, wherein the second reference application scenario is a high-concurrency long-text scenario; and in a case where the target threshold range is a third range, determining a third reference application scenario as the target application scenario, wherein the third reference application scenario is an edge low-power-consumption scenario.

5. The method of claim 3, wherein the determining the reference application scenario corresponding to the target threshold range as the target application scenario comprises: ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ Before determining that the model parameters, the framework parameters and the computing parameters meet a target threshold range, comprising: Obtaining a reference data set corresponding to at least one reference model, wherein the reference data set is used to indicate reference model parameters, reference framework parameters and reference computing parameters; Performing cluster analysis on the reference data set to obtain at least one cluster point; Determining a reference threshold range based on the cluster point.

6. The method of claim 5, wherein: The reference threshold range is determined based on the cluster point, comprising: Determining a first index range, a second index range and a third index range corresponding to the cluster point; Taking the cluster point as a midpoint, the first index range, the second index range and the third index range are determined as the reference threshold range.

7. The method of any one of claims 1 to 6, wherein: The weight set is determined according to the target application scenario, comprising: Obtaining a scene weight corresponding table, wherein the scene weight corresponding table is used to indicate the corresponding relationship between at least one reference application scenario and a reference weight set; According to the target application scenario, the scene weight corresponding table is searched to determine the weight set.

8. The method of claim 7, wherein: Before obtaining the scene weight corresponding table, one of the following is included: Based on the reference data set of at least one reference model, a reference weight set corresponding to the reference data set is obtained through a weight model; Obtaining a weight set generation rule, and determining the reference weight set corresponding to the reference application scenario according to the weight set generation rule.

9. The method of any one of claims 1 to 6, wherein: The first score of the target model is calculated using the weight set and the standard value corresponding to each of the model parameters, the framework parameters and the computing parameters, comprising: Determining target weights in the weight set corresponding to each of the at least one model parameter, the framework parameter and the computing parameter; According to the target weight corresponding to each of the at least one model parameter, the framework parameter and the computing parameter, and the standard value corresponding to each of the at least one model parameter, the framework parameter and the computing parameter, weighted sum is performed to obtain the first score.

10. The method of claim 9, wherein: Based on the reference value corresponding to each of the model parameters, the framework parameters and the computing parameters, the standard value corresponding to each of the model parameters, the framework parameters and the computing parameters is determined, comprising: Determining the reference value corresponding to each of the at least one model parameter, the framework parameter and the computing parameter; According to the reference value corresponding to each of the at least one model parameter, the framework parameter and the computing parameter, the standard value corresponding to each of the at least one model parameter, the framework parameter and the computing parameter is obtained.

11. The method of claim 10, wherein: According to at least one of the model parameters, the framework parameters and the computing parameters and the corresponding reference values, the standard values corresponding to the at least one of the model parameters, the framework parameters and the computing parameters are obtained, comprising: In the case of the model parameters, the framework parameters and the computing parameters being first parameters, the ratio of the model parameters, the framework parameters and the computing parameters to the corresponding reference values is determined as the standard value, wherein the first parameters are positive indicators; In the case of the model parameters, the framework parameters and the computing parameters being second parameters, the ratio of the corresponding reference values to the model parameters, the framework parameters and the computing parameters is determined as the standard value, wherein the second parameters are negative indicators.

12. The method of claim 11, wherein, According to at least one of the model parameters, the framework parameters and the computing parameters and the corresponding reference values, the standard values corresponding to the at least one of the model parameters, the framework parameters and the computing parameters are obtained, comprising: In the case of the model parameters, the framework parameters and the computing parameters being qualitative parameters, at least one reference value corresponding to the model parameters, the framework parameters and the computing parameters is determined; The standard value is determined according to the model parameters, the framework parameters and the computing parameters at the at least one reference value.

13. The method of claim 10, wherein, Determining the reference values corresponding to the at least one of the model parameters, the framework parameters and the computing parameters, comprising: Obtaining a reference data set corresponding to each of the at least one reference model; The median of the reference data set is determined as the reference value corresponding to each of the at least one of the model parameters and the framework parameters.

14. The method of claim 10, wherein, Determining the reference values corresponding to the at least one of the model parameters, the framework parameters and the computing parameters, comprising: Determining the hardware specifications corresponding to the computing nodes of the target model; The hardware specifications are determined as the reference values corresponding to the at least one of the computing parameters.

15. The method of claim 1, wherein, The model parameters corresponding to the target model, the framework parameters of the running environment of the target model and the computing parameters of the computing nodes of the target model are obtained, comprising: The generation efficiency, response delay and video memory occupancy of the target model are determined as the model parameters; The throughput rate, tail delay and quantization support of the running environment are determined as the framework parameters; The focus peak value of the computing nodes, the memory bandwidth and the energy efficiency ratio are determined as the computing parameters.

16. The method of claim 1, wherein, The first score of the target model is calculated by using the weight set and the standard values corresponding to the model parameters, the framework parameters and the computing parameters, comprising: determine a sub-score set calculated between the weight set and the standard value corresponding to each of the model parameter, the framework parameter and the calculation parameter; sum the sub-score set to obtain the first score; display the sub-score set and the first score on a display interface.

17. An efficiency evaluation apparatus of a model inference system, comprising: a scene determination module configured to obtain a model parameter corresponding to a target model, a framework parameter of a running environment of the target model and a calculation parameter of a calculation node of the target model, and determine a target application scenario of the target model, wherein the target application scenario is used to indicate a task performed by the target model; a weight determination module configured to determine a weight set according to the target application scenario, wherein the weight set is used to indicate a weight value corresponding to each of the model parameter, the framework parameter and the calculation parameter; a normalization module configured to determine a standard value corresponding to each of the model parameter, the framework parameter and the calculation parameter based on a benchmark reference value corresponding to each of the model parameter, the framework parameter and the calculation parameter; a model evaluation module configured to calculate a first score of the target model by using the weight set and the standard value corresponding to each of the model parameter, the framework parameter and the calculation parameter, wherein the first score is used to indicate an efficiency of a target model inference system.

18. An electronic device, comprising: a memory configured to store a computer program; a processor configured to implement steps of the efficiency evaluation method of the model inference system according to any one of claims 1 to 16 when executing the computer program.

19. A computer readable storage medium, comprising: a computer program stored in the computer readable storage medium, wherein the computer program is configured to implement steps of the efficiency evaluation method of the model inference system according to any one of claims 1 to 16 when executed by a processor.

20. A computer program product, comprising computer instructions, wherein the computer instructions are configured to implement steps of the efficiency evaluation method of the model inference system according to any one of claims 1 to 16 when executed by a processor. ​ ​ ​

Images