Model performance testing method and device, electronic equipment and storage medium

Abstract

The application discloses a model performance test method and device, an electronic device and a storage medium, relates to the field of artificial intelligence technology, and calculates the theoretical maximum token throughput of a target large language model based on the video memory bandwidth of a graphics processing unit, the number of bytes corresponding to the model parameter quantity and the quantization precision, and the memory bandwidth utilization rate, simultaneously obtains a benchmark performance throughput, combines the theoretical maximum token throughput and a concurrent competition loss coefficient to calculate a theoretically corresponding first concurrency, and then performs model testing based on the first concurrency to obtain an actual maximum token throughput and a corresponding second concurrency and generate a model performance test result. Therefore, the problem that the evaluation is time-consuming and labor-intensive, the result is easily affected by human factors, and the accuracy and consistency are poor due to the manual testing by relying on manual intervention, the constantly approximating and trying manner, and the lack of combination of the hardware core bottleneck and the key parameters to derive a reasonable test starting point can be solved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a model performance testing method and apparatus, electronic device and storage medium. Background Technology

[0002] With the continuous evolution of artificial intelligence technology, large language models, as a core technology in the field of natural language processing, have been widely applied in various industry scenarios such as intelligent customer service, content generation, and data analysis. Performance evaluation of large language models typically involves key indicators such as computing speed, memory usage, and power consumption. The evaluation system covers the entire process from hardware resource assessment to software service optimization, including GPU computing power analysis, model parameter configuration, and concurrent load simulation. Specifically, current mainstream evaluation methods rely on GPU-accelerated devices, simulating concurrent requests through multi-threading or containerization to evaluate the throughput (TPS) and resource utilization of the model service under different loads. Among these, GPU memory bandwidth, as one of the performance bottlenecks, has a significant impact on model inference efficiency; therefore, its quantitative modeling has become an important component of the evaluation system.

[0003] However, existing performance evaluation methods for large language models rely on manually setting concurrency levels and gradually approximating them, lacking systematic modeling of the dynamic relationship between hardware resources and model parameters. This results in lengthy evaluation processes and poor consistency of results. Therefore, how to quickly locate the performance boundaries of model services under specific hardware configurations and improve the universality and reliability of evaluation results through cloud-based sharing mechanisms has become a pressing technical challenge. Summary of the Invention

[0004] This disclosure provides a model performance testing method, apparatus, electronic device, and storage medium to at least address the problem of quickly locating the performance boundaries of a model service under specific hardware configurations in related technologies.

[0005] This application provides a model performance testing method, including:

[0006] Based on the graphics processor memory bandwidth, model parameter count, quantization precision corresponding to the number of bytes, and memory bandwidth utilization, the theoretical maximum word throughput of the target large language model is calculated.

[0007] Obtain the baseline performance throughput of the target large language model, and calculate the first concurrency corresponding to the theoretical maximum word throughput based on the theoretical maximum word throughput, the baseline performance throughput, and the concurrency competition loss coefficient.

[0008] Based on the first concurrency level, perform model testing on the target large language model until the actual maximum word throughput and its corresponding second concurrency level are obtained, and generate model performance test results.

[0009] Optionally, based on the first concurrency level, perform model tests on the target large language model, including:

[0010] An adaptive step size mechanism is adopted to dynamically adjust the step size of the concurrency based on the resource utilization of the graphics processor, and the model test is performed based on the first concurrency.

[0011] Optionally, resource utilization includes: video memory utilization and computing power utilization;

[0012] An adaptive step-size mechanism is adopted to dynamically adjust the increment / decrement step size of the concurrency based on the resource utilization of the graphics processor, including:

[0013] When the video memory utilization rate exceeds the first threshold or the computing power utilization rate exceeds the second threshold, the increment / decrement step size of the concurrency number is set to 1.

[0014] When the video memory utilization rate is lower than the third threshold and the computing power utilization rate is lower than the fourth threshold, the increment / decrement step size of the concurrency number is set to an integer greater than 1.

[0015] Optionally, the method also includes:

[0016] The theoretical maximum word throughput, actual maximum word throughput, first concurrency, and second concurrency are uploaded to the cloud sharing module.

[0017] Optionally, before performing model testing on the target large language model based on the first concurrency level, the method further includes:

[0018] Based on the model parameters of the local hardware and the target large language model, the cloud records in the cloud sharing module are matched, and the concurrent number with the highest number of votes is selected as the first concurrent number for model testing.

[0019] Optionally, obtain the baseline performance throughput of the target large language model, including:

[0020] The target large language model is subjected to single-concurrency testing. The single-concurrency test is executed in a multi-threaded manner, with each thread independently calling the application interface of the target large language model service for inference testing.

[0021] Calculate the lexical throughput for each thread, and use the average lexical throughput of all threads as the baseline performance throughput.

[0022] Optionally, the concurrency competition loss coefficient is obtained by constructing a regression model based on historical test data and dynamically predicting it according to the current graphics processor model and the type of the target large language model.

[0023] This application also provides a model performance testing device, including:

[0024] The first computing unit is used to calculate the theoretical maximum word throughput of the target large language model based on the graphics processor memory bandwidth, the number of model parameters, the number of bytes corresponding to the quantization precision, and the memory bandwidth utilization.

[0025] The second computing unit is used to obtain the baseline performance throughput of the target large language model, and calculate the first concurrency corresponding to the theoretical maximum word throughput based on the theoretical maximum word throughput, the baseline performance throughput, and the concurrency competition loss coefficient.

[0026] The test unit is used to perform model tests on the target large language model based on the first concurrency level until the actual maximum word throughput and its corresponding second concurrency level are obtained, and the model performance test results are generated.

[0027] Optionally, the test unit is also used for:

[0028] An adaptive step size mechanism is adopted to dynamically adjust the step size of the concurrency based on the resource utilization of the graphics processor, and the model test is performed based on the first concurrency.

[0029] Optionally, resource utilization includes: video memory utilization and computing power utilization;

[0030] The test unit includes:

[0031] The first setting module is used to set the increment / decrement step size of the number of concurrent users to 1 when the video memory utilization exceeds the first threshold or the computing power utilization exceeds the second threshold.

[0032] The second setting module is used to set the increment / decrement step size of the number of concurrent users to an integer greater than 1 when the video memory utilization rate is lower than the third threshold and the computing power utilization rate is lower than the fourth threshold.

[0033] Optionally, the model performance testing apparatus may also include:

[0034] The upload unit is used to upload the theoretical maximum word throughput, the actual maximum word throughput, the first concurrency, and the second concurrency to the cloud sharing module.

[0035] Optionally, before performing model testing on the target large language model based on the first concurrency level, the model performance testing apparatus further includes:

[0036] The selection unit is used to match the model parameters of the target large language model with the cloud records in the cloud sharing module based on the local hardware, and selects the concurrent number with the highest number of votes as the first concurrent number for model testing.

[0037] Optionally, the second computing unit includes:

[0038] The testing module is used to perform single-concurrency testing on the target large language model. The single-concurrency test is executed in a multi-threaded manner, with each thread independently calling the application interface of the target large language model service to perform inference testing.

[0039] The computation module is used to calculate the lexical throughput of each thread, and uses the average lexical throughput of all threads as the baseline performance throughput.

[0040] Optionally, the concurrency competition loss coefficient is obtained by constructing a regression model based on historical test data and dynamically predicting it according to the current graphics processor model and the type of the target large language model.

[0041] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above methods.

[0042] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above methods.

[0043] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above methods.

[0044] This application solves the problems in existing technologies where manual testing relies on human intervention, employs continuous approximation and attempts, and fails to derive a reasonable testing starting point based on hardware bottlenecks and key parameters. This results in time-consuming and labor-intensive evaluations, and the results are easily affected by human factors, leading to poor accuracy and consistency. The method achieves the technical effect of quickly locating the maximum performance throughput boundary and optimal concurrent service capacity of the target large language model under specific hardware configurations, improving evaluation efficiency, and ensuring the reliability of evaluation results. It also addresses the need to calculate the theoretical maximum word throughput based on graphics processor memory bandwidth, model parameter quantity, quantization precision corresponding to the number of bytes, and memory bandwidth utilization. Simultaneously, it obtains the baseline performance throughput and, combined with the theoretical maximum word throughput and concurrent competition loss coefficient, calculates the theoretical corresponding first concurrency. Then, it performs model testing based on this first concurrency to obtain the actual maximum word throughput and corresponding second concurrency, generating model performance test results.

[0045] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0046] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0047] Figure 1 A flowchart illustrating a model performance testing method provided in an embodiment of this application;

[0048] Figure 2 This is a schematic diagram of the structure of a model performance testing device provided in an embodiment of this application;

[0049] Figure 3 This is a schematic diagram of another model performance testing device provided in an embodiment of this application. Detailed Implementation

[0050] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0051] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] The specific application environment architecture or specific hardware architecture on which the execution of the combined model performance testing method depends is described here.

[0053] The embodiments of this application provide a model performance testing method. Figure 1 This is a flowchart illustrating a model performance testing method provided in an embodiment of this application.

[0054] like Figure 1 As shown, the method includes the following steps:

[0055] Step 101: Calculate the theoretical maximum word throughput of the target large language model based on the graphics processor memory bandwidth, the number of model parameters, the number of bytes corresponding to the quantization precision, and the memory bandwidth utilization.

[0056] In some embodiments, during the performance evaluation of the target large language model, step 101 is a key step in accurately defining the upper limit of the model's performance from a theoretical perspective. Its core purpose is to derive the theoretical maximum token throughput of the target large language model by integrating the core hardware parameters on which the large language model depends for operation with the key characteristics of the model itself. This provides a clear and reliable theoretical reference for subsequent actual performance testing, avoiding the problem of unclear testing direction caused by the lack of theoretical support in traditional evaluations.Among them, GPU memory bandwidth (hereinafter referred to as GPU_BW) refers to the amount of data transfer that can be completed per second between the GPU's computing units and storage units. Its unit is usually measured in GB / s. This parameter is the core bottleneck determining the throughput of large language models—because when large language models perform inference services, they need to continuously transfer model parameters, input text data, and intermediate calculation results generated during inference between the GPU's memory and computing units. If GPU_BW is insufficient, even if the GPU's computing units have strong computing power, the data transfer speed will not be able to keep up with the computing demands, causing the computing units to be idle, ultimately leading to… The efficiency of token generation is directly constrained; Model_Size refers to the total number of parameters contained in the target large language model, usually measured in "B" (billions), such as 1.5B, 7B, etc. This parameter directly determines the total amount of data that the large language model needs to load and process in a single inference process. The larger the Model_Size, the more parameter data needs to be transmitted in a single inference, and the demand for GPU_BW will also increase accordingly. It is a core model characteristic indicator affecting token throughput; Bytes corresponding to quantization precision (hereinafter referred to as Byte_Size) refers to the number of bytes that the target large language model has processed. After quantization optimization, the number of bytes of storage space occupied by each model parameter varies depending on the quantization precision level. For example, when using FP16 (half-precision floating-point) quantization, each parameter occupies 2 bytes, while when using INT8 (8-bit integer) quantization, each parameter occupies only 1 byte. This parameter directly affects the space occupied by the model parameters in GPU memory, and thus relates to the actual utilization efficiency of GPU_BW when transmitting parameter data. The higher the quantization precision (i.e., the larger the Byte_Size), the more GPU_BW resources are required to transmit the same number of parameters; the memory bandwidth utilization (hereinafter referred to as BW_efficiency) is also affected. Memory bandwidth utilization (MBB) is a parameter that measures the actual utilization rate of GPU BW. Its value ranges from 0 to 1. Due to objective factors such as scheduling latency in GPU hardware architecture, differences in data flow access patterns for model inference, and potential competition for hardware resources, GPU memory bandwidth cannot be fully utilized. This parameter can be calibrated through machine learning using a large amount of historical evaluation data. It can also be initially filled with an empirical value (usually 0.6). Furthermore, it needs to be dynamically adjusted according to the hardware differences of different GPU brands (such as differences in architecture design and manufacturing processes of GPUs from different manufacturers, which can lead to differences in memory bandwidth utilization efficiency) to ensure the accuracy of the parameter value.When calculating the theoretical maximum token throughput, the four key parameters GPU_BW, Model_Size, Byte_Size, and BW_efficiency must be substituted into the preset calculation formula (i.e., theoretical maximum token throughput TPS_theory_max = GPU_BW / (Model_Size / Byte_Size) / BW_efficiency). Through the derivation of this formula, the theoretical performance ceiling of the target large language model under the current hardware configuration and model characteristics can be accurately obtained. This sets a reasonable target range for subsequent actual testing, avoiding blind testing. The beneficial effect of this step is that by focusing on the core parameters affecting token throughput and combining scientific formulas and calibrable coefficients to calculate the theoretical maximum throughput, it solves the problem of lacking theoretical basis and blindly estimating performance ceilings in traditional evaluations. It provides a precise starting point for subsequent actual testing, reduces invalid testing steps, and improves overall evaluation efficiency.

[0057] Step 102: Obtain the baseline performance throughput of the target large language model, and calculate the first concurrency corresponding to the theoretical maximum word throughput based on the theoretical maximum word throughput, the baseline performance throughput, and the concurrency competition loss coefficient.

[0058] In some embodiments, step 102 is a core transitional step connecting theoretical derivation and actual testing in the performance evaluation of the target large language model. Its core logic is to obtain the model's basic performance data in a scenario without resource contention, and combine it with the theoretical performance upper limit and resource contention loss characteristics to calculate the first concurrency number that matches the theoretical maximum token throughput. This provides an accurate starting point for subsequent actual testing and avoids the resource waste and inefficiency caused by blindly setting the concurrency number in traditional evaluations. Among them, the benchmark performance throughput (denoted as TPS_Bench) refers to the number of tokens generated per second by the target large language model in a single-concurrency scenario. This parameter is a direct reflection of the model's basic performance. Since the GPU's computing power, memory bandwidth, and other core hardware resources are allocated only for the model's single inference in a single-concurrency state, there is no resource contention between multiple tasks. Therefore, the token throughput measured at this time can most realistically reflect the model's own basic performance level and is a key reference for calculating the reasonable concurrency number in subsequent multi-concurrency scenarios. To obtain this benchmark performance throughput, actual testing is required. Specifically, the testing module in the evaluation device needs to be used. First, configure the API address and access key of the target large language model service, and ensure that no other tasks occupy GPU resources in the test environment. Then, initiate a single-threaded (or single-container instance, to further reduce inter-thread interference) inference request, record the total number of tokens output by the model in this request and the total time consumed to complete the inference, and then obtain the benchmark performance throughput TPS_Bench under single concurrency by calculating "total number of tokens ÷ total time". The entire testing process must ensure the stability of the hardware environment to avoid the accuracy of the benchmark data being affected by temporary load fluctuations.

[0059] The concurrency contention loss coefficient (denoted as Loss) is a key parameter for quantifying the impact of hardware resource contention on performance in multi-concurrency scenarios, with a value ranging from 0 to 1. When multiple concurrent tasks call a large language model service simultaneously, GPU computing power, memory bandwidth, and other resources are shared by multiple tasks, inevitably leading to resource contention. For example, multiple tasks simultaneously requesting memory data transfer can cause bandwidth scheduling delays, or contention for computing units can cause computing resources to wait. These factors can prevent the total throughput of multi-concurrency scenarios from reaching the ideal value of "baseline performance throughput × number of concurrency". Therefore, a Loss coefficient needs to be introduced to correct for this performance loss. The determination of this coefficient needs to be based on a large amount of historical evaluation data and dynamically calibrated through machine learning. For example, for different GPU models and different combinations of model parameters, the ratio of "actual total throughput to ideal total throughput (TPS_Bench × number of concurrency)" in past multi-concurrency tests is statistically analyzed. These ratios are used as samples to train and obtain the loss coefficient adapted to specific scenarios. At the same time, it can also be initially filled based on empirical values ​​and then continuously optimized through subsequent test data to ensure that the coefficient can accurately reflect the degree of resource competition in different scenarios.

[0060] When calculating the first concurrency (denoted as C, i.e., the concurrency corresponding to the theoretical maximum word throughput), the theoretical maximum word throughput (TPS_theory_max) obtained in step 101, the benchmark performance throughput (TPS_Bench) obtained in this step, and the concurrency contention loss coefficient (Loss) are used as core parameters and substituted into the preset calculation formula "C = TPS_theory_max / TPS_Bench × Loss" for derivation. The logic of this formula is as follows: Under ideal conditions (no resource contention, Loss = 1), the concurrency required to reach the theoretical maximum word throughput should be "theoretical maximum word throughput ÷ benchmark performance throughput", that is, the basic performance of a single concurrency is superimposed to the theoretical upper limit through multiple concurrency; however, in reality, there is resource contention, and the total performance of multiple concurrency will be lost. Therefore, the ideal concurrency needs to be corrected with the Loss coefficient to finally obtain the first concurrency that can adapt to the theoretical maximum word throughput. This value provides a clear starting point for subsequent actual testing and avoids the inefficient problem of blindly trying different concurrency in traditional testing.

[0061] Step 103: Based on the first concurrency level, perform model testing on the target large language model until the actual maximum word throughput and its corresponding second concurrency level are obtained, and generate model performance test results.

[0062] In some embodiments, the first concurrency count calculated in step 102 is used as the initial test starting point. Through systematic iterative testing and dynamic adjustment, the actual maximum token throughput of the model and the corresponding optimal concurrency count (i.e., the second concurrency count) are accurately located. The test process data is integrated to generate complete model performance test results, providing a practical basis for subsequent model optimization, hardware adaptation or selection. When performing initial model testing based on the first concurrency level, it is necessary to rely on the actual testing module in the evaluation device. Specifically, a clear load simulation method can be adopted: one is to simulate concurrency using local multi-threading, configuring API access permissions (including API address and access key) for each thread to enable all threads to initiate inference requests synchronously. After the requests are completed, the total number of token outputs and inference time of each thread are counted, and the word throughput (TPS_thread) of a single thread is calculated. Then, the average of TPS_thread of all threads is taken to obtain the actual word throughput (TPS_C) under the first concurrency level. The other is to simulate concurrency using container instances. If it is necessary to reduce resource interference between threads to improve test accuracy, container instances with the same number as the first concurrency level can be created through container cluster orchestration software (such as Kubernetes). Each container instance image has built-in independent testing tools that can call the large language model API to perform inference independently. The actual testing module summarizes the total number of token outputs and total time of all container instances, and calculates TPS_C by "total number of tokens ÷ total time". Both methods require real-time shielding of irrelevant tasks from GPU resource consumption to ensure the stability of the test environment.

[0063] When iteratively testing to find the actual maximum word throughput, a clear judgment logic must be followed: If the TPS_C under the current first concurrency is greater than the theoretical maximum word throughput (TPS_theory_max) in step 101, or the difference between the two is less than the preset error threshold (e.g., 3%-5%), it means that the current concurrency has not reached the hardware performance bottleneck, and the concurrency needs to be increased to continue testing (i.e., test TPS_C+1 corresponding to C+1 concurrency); if TPS_C is less than TPS_theory_max and the difference exceeds the threshold, it means that the current concurrency may exceed the hardware carrying capacity, and the concurrency needs to be reduced to test (i.e., test TPS_C-1 corresponding to C-1 concurrency). At the same time, the concurrency adjustment step size needs to be adjusted in conjunction with the hardware resource awareness step size adaptive mechanism: by monitoring the GPU's computing power utilization and memory usage in real time, if the GPU computing power utilization exceeds 90% or the memory usage exceeds 95% (i.e., the critical saturation state), in order to avoid skipping the optimal concurrency, the step size is set to 1 and finely adjusted step by step; if the GPU computing power utilization is less than 50% and the memory usage is less than 60% (i.e., the resource idle state), the step size can be set to 2 or larger to speed up the testing process.

[0064] Iterative testing must continue until the performance bottleneck condition is met: when testing a certain concurrency level (C+X), its TPS_C+X is greater than the TPS_C+X+1 of the adjacent higher concurrency level (C+X+1), and greater than the TPS_C+X-1 of the adjacent lower concurrency level (C+X-1); or when testing a certain concurrency level (CX), its TPS_C-X is greater than the TPS_C-X+1 of C-X+1 and greater than the TPS_C-X-1 of CX-1. In this case, TPS_C+X or TPS_C-X is the actual maximum word throughput, and the corresponding C+X or CX is the second concurrency level. The final model performance test results, in addition to including the actual maximum word throughput and the second concurrency level, also need to integrate real-time changes in GPU computing power utilization and memory bandwidth usage during the testing process to form a complete performance report.

[0065] This application solves the problems in existing technologies where manual testing relies on human intervention, employs continuous approximation and attempts, and fails to derive a reasonable testing starting point based on hardware bottlenecks and key parameters. This results in time-consuming and labor-intensive evaluations, and the results are easily affected by human factors, leading to poor accuracy and consistency. The method achieves the technical effect of quickly locating the maximum performance throughput boundary and optimal concurrent service capacity of the target large language model under specific hardware configurations, improving evaluation efficiency, and ensuring the reliability of evaluation results. It also addresses the need to calculate the theoretical maximum word throughput based on graphics processor memory bandwidth, model parameter quantity, quantization precision corresponding to the number of bytes, and memory bandwidth utilization. Simultaneously, it obtains the baseline performance throughput and, combined with the theoretical maximum word throughput and concurrent competition loss coefficient, calculates the theoretical corresponding first concurrency. Then, it performs model testing based on this first concurrency to obtain the actual maximum word throughput and corresponding second concurrency, generating model performance test results.

[0066] In this disclosure, the specific implementation of the operation of "performing model testing on the target large language model based on the first concurrency" is varied. For clarity, some exemplary implementation methods are described below: an adaptive step size mechanism is adopted to dynamically adjust the increase or decrease step size of the concurrency based on the resource utilization of the graphics processor, and model testing is performed by moving up and down steps based on the first concurrency.

[0067] Furthermore, resource utilization includes: video memory utilization and computing power utilization; an adaptive step size mechanism is adopted to dynamically adjust the increment / decrement step size of the number of concurrent connections based on the resource utilization of the graphics processor, including: when the video memory utilization exceeds the first threshold or the computing power utilization exceeds the second threshold, the increment / decrement step size of the number of concurrent connections is set to 1; when the video memory utilization is lower than the third threshold and the computing power utilization is lower than the fourth threshold, the increment / decrement step size of the number of concurrent connections is set to an integer greater than 1.

[0068] Specifically, the test begins with the initial concurrency level and dynamically adjusts the increase or decrease of the concurrency level. This avoids the problems of "low efficiency when resources are idle" or "poor accuracy when resources are saturated" in traditional fixed-step tests, while accurately approximating the actual performance boundaries of the target large language model. The core criterion for the adaptive step-size mechanism is the real-time resource utilization of the graphics processing unit (GPU). This utilization includes two key indicators: memory utilization and computing power utilization. Memory utilization reflects the proportion of GPU memory occupied by the target large language model's inference tasks. Since GPU memory bandwidth is the core bottleneck for the throughput of large model tokens, excessively high memory utilization (e.g., approaching saturation) means that the data transmission channel is about to become congested. At this point, a significant adjustment of the concurrency level could easily skip the optimal performance point. Computing power utilization reflects the proportion of GPU computing units occupied by inference tasks. Excessively high computing power utilization indicates that computing resources are approaching their capacity limit, and the concurrency level also needs to be carefully adjusted to ensure test accuracy.

[0069] Specifically, the execution logic of the adaptive step-size mechanism needs to be based on two types of resource utilization threshold judgments: First, when the GPU memory utilization exceeds the first threshold, or the computing power utilization exceeds the second threshold, it indicates that the current GPU resources are in a critical saturation state. For example, if the first threshold is set to 92% (which can be calibrated based on historical data of memory stability for different GPU models), when the memory utilization reaches 93%, it means that further increasing the number of concurrent connections is likely to cause memory bandwidth congestion, thereby causing a sharp drop in word throughput. If the second threshold is set to 88% (similarly calibrated based on GPU computing power characteristics), when the computing power utilization reaches 89%, it means that the computing unit is close to full load, and continuing to increase the number of concurrent connections in large steps is likely to cause task queuing delays. At this time, in order to avoid skipping the "second number of concurrent connections corresponding to the actual maximum word throughput", the step size of increasing or decreasing the number of concurrent connections needs to be strictly set to 1. That is, each test only increases or decreases the current number of concurrent connections by 1 before starting the next round of testing. By fine-tuning step by step, it is ensured that the critical point of performance transition from increase to decrease can be accurately captured. Secondly, when the GPU's memory utilization is below the third threshold and the computing power utilization is below the fourth threshold, it indicates that the GPU resources are currently idle. For example, if the third threshold is set to 55% and the fourth threshold is set to 45%, and the memory utilization is 52% and the computing power utilization is 42%, it means that the GPU still has a large amount of memory bandwidth and computing resources that are not being utilized. If the number of concurrent connections is adjusted in increments of 1, it will lead to too many test rounds and low efficiency. Therefore, the increment / decrement step size of the number of concurrent connections should be set to an integer greater than 1 (such as 2, 3, etc., the specific value can be dynamically selected according to the degree of resource idleness, and the step size can be appropriately increased when the resources are more idle). By adjusting in large steps, the test range can be quickly narrowed and the number of invalid test rounds can be reduced. For example, starting from the first number of concurrent connections C, if the resources are idle, the number of concurrent connections C+2 can be tested directly. Then, based on the resource utilization and word throughput under this number of concurrent connections, the direction of subsequent step size adjustments can be further determined to achieve the test goal of "improving efficiency when idle and refining when saturated".

[0070] In the actual test execution process, the first round of model testing should be initiated with the first concurrency calculated in step 102 as the initial value, and the real-time memory usage and computing power usage of the GPU should be collected simultaneously. Then, the step size should be determined according to the threshold judgment logic mentioned above. If the resources are idle after the first round of testing, the concurrency should be adjusted by a large step size (e.g., C→C+2). If the resources are critically saturated, the concurrency should be adjusted by a step size of 1 (e.g., C→C+1 or C→C-1). After each round of testing, the GPU resource usage should be re-monitored, the step size should be dynamically updated, and the next round of testing should be carried out until the critical point of "the word throughput at a certain concurrency is greater than the word throughput of the adjacent higher and lower concurrency" is captured. At this time, the concurrency is the second concurrency, and the corresponding word throughput is the actual maximum word throughput.

[0071] In this embodiment of the disclosure, the model performance testing method includes other specific implementation steps in addition to the foregoing content. To clearly present these components, the following detailed description of the relevant specific implementation methods is provided: The theoretical maximum word throughput, the actual maximum word throughput, the first concurrency, and the second concurrency are uploaded to the cloud sharing module.

[0072] Specifically, the core performance data generated during a single local evaluation will be stored in the cloud, providing a reference for subsequent evaluations of other clients using the same or similar hardware and model configurations. This avoids repeating the entire process from theoretical derivation to actual testing, reducing the waste of human and hardware resources. The core data content and sources to be uploaded are clearly defined: The theoretical maximum word throughput (TPS_theory_max) is the theoretical performance ceiling calculated in step 101, combining GPU memory bandwidth, model parameter count, quantization precision (number of bytes), and memory bandwidth utilization. The actual maximum word throughput (TPS_real) is the peak performance of the target large language model under the current hardware configuration, determined in step 103 after iterative testing using the first concurrency level as a starting point and an adaptive step-size mechanism. The first concurrency level (C) is the theoretical starting point derived in step 102 based on the theoretical maximum word throughput, single-concurrency baseline performance throughput, and concurrency contention loss coefficient. The second concurrency level (C_real) is the optimal concurrency level corresponding to the actual maximum word throughput in step 103, a key parameter reflecting the optimal service capacity of the target large language model under the current hardware environment. These four types of data together constitute the core results of a single evaluation, covering theoretical performance boundaries, actual performance peaks, and corresponding concurrency references, possessing extremely high reusability.

[0073] By uploading the core data from a single evaluation to a cloud-based sharing module, the reuse of evaluation data under the same hardware and model configuration is achieved. When other clients conduct evaluations for the same configuration such as "GPU model + model parameter count", they can directly pull historical data (such as the second highest number of concurrent connections with the most votes) from the cloud as the starting point for testing, without having to re-derive the theoretical maximum word throughput and the first number of concurrent connections, thus significantly reducing redundant testing steps. At the same time, the cloud uses a voting mechanism to count the actual concurrent connections of different clients, which can filter out the optimal number of concurrent connections that is more in line with the actual application scenario, further improving the efficiency and accuracy of subsequent evaluations and avoiding the resource waste problem of "each client repeatedly starting from zero" in traditional evaluations.

[0074] In this embodiment of the disclosure, before performing model testing on the target large language model based on the first concurrency, the model performance testing method includes other specific implementation steps in addition to the foregoing content. To clearly present these components, the relevant specific implementation methods are described in detail below: Based on the cloud records in the cloud sharing module that match the model parameters of the local hardware and the target large language model, the concurrency with the highest number of votes is selected as the first concurrency for model testing.

[0075] Specifically, by leveraging historical evaluation data resources, the starting point for evaluation is optimized and reused, avoiding redundant theoretical derivations and benchmark tests, and further improving overall evaluation efficiency. The "local hardware parameters" specifically cover key information about the graphics processing unit (GPU) upon which the target large language model relies, including GPU model (determining hardware architecture and basic performance), GPU memory bandwidth (e.g., GPU_BW, a core bottleneck indicator for word throughput), and GPU manufacturer (different manufacturers exhibit different performance under the same parameters due to design process differences, and this needs to be used as a matching dimension). The "model parameters of the target large language model" specifically refer to the model series name (e.g., Deepseek-R1, Qwen-v1, GPT-3.5, etc., as different series of models have different inference characteristics) and the number of model parameters (e.g., 1.5B, 7B, etc., directly affecting resource consumption requirements). These hardware and model parameters together constitute the "unique identifier dimension" for matching cloud records, ensuring that only historical data completely consistent with the hardware and model configuration of the current evaluation scenario is matched, avoiding the invalidation of reference data due to configuration differences.

[0076] By reusing historical evaluation data accumulated from multiple cloud clients, the highest number of concurrent connections is used as the first concurrent connection. This eliminates the need to repeatedly derive theoretical concurrent connections and perform single-concurrency benchmark tests for the same configuration, significantly reducing evaluation time and resource consumption. Furthermore, since the highest number of concurrent connections is the result of real-world testing on multiple clients, it is more in line with actual application scenarios than the theoretical concurrent connections derived from a single client. This can further improve the accuracy of subsequent model testing and avoid the inefficiency of traditional "evaluation from scratch".

[0077] In this disclosure, there are various specific implementation methods for the operation of "obtaining the baseline performance throughput of the target large language model". For clarity, some exemplary implementation methods are described below: a single-concurrency test is performed on the target large language model. The single-concurrency test is executed in a multi-threaded manner, with each thread independently calling the application interface of the target large language model service to perform inference test; the word throughput of each thread is calculated, and the average word throughput of all threads is used as the baseline performance throughput.

[0078] Specifically, the test needs to be conducted in a single-concurrency environment without resource contention: unrelated processes are blocked, and GPU monitoring tools are used to ensure that computing power and video memory usage are both below 10% to avoid resource contention interference.

[0079] The test employs a multi-threaded, independent execution approach: 3-5 independent threads are launched, each with its own API address and access key, initiating inference requests sequentially. The total number of tokens (consistent with the model's tokenizer) and the total execution time, accurate to milliseconds, are recorded throughout the process. All threads receive identical input data, and their results are fully validated. For calculation, the TPS of each thread is first calculated as "Single-thread TPS = Total number of tokens ÷ Total execution time," and then the arithmetic average is taken as the benchmark throughput (TPS_Bench). The purpose of multi-threading is to offset random errors such as instantaneous GPU fluctuations through multiple independent tests. This method ensures the accuracy and stability of the benchmark data, providing a reliable basis for subsequent calculations of concurrency, and effectively avoiding the data fluctuation problems of traditional single-threaded tests.

[0080] Furthermore, the concurrency competition loss coefficient is obtained by constructing a regression model based on historical test data and dynamically predicting it according to the current graphics processor model and the type of the target large language model.

[0081] Specifically, the determination of the concurrency competition loss coefficient (Loss) does not rely on fixed empirical values, but is achieved through a method of "building a regression model based on historical test data + dynamic prediction combined with current hardware and model type". The core is to ensure that the Loss coefficient accurately adapts to the current graphics processing unit (GPU) model and the type characteristics of the target large language model, avoiding the deviation in the calculation of the first concurrency number due to the inability of a fixed coefficient to cover different scenarios, and further improving the accuracy of the evaluation data. Among them, the collection of "historical test data" must strictly focus on the key dimensions affecting resource competition, specifically including hardware parameters, model parameters, measured performance data and corresponding loss ratios recorded in previous large language model performance evaluations—hardware parameters mainly collect GPU models (such as NVIDIA A100, AMD...). The differences in architecture design and resource scheduling mechanisms among different models, such as MI250, directly affect the degree of concurrency contention. GPU memory bandwidth (consistent with GPU_BW, it is the core carrier of resource contention) is also considered. Model parameters are primarily collected based on the type of the target large language model (i.e., model series name, such as Deepseek-R1, Qwen-v1, GPT-3.5, etc., as different series have different inference data access patterns and computational complexity, resulting in varying intensity of resource contention) and the number of model parameters (e.g., 1.5B, 7B, etc., the larger the number of parameters, the more resources a single concurrent task consumes, and the more intense the contention becomes in multiple concurrent tasks). Actual performance data includes the single-concurrency benchmark throughput (TPS_Bench) and the actual total throughput (TPS_real_total) under different concurrency levels. The Loss prototype value (i.e., the actual loss ratio caused by resource contention in this test) is calculated using "Loss calculation prototype = TPS_real_total / (TPS_Bench × number of concurrency)".

[0082] It should be noted that the embodiments of this disclosure may include multiple steps. For ease of description, these steps are numbered, but these numbers are not a limitation on the execution time slots or execution order between the steps; these steps can be implemented in any order, and the embodiments of this disclosure do not limit this.

[0083] Corresponding to the model performance testing method described above, this disclosure also proposes a model performance testing apparatus. Since the apparatus embodiments of this disclosure correspond to the method embodiments described above, details not disclosed in the apparatus embodiments can be referred to the method embodiments described above, and will not be repeated here.

[0084] Figure 2 This is a schematic diagram of the structure of a model performance testing device provided in an embodiment of the present disclosure, as shown below. Figure 2 As shown, it includes:

[0085] The first computing unit 21 is used to calculate the theoretical maximum word throughput of the target large language model based on the graphics processor memory bandwidth, model parameter quantity, number of bytes corresponding to quantization precision, and memory bandwidth utilization.

[0086] The second computing unit 22 is used to obtain the baseline performance throughput of the target large language model, and calculate the first concurrency corresponding to the theoretical maximum word throughput based on the theoretical maximum word throughput, the baseline performance throughput and the concurrency competition loss coefficient.

[0087] Test unit 23 is used to perform model testing on the target large language model based on the first concurrency level until the actual maximum word throughput and its corresponding second concurrency level are obtained, and to generate model performance test results.

[0088] This application solves the problems in existing technologies where manual testing relies on human intervention, employs continuous approximation and attempts, and fails to derive a reasonable testing starting point based on hardware bottlenecks and key parameters. This results in time-consuming and labor-intensive evaluations, and the results are easily affected by human factors, leading to poor accuracy and consistency. The method achieves the technical effect of quickly locating the maximum performance throughput boundary and optimal concurrent service capacity of the target large language model under specific hardware configurations, improving evaluation efficiency, and ensuring the reliability of evaluation results. It also addresses the need to calculate the theoretical maximum word throughput based on graphics processor memory bandwidth, model parameter quantity, quantization precision corresponding to the number of bytes, and memory bandwidth utilization. Simultaneously, it obtains the baseline performance throughput and, combined with the theoretical maximum word throughput and concurrent competition loss coefficient, calculates the theoretical corresponding first concurrency. Then, it performs model testing based on this first concurrency to obtain the actual maximum word throughput and corresponding second concurrency, generating model performance test results.

[0089] Furthermore, in one possible implementation of this embodiment, the test unit 23 is also used for:

[0090] An adaptive step size mechanism is adopted to dynamically adjust the step size of the concurrency based on the resource utilization of the graphics processor, and the model test is performed based on the first concurrency.

[0091] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, resource utilization includes: video memory utilization and computing power utilization;

[0092] Test unit 23 includes:

[0093] The first setting module 231 is used to set the increment / decrement step size of the number of concurrent users to 1 when the video memory utilization rate exceeds the first threshold or the computing power utilization rate exceeds the second threshold.

[0094] The second setting module 232 is used to set the increment / decrement step size of the number of concurrent users to an integer greater than 1 when the video memory utilization rate is lower than the third threshold and the computing power utilization rate is lower than the fourth threshold.

[0095] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, the model performance testing device also includes:

[0096] Upload unit 24 is used to upload the theoretical maximum word throughput, the actual maximum word throughput, the first concurrency, and the second concurrency to the cloud sharing module.

[0097] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, before performing model testing on the target large language model based on the first concurrency level, the model performance testing device also includes:

[0098] Unit 25 is selected to match the cloud records in the cloud sharing module with the model parameters of the target large language model based on the local hardware, and selects the highest number of concurrent votes as the first number of concurrent votes for model testing.

[0099] Furthermore, in one possible implementation of this embodiment, such as Figure 3 As shown, the second computing unit 22 includes:

[0100] Test module 221 is used to perform single-concurrency testing on the target large language model. The single-concurrency test is executed in a multi-threaded manner, with each thread independently calling the application interface of the target large language model service to perform inference testing.

[0101] The calculation module 222 is used to calculate the lexical throughput of each thread, and uses the average lexical throughput of all threads as the baseline performance throughput.

[0102] Furthermore, in one possible implementation of this embodiment, the concurrency competition loss coefficient is obtained by constructing a regression model based on historical test data and dynamically predicting it according to the current graphics processor model and the type of the target large language model.

[0103] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of this embodiment, and the principle is the same, so it is not limited in this embodiment.

[0104] For a description of the features in the embodiment corresponding to the model performance testing device, please refer to the relevant description of the embodiment corresponding to the model performance testing method, which will not be repeated here.

[0105] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described embodiments of the model performance testing method.

[0106] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described model performance testing method embodiments at runtime.

[0107] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0108] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described model performance testing method embodiments.

[0109] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described model performance testing method embodiments.

[0110] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] The foregoing has provided a detailed description of a model performance testing method, apparatus, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A model performance testing method, characterized in that, include: Based on the graphics processor memory bandwidth, model parameter count, quantization precision corresponding to the number of bytes, and memory bandwidth utilization, the theoretical maximum word throughput of the target large language model is calculated. Obtain the baseline performance throughput of the target large language model, and calculate the first concurrency corresponding to the theoretical maximum word throughput based on the theoretical maximum word throughput, the baseline performance throughput, and the concurrency competition loss coefficient; Based on the first concurrency level, perform model testing on the target large language model until the actual maximum word throughput and its corresponding second concurrency level are obtained, and generate model performance test results; The step of performing model testing on the target large language model based on the first concurrency level includes: An adaptive step size mechanism is adopted to dynamically adjust the increase or decrease step size of the concurrency based on the resource utilization of the graphics processor, and to perform model testing by moving up and down steps based on the first concurrency; the resource utilization includes: video memory utilization and computing power utilization. The adaptive step size mechanism dynamically adjusts the increment / decrement step size of the concurrency based on the resource utilization of the graphics processor, including: when the video memory utilization exceeds a first threshold or the computing power utilization exceeds a second threshold, setting the increment / decrement step size of the concurrency to 1; when the video memory utilization is lower than a third threshold and the computing power utilization is lower than a fourth threshold, setting the increment / decrement step size of the concurrency to an integer greater than 1.

2. The model performance testing method according to claim 1, characterized in that, The method further includes: The theoretical maximum word throughput, the actual maximum word throughput, the first concurrency, and the second concurrency are uploaded to the cloud sharing module.

3. The model performance testing method according to claim 2, characterized in that, Before performing model testing on the target large language model based on the first concurrency level, the method further includes: Based on the matching of local hardware with the model parameters of the target large language model, the cloud records in the cloud sharing module are used to select the concurrent number with the highest number of votes as the first concurrent number for model testing.

4. The model performance testing method according to claim 1, characterized in that, The process of obtaining the baseline performance throughput of the target large language model includes: The target large language model is subjected to single-concurrency testing, which is performed in a multi-threaded manner, with each thread independently calling the application programming interface of the target large language model service for inference testing; Calculate the lexical throughput for each thread, and use the average of the lexical throughput of all threads as the baseline performance throughput.

5. The model performance testing method according to claim 1, characterized in that, The concurrent competition loss coefficient is obtained by constructing a regression model based on historical test data and dynamically predicting it according to the current graphics processor model and the type of the target large language model.

6. A model performance testing device, characterized in that, include: The first computing unit is used to calculate the theoretical maximum word throughput of the target large language model based on the graphics processor memory bandwidth, the number of model parameters, the number of bytes corresponding to the quantization precision, and the memory bandwidth utilization. The second calculation unit is used to obtain the baseline performance throughput of the target large language model, and calculate the first concurrency corresponding to the theoretical maximum word throughput based on the theoretical maximum word throughput, the baseline performance throughput and the concurrency competition loss coefficient. The testing unit is used to perform model testing on the target large language model based on the first concurrency number until the actual maximum word throughput and its corresponding second concurrency number are obtained, and to generate model performance test results. The step of performing model testing on the target large language model based on the first concurrency level includes: An adaptive step size mechanism is adopted to dynamically adjust the increase or decrease step size of the concurrency based on the resource utilization of the graphics processor, and to perform model testing by moving up and down steps based on the first concurrency; the resource utilization includes: video memory utilization and computing power utilization. The adaptive step size mechanism dynamically adjusts the increment / decrement step size of the concurrency based on the resource utilization of the graphics processor, including: when the video memory utilization exceeds a first threshold or the computing power utilization exceeds a second threshold, setting the increment / decrement step size of the concurrency to 1; when the video memory utilization is lower than a third threshold and the computing power utilization is lower than a fourth threshold, setting the increment / decrement step size of the concurrency to an integer greater than 1.

7. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the model performance testing method according to any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the model performance testing method according to any one of claims 1-5.

Images