Simulation test method and device of large model, electronic equipment and storage medium

Abstract

The present disclosure provides a simulation test method and device of a large model, electronic equipment and storage medium, relates to the technical field of computers, and particularly relates to the technical field of artificial intelligence such as deep learning and large models. The specific implementation scheme is as follows: obtaining inference performance data of a large model to be tested; determining a first mapping relationship of a pre-filling stage and a second mapping relationship of a decoding stage based on the inference performance data, the first mapping relationship at least including a corresponding relationship between pre-filling time consumption and input token quantity, and the second mapping relationship at least including a corresponding relationship between decoding time consumption and concurrency quantity; and performing simulated inference on the large model to be tested by using the first mapping relationship and the second mapping relationship to obtain a simulation test result. The present disclosure can improve the accuracy of the simulation test result.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the field of artificial intelligence technology such as deep learning and large models, and especially to a simulation testing method, apparatus, electronic device and storage medium for large models. Background Technology

[0002] Before large-scale models are deployed in real-world applications, their performance in different scenarios typically needs to be thoroughly evaluated. Therefore, a testing solution capable of effectively evaluating model performance is required. Summary of the Invention

[0003] This disclosure provides a simulation testing method, apparatus, device, and storage medium for large models.

[0004] According to one aspect of this disclosure, a simulation testing method for a large model is provided, comprising: acquiring inference performance data of the large model to be tested; determining a first mapping relationship for a pre-filling stage and a second mapping relationship for a decoding stage based on the inference performance data, wherein the first mapping relationship includes at least a correspondence between pre-filling time and the number of input tokens, and the second mapping relationship includes at least a correspondence between decoding time and the number of concurrent connections; and performing simulated inference on the large model to be tested using the first mapping relationship and the second mapping relationship to obtain simulation test results.

[0005] According to another aspect of this disclosure, a simulation testing apparatus for a large model is provided, comprising: an acquisition module for acquiring inference performance data of the large model to be tested; a determination module for determining a first mapping relationship for a pre-filling stage and a second mapping relationship for a decoding stage based on the inference performance data, wherein the first mapping relationship includes at least a correspondence between pre-filling time and the number of input tokens, and the second mapping relationship includes at least a correspondence between decoding time and the number of concurrent connections; and a simulation testing module for performing simulated inference on the large model to be tested using the first mapping relationship and the second mapping relationship to obtain simulation test results.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method as described above.

[0008] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method as described above.

[0009] This disclosure utilizes the first and second mapping relationships to simulate the pre-filling and decoding stages, respectively, enabling accurate prediction of inference time in various scenarios without performing actual large-scale model calculations. This significantly reduces the consumption of expensive computing resources and avoids the high computational overhead caused by repeatedly loading models.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram of timing scheduling for a large model inference architecture according to an embodiment of the present disclosure.

[0012] Figure 2 This is a flowchart of a large-scale model simulation test method according to an embodiment of the present disclosure.

[0013] Figure 3 This is a flowchart of a method for obtaining inference performance data of a large model to be tested according to an embodiment of the present disclosure.

[0014] Figure 4 This is a flowchart of a method for determining the pre-filling time using a first mapping relationship according to an embodiment of the present disclosure.

[0015] Figure 5 This is an experimental data graph showing the number of input tokens and the pre-filling time according to an embodiment of this disclosure.

[0016] Figure 6 This is an experimental data graph showing the number of concurrent requests and additional correction time according to embodiments of this disclosure.

[0017] Figure 7 This is a schematic diagram of the initial decoding time fitting according to an embodiment of the present disclosure.

[0018] Figure 8 This is a schematic diagram comparing the optimized decoding time prediction value with the actual value according to an embodiment of the present disclosure.

[0019] Figure 9 This is a difference distribution diagram of the decoding time prediction model according to an embodiment of the present disclosure.

[0020] Figure 10This is a bar chart comparing the simulated value and the actual value of the inference side first character consumption time according to the embodiments of this disclosure.

[0021] Figure 11 This is a bar chart comparing the simulated value and the actual value of the average room delay according to the embodiments of this disclosure.

[0022] Figure 12 This is a schematic diagram of the structure of a large-scale model simulation test device according to an embodiment of the present disclosure.

[0023] Figure 13 This is a block diagram of an electronic device used to implement the large-model simulation testing method of the embodiments of this disclosure. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] Large models refer to deep neural network models with hundreds of millions of parameters. These models are usually based on the Transformer architecture and learn extensive knowledge and general feature representation capabilities through unsupervised pre-training on massive datasets, thus possessing powerful natural language understanding, generation, and reasoning capabilities.

[0026] In the realm of large models, a token is the most basic unit of information used by models to process, analyze, and generate text. It can be intuitively understood as the "smallest lexical block" when machines read and understand language.

[0027] In current large-scale model inference processing, a phased processing model is typically used to manage the inference process, which clearly divides the inference flow into a prefill phase and a decoding phase. Under this model, after receiving a user request, the large model first enters the prefill phase, where it reads and processes all input tokens in parallel to understand the contextual semantics. After prefilling is complete, it enters the decoding phase, where the model adds each new output token it generates to the historical context, preparing for the prediction of the next token.

[0028] Figure 1 This is a schematic diagram of the timing scheduling of a large model inference architecture according to an embodiment of this disclosure. Figure 1As shown, during the pre-filling phase, multiple inference requests enter the pre-filling queue in the order of arrival. Each request needs to perform a one-time computation on all input tokens during its pre-filling phase to build intermediate states such as the KV cache required for subsequent decoding. Figure 1 In this code, P1, P2, and P3 represent the pre-filling time for each request, and the pre-filling time varies for different requests.

[0029] During the decoding phase, the large model generates output tokens one by one based on the intermediate states generated in the pre-filling phase. The decoding phase typically employs a continuous batching scheduling method, merging and executing decoding tasks from multiple requests. Figure 1 In the diagram, D1_1, D1_2...D2_1, D2_2... represent the decoding process of each request within different time steps. As can be seen on the timeline, the decoding stages of multiple requests are performed concurrently within the same time window, thereby maximizing the utilization of computing resources.

[0030] Currently, the industry generally lacks a comprehensive and accurate testing system for large-scale models, posing challenges to their practical application. First, it's difficult to fully assess the performance and reliability of output results in complex scenarios before deploying large models, which not only impacts user experience but also significantly increases project risk. Second, large models rely on expensive GPU clusters, making it difficult to coordinate massive computing power for full-scale load testing during the testing phase, resulting in extremely high testing costs.

[0031] To address the aforementioned issues, this disclosure provides a large-scale model simulation testing method that can be applied to the development and testing process of large-scale model systems.

[0032] Figure 2 This is a flowchart of a large-scale model simulation testing method according to an embodiment of this disclosure. Figure 2 As shown, the method includes: S101, Obtain the inference performance data of the large model to be tested.

[0033] Before obtaining inference performance data for the large model to be tested, it is necessary to determine the configuration parameters of the large model to be tested. In some embodiments, the configuration parameters include at least one of the following: maximum number of sequences (max_num_seqs), number of memory blocks (block_num), quantization parameters (e.g., quantization precision such as Int8, FP16, etc.), and hardware configuration (e.g., GPU model, memory size, computing power level, etc.).

[0034] The aforementioned configuration parameters collectively determine the upper limit of computational throughput and resource consumption characteristics of the large model under a specific environment. Based on these configuration parameters, the large model under test is run to obtain inference performance data. Inference performance data refers to a series of measured records of "input state-response time" collected during benchmark testing of the large model under test, under the resource constraints defined by the above configuration parameters, by changing input load conditions (such as different input token lengths, different numbers of concurrent requests). This data reflects the performance level of the large model under test under the current hardware environment and configuration, laying the foundation for subsequently determining the mapping relationship.

[0035] S102, determine a first mapping relationship for the pre-filling stage and a second mapping relationship for the decoding stage based on inference performance data. The first mapping relationship includes at least the correspondence between pre-filling time and the number of input tokens, and the second mapping relationship includes at least the correspondence between decoding time and the number of concurrent connections.

[0036] In some embodiments, inference performance data includes at least: the number of input tokens, the number of concurrent requests, and the corresponding pre-filling and decoding times. By analyzing the inference performance data and constructing a mapping relationship, discrete performance data can be transformed into a continuous time prediction function, thereby enabling the simulation testing system to predict unknown inference scenarios.

[0037] The first mapping relationship can be used to predict the computational latency, or prefilling time, of the large model under test during the prefill phase when processing input prompts. This prefilling time is mainly limited by computational resources, and therefore has at least a mapping relationship with the number of input tokens. The number of input tokens refers to the total number of tokens corresponding to the input text that the large model under test needs to process during the prefill phase when performing the inference task.

[0038] In one embodiment, the pre-filling time is a power function of the number of input tokens.

[0039] Because the self-attention mechanism of the underlying Transformer architecture of large models has O(N) 2 The computational complexity of pre-padding increases non-linearly with the input length. The power function relationship mathematically aligns perfectly with this underlying operational logic, ensuring the reliability of the full-scene simulation.

[0040] In another embodiment, the pre-filling time is a function of the number of input tokens as a quadratic polynomial.

[0041] Compared to a single power function, a quadratic polynomial model can simultaneously cover the mixed characteristics of computationally intensive and memory-intensive workloads. Specifically, this mapping relationship can be represented by the following formula:

[0042] in, Pre-filling time, Input the number of tokens. This is a constant term.

[0043] The second mapping relationship can be used to predict the single-step latency, i.e., decoding time, when the large model under test generates each subsequent token during the decoding phase. Decoding time is limited by memory bandwidth and resource contention, therefore it has at least a mapping relationship with the current concurrency level. Concurrency level refers to the number of inference requests simultaneously in the inference processing state within the same time window.

[0044] In constructing the first and second mapping relationships, the correlation between each candidate influencing factor and the actual time consumption is analyzed. Only relevant terms that have a significant impact on the prediction results (such as the strong correlation between pre-filling time and the number of input tokens) are retained as effective variables in the mapping relationships. This approach reduces the complexity of the mapping relationships while improving the fitting accuracy of the inference time consumption, thus making the test results obtained in subsequent simulation inference processes closer to the real inference scenario.

[0045] S103, use the first mapping relationship and the second mapping relationship to perform simulation reasoning on the large model to be tested in order to obtain simulation test results.

[0046] During the simulated inference process, the complex context encoding process is simplified to a single mapping query using the first mapping relationship, that is, the pre-filling time is directly calculated based on the input length, simulating the exclusive behavior of large models on computing resources; at the same time, the iterative token generation process is simplified to a series of dynamic delay accumulations using the second mapping relationship, simulating the dynamic contention for memory bandwidth when multiple tasks are running in parallel.

[0047] In some embodiments, the simulation test results include at least one of the following: decoding rate, end-to-end first token latency, inference-side first token latency, end-to-end whole sentence latency, inference-side whole sentence latency, inter-class latency, and maximum inter-class latency.

[0048] Based on the above technical solution, the inference process of the pre-filling stage and the decoding stage are simulated using the first and second mapping relationships, respectively. This allows for the prediction of inference time under different conditions without actually performing large-scale model inference calculations. This significantly reduces the consumption of computing resources and avoids the high computational overhead caused by repeatedly loading the model during performance testing. Furthermore, by transforming discrete inference performance data into continuous time prediction mapping relationships, this disclosure enables inference analysis of unknown inference scenarios, thereby achieving comprehensive, efficient, and accurate testing of large-scale model performance indicators and providing a reliable basis for performance evaluation and inference scheduling strategy verification.

[0049] Figure 3 This is a flowchart illustrating a method for obtaining inference performance data of a large model under test according to an embodiment of this disclosure. Figure 3 As shown, the method includes: S201 builds an inference test environment based on the configuration parameters of the large model to be tested.

[0050] Setting up an inference testing environment mainly involves two aspects: building a benchmark testing platform and initializing configuration parameters. When building the benchmark testing platform, based on the hardware information in the configuration parameters, such as the specified graphics processor model, memory capacity, and interconnect bandwidth, the corresponding physical computing nodes are selected or allocated to ensure that the computing power characteristics and memory bandwidth of the physical environment are consistent with the test target.

[0051] When initializing configuration parameters, for example, setting the maximum number of sequences (max_num_seqs) as the scheduling limit of the inference engine to determine the maximum request concurrency of the test environment; pre-allocating a key-value cache pool in video memory by using the number of video memory blocks (block_num) to define the supported context length boundary; and using quantization parameters to determine the loading method and computation precision of model weights.

[0052] S202, increment the number of input tokens according to a preset step size, and obtain the pre-filled time data corresponding to different numbers of input tokens.

[0053] In the testing program, you can set a starting value (e.g., 30 tokens) and an ending value (e.g., 30,000 tokens) for the number of input tokens, and define a fixed increment step (e.g., increasing by 500 tokens each time) to generate prompts of different lengths. For each prompt, the time interval from when the prompt is issued to when the first token is generated is recorded, i.e., the pre-filling time.

[0054] In some embodiments, to ensure data stability, the test can be repeated multiple times for each input token length, and the average value can be taken. Ultimately, a set of discrete "input token count - pre-filling time" data pairs are collected, which demonstrate the changing trend of model pre-filling time as the context length increases. These data pairs are used as inference performance data for the large model under test, providing high-confidence sample points for subsequent fitting of the first mapping relationship.

[0055] S203: For each concurrency level, change the combination of the number of input tokens and the number of output tokens to obtain decoding time data under different concurrency levels and different token combinations.

[0056] When evaluating inference performance data during the decoding phase, the number of concurrent requests is first used as the primary variable, with an initial value (e.g., 1) and an end value (e.g., 50, corresponding to max_num_seqs). A fixed increment step (e.g., increasing by 5 each time) is defined to simulate different levels of congestion. For each fixed number of concurrent requests, different combinations of input and output tokens are further iterated to cover different context length scenarios. For example, the number of input tokens is gradually increased from an initial value (e.g., 30) to an end value (e.g., 30000), with a larger step size (e.g., increasing by 3000 each time), and the number of output tokens is dynamically adjusted (e.g., controlled to 30000 minus the current number of input tokens).

[0057] For each of the above test scenarios involving "concurrency + input / output token combinations," the time interval for generating each subsequent token was recorded, i.e., the decoding time. Ultimately, a set of "concurrency quantity - decoding time" data pairs containing multi-dimensional features was collected. This data pair was used as inference performance data for the large model under test, providing crucial sample data for subsequently fitting the second mapping relationship.

[0058] As described above, by building an inference test environment based on the configuration parameters of the large model under test, the authenticity and reproducibility of the inference performance data can be ensured. By changing the number of input tokens and the number of concurrent requests according to a preset step size, inference scenarios under different context lengths and load intensities can be covered. Furthermore, by collecting the time consumption data of the pre-filling stage and the decoding stage respectively, the inference process with different performance characteristics can be distinguished and modeled, thereby providing reliable samples for subsequent construction of mapping relationships and improving the accuracy of simulation test results.

[0059] According to embodiments of this disclosure, the pre-filling time further includes: a random perturbation value; the range of the random perturbation value is determined based on the standard deviation between the inference performance data and the first mapping relationship.

[0060] In the first mapping relationship, although the quadratic polynomial can accurately predict pre-filling time, it is essentially a deterministic model. In real-world GPU inference, factors such as memory read / write fluctuations, operating system scheduling latency, and chip temperature-induced frequency throttling cause the actual pre-filling time to fluctuate randomly within a certain range, even with the exact same input length. Therefore, by superimposing random perturbation values, the actual physical noise filtered out by the fitted curve can be artificially restored, making the simulated data distribution closer to the real data distribution.

[0061] The random perturbation value disclosed herein is not a randomly generated infinite value, but rather a randomly generated value within a preset range that does not affect prediction accuracy. This preset range is obtained based on rigorous statistical analysis and experimental verification, as detailed below: First, calculate the difference between each measured sample point in the inference performance data and the predicted value of the first mapping relationship (quadratic polynomial). That is: ; in, For the actual pre-filling time, The pre-filling time calculated for the fitted curve. The difference is the sum of its parts.

[0062] Then, statistical analysis is performed on all the differences to calculate the standard deviation. This standard deviation reflects the degree of dispersion of the true data from the fitted curve.

[0063] Finally, the optimal perturbation range was determined based on this standard deviation. To balance the realism of the simulation with the accuracy of the prediction, comparative experiments were conducted to analyze the impact of different proportions (e.g., 2% to 100%) of the standard deviation as the perturbation range on the model performance. The experimental results are shown in Table 1 below, using mean squared error (MSE), root mean square error (RMSE), mean absolute error (MAE), and coefficient of determination (R²) as evaluation indicators. MSE reflects the mean squared difference between the predicted and actual values; the smaller the value, the better. RMSE is the square root of MSE, and it has the same dimensions as the original data. The smaller the value, the better. MAE reflects the average absolute difference between the predicted and actual values; the smaller the value, the better. R² reflects the goodness of fit of the model; the closer it is to 1, the better the fit.

[0064] Table 1

[0065] As shown in Table 1 above, excessive noise (such as 100% of the standard deviation of the difference) leads to a significant increase in MSE and RMSE, compromising prediction accuracy. Comparative tests revealed that the model performed best when the random disturbance value generation range was set to ±2% of the standard deviation of the difference. After adding random noise within this range, the MSE dropped to a minimum of 4605.1078, while R² remained at an extremely high level of 0.9928. This indicates that this setting successfully introduced the necessary physical randomness while maximizing the preservation of the prediction model's accuracy, achieving the optimal balance between simulation realism and accuracy.

[0066] According to embodiments of this disclosure, the first mapping relationship further includes a functional relationship between pre-filling time and the number of concurrent connections.

[0067] This disclosure significantly corrects simulation biases under high-load scenarios by introducing a concurrency dimension. Although the pre-filling stage is usually computation-driven, in actual high-throughput environments, memory bandwidth contention and scheduling overhead caused by concurrency backlog can lead to non-linear performance degradation. By introducing the concurrency factor when predicting pre-filling time, the long-tail latency under extreme pressure can be effectively predicted, ensuring the realism of the end-to-end simulation.

[0068] According to embodiments of this disclosure, a cubic polynomial function is used to represent the functional relationship between pre-filling time and the number of concurrent connections. Further, the following formula is used to represent the functional relationship between pre-filling time and the number of concurrent connections:

[0069] in, The pre-filling time is the number of concurrent processes, where n is the number of concurrent processes. This is a constant term.

[0070] The changing trend of the cubic term perfectly matches the nonlinear characteristics of the system load: when the number of concurrent users n is small, the value of the cubic term is extremely low and the model performs stably; however, when the number of concurrent users n exceeds a certain threshold, the value of the cubic term increases explosively, accurately reproducing the sharp deterioration of performance and the long-tail delay effect caused by excessive congestion.

[0071] Figure 4 This is a flowchart of a method for determining pre-filling time using a first mapping relationship according to an embodiment of this disclosure. Figure 4 As shown, the method includes: S301, the first time item is determined based on the correspondence between the pre-filling time and the number of input tokens and the number of requested input tokens.

[0072] The above steps are used to calculate the static computation baseline time of the request under ideal conditions. By calling the pre-built mapping relationship of the input token quantity dimension (such as the quadratic polynomial mentioned above), the input token quantity is substituted as an independent variable into the calculation to obtain the first time consumption term. This first time consumption term represents the theoretical physical time consumption under an ideal environment without resource contention, determined only by the computational cost of the model's self-attention mechanism and the amount of data loaded.

[0073] S302, a second time item is determined based on the correspondence between the pre-filling time and the number of concurrent requests.

[0074] The steps described above are used to calculate the additional time consumed by dynamic congestion caused by system load. The system monitors the number of global concurrent requests in the current simulation environment in real time. Subsequently, a pre-built mapping relationship for the concurrency dimension (such as the cubic polynomial model mentioned above) is invoked, and the number of concurrent requests is substituted as an independent variable to obtain the second time consumption term. This second time consumption term represents the performance degradation time caused by pipeline blocking or context switching when multiple requests simultaneously compete for memory bandwidth and scheduling resources.

[0075] S303, determine the pre-filling time based on the first time-consuming item, the second time-consuming item, and the random disturbance value.

[0076] The first and second time-consuming terms calculated above are summed to obtain the total predicted time for the deterministic part. Then, a random perturbation value satisfying a specific statistical distribution is added to this to obtain the final pre-filled time.

[0077] According to embodiments of this disclosure, the first mapping relationship is represented by the following formula:

[0078]

[0079]

[0080] in, Pre-filling time, As the first time-consuming item, This is the second time-consuming item. For random perturbation values, The input is the number of tokens, and n is the number of concurrent connections. This is a constant term.

[0081] As shown in the above equation, while the first time-consuming term based solely on the input token can characterize the theoretical computational load, it neglects the mutual interference during multi-task parallelism. By introducing a second time-consuming term based on the number of concurrent tasks, this disclosure successfully addresses the nonlinear performance degradation caused by memory bandwidth contention, resolving the problem of overly optimistic predictions under high loads. Furthermore, the superimposed random perturbation values ​​break the ideal smoothness of the mathematical model, accurately reproducing the unavoidable physical jitter in real hardware. Therefore, this comprehensive modeling approach of "static benchmark + dynamic congestion + random noise" enables the simulation system to not only predict average performance but also accurately capture long-tail latency under extreme loads, achieving accurate simulation across all dimensions.

[0082] According to embodiments of this disclosure, the second mapping relationship further includes: a functional relationship between decoding time and the number of input tokens; and / or a functional relationship between decoding time and the number of output tokens.

[0083] In the inference process of large models, although the decoding stage is mainly used to generate output tokens, the execution efficiency of the decoding stage is not only affected by the number of concurrent requests, but also closely related to the context size and output length corresponding to the inference request.

[0084] On the one hand, when the number of input tokens is large, the model needs to process more historical context information during the decoding process, which in turn consumes additional computing and memory resources, thus affecting the single-step time of the decoding stage. Therefore, the functional relationship between decoding time and the number of input tokens is introduced into a second mapping relationship to characterize the performance change trend of the decoding stage under different input scales.

[0085] On the other hand, as the number of output tokens increases, the number of generation rounds required in the decoding phase also increases, leading to an increase in overall decoding time. Therefore, the functional relationship between decoding time and the number of output tokens is incorporated into the second mapping relationship to reflect the characteristic that decoding time varies with output length.

[0086] By introducing influencing factors such as the number of input tokens and / or output tokens during the decoding stage, inference performance modeling can more comprehensively reflect the operational characteristics under actual inference scenarios. Based on the above mapping relationship, simulated inference can predict inference performance under different input sizes and concurrent loads without actually running large models, thereby reducing computational resource consumption while improving the accuracy and reliability of simulation test results.

[0087] According to embodiments of this disclosure, determining the decoding time using the second mapping relationship includes: Calculate the time components of the first-order and second-order terms corresponding to the number of input tokens, the number of output tokens, and the number of concurrent transactions, respectively. Calculate the cross-time components corresponding to the product of each pair of input token count, output token count, and concurrency count; The decoding time is determined based on the first-order time consumption component, the second-order time consumption component, the cross-time consumption component, and the preset constant term.

[0088] According to embodiments of this disclosure, the second mapping relationship is represented by the following formula:

[0089] in, Decoding time Input the number of tokens. The output token count is n, where n is the concurrency level. This is a constant term.

[0090] In this embodiment, by introducing the number of concurrent requests, the number of input tokens, and the number of output tokens as independent variables, the second mapping relationship can simultaneously reflect the comprehensive impact of system concurrent load, context size, and generation length on the performance of the decoding stage. Specifically, the quadratic term characterizes the nonlinear trend of the decoding stage under high concurrency or large-scale input conditions; the cross term reflects the coupling relationship between different influencing factors; and the linear term and constant term describe the basic time consumption and linear variation characteristics.

[0091] This disclosure breaks through the limitations of traditional models that only focus on "concurrency level" by introducing input and output variables, effectively filling the gap in considering the bandwidth consumption of KV Cache memory. More importantly, by utilizing cross-time components (such as Dnx), it accurately captures the nonlinear losses under the dual extreme conditions of "high concurrency" and "long context". This design effectively characterizes the combined performance degradation caused by memory fragmentation and cache failure, solves the problem that single-factor linear superposition cannot reproduce the coupling effect of multiple factors, and significantly improves the robustness of simulation predictions.

[0092] The implementation methods and advantages of the embodiments of this disclosure have been described above through multiple examples. The following is a complete simulation testing process embodiment based on the "Qwen3-30B-A3B wint model". This embodiment details how to build a scene from scratch, collect data, select fitting formulas, and finally verify the simulation accuracy.

[0093] I. Scene Modeling and Data Acquisition The benchmark testing environment was built based on the configuration parameters of the model under test, "Qwen3-30B-A3B (wint8 quantization version)". Specifically, the hardware environment used a single NVIDIA H800 GPU; the initial configuration of the inference engine included setting the maximum concurrency (max_num_seqs) to 50, configuring sufficient memory blocks (block_num) to support a 32k context length, and setting the quantization parameter to wint8 (Weight Int8).

[0094] After the environment was set up, an automated data acquisition strategy consisting of two sets of orthogonal experiments was implemented.

[0095] 1. Data acquisition experiment during the prefill phase (1) Pre-filled baseline time experiment In this experiment, the concurrency was strictly locked at 1 to eliminate interference from multi-task resource contention. The test program generated a series of requests with the number of input tokens increasing from 30 to 30,000, with an increment step of 500, and fixed the number of output tokens at 100 (ignoring the effect of decoding). The total time to complete the acquisition of all 60 sampling points in this set of experiments was approximately 5 minutes.

[0096] (2) Dynamic concurrent correction experiment In this experiment, a fixed standard test set (e.g., containing 100 requests) was selected to ensure that the total number of input tokens sent in each test remained consistent, in order to control the computational load. The standard test set was sent at different concurrency levels (e.g., 1, 2, 5, 10, 20, 50).

[0097] 2. Data Acquisition Experiment in the Decoding Phase In this experiment, the system incrementally increased the concurrency from 1 to 50 (i.e., max_num_seqs) in increments of 5, covering a total of 10 concurrency levels. For each concurrency level, different "input + output" token combinations (total length controlled at approximately 30k) were further traversed to simulate decoding performance under different context lengths. Some typical token combinations and measured latency data are shown in Table 2 below: Table 2

[0098] II. Function Fitting and Formula Selection 1. Fitting optimization in the pre-filling stage In the early pre-filling stage, it was found that if the concurrency number was directly incorporated as a variable into the main prediction formula, the model's prediction performance in the low-load range was not ideal (poor fit). To solve this problem, this embodiment adopts a "step-by-step fitting" strategy, that is, constructing a baseline prediction model based on the number of input tokens and a modified model based on the number of concurrency, and finally achieving high-precision prediction through linear superposition.

[0099] (1) Determine the first time-consuming item (benchmark value) based on the number of input tokens. Figure 5 This is a graph showing experimental data on the number of input tokens and the pre-filling time according to embodiments of this disclosure. For example... Figure 5 As shown in the chart, the horizontal axis represents the number of input tokens requested, and the vertical axis represents the corresponding prefill time, in milliseconds. Through analysis... Figure 5 It can be seen that, after removing concurrency interference (concurrency = 1), the pre-filling time exhibits a clear quadratic function characteristic. Therefore, the following quadratic polynomial is used to perform regression fitting on the data:

[0100] in, As the first time-consuming item, Enter the number of tokens.

[0101] The coefficient of determination R of the above fitting model 2 =0.9928, indicating that the quadratic function accurately describes the impact of input length on computation time.

[0102] (2) Determine the second time-consuming item (correction value) based on the number of concurrent requests. Figure 6 This is an experimental data graph showing the number of concurrent requests and the additional correction time according to embodiments of this disclosure. For example... Figure 6 As shown, the horizontal axis of this chart represents the current number of concurrent requests, and the vertical axis represents the additional adjusted time relative to the baseline time (time for 1 concurrent request), in milliseconds. Based on Figure 6 For the data in this disclosure, the following cubic polynomial is used to perform regression fitting on the concurrent correction data:

[0103] in, This is the second time-consuming item, and n is the number of concurrent processes.

[0104] The determination coefficient R of the above modified model 2 =0.99, accurately reproducing the non-linear performance degradation trend caused by resource contention as the number of concurrent users increases.

[0105] (3) Combining the above two fitting results, and introducing a random perturbation value based on the standard deviation of the difference ±2%. The final simulation prefilling time is calculated using the following formula:

[0106] By decoupling the "computation dimension" and the "congestion dimension" and fitting them separately, this disclosure successfully solves the problem of prediction distortion of a single model under complex loads, and significantly improves the robustness of the simulation system.

[0107] 2. Fitting optimization in the decoding stage In the initial modeling process of the decoding phase, a simplifying assumption was first attempted: treating the number of input tokens and the number of output tokens as single variables with equal weight (i.e., fitting the model using the total length of Input + Output). However, experimental results showed that this simplified model did not perform ideally.

[0108] Figure 7 This is a schematic diagram of the initial decoding time fitting according to an embodiment of this disclosure (treating input and output as the same variable). Figure 7 As shown, the fitting effect is acceptable when the concurrency is low (for example, on the far left, the red line overlaps with the blue area quite a bit). However, the fitting effect is poor when the concurrency is high (for example, in the middle area of ​​the figure, the red line overlaps with the blue area less), resulting in a poor fitting effect.

[0109] To address the aforementioned issues, this disclosure employs a "multivariate decoupled fitting" strategy. This strategy treats the number of input tokens and the number of output tokens as two independent dimensions and introduces an interaction term between them and the concurrency level to construct a high-order polynomial model. This disclosure uses a ternary quadratic polynomial of the following form to perform regression fitting on the data:

[0110] in, Decoding time Input the number of tokens. The number of output tokens is n, where n is the number of concurrent connections.

[0111] Figure 8 This is a schematic diagram comparing the optimized predicted decoding time with the actual time according to an embodiment of this disclosure. Figure 8 As shown in the chart, the horizontal axis represents the actual observed decoding time (Actual Values), and the vertical axis represents the predicted time calculated by the simulation model (Predicted Values). The red dashed line in the chart represents the ideal diagonal (i.e., the predicted value is exactly equal to the true value), and the blue dots represent each sample point in the test set.

[0112] As can be clearly seen from the graph, the vast majority of blue sample points are closely clustered around the red dashed line, exhibiting a very high linear correlation. The coefficient of determination R is marked in the graph. 2 =0.9913, indicating that the ternary quadratic polynomial model can explain 99.13% of the decoding time variation, and the fitting accuracy is extremely high.

[0113] Figure 9 This is a difference distribution diagram of the decoding time prediction model according to an embodiment of this disclosure. For example... Figure 9 As shown in the chart, the horizontal axis represents the decoding time of the model predictions (Predicted Values), and the vertical axis represents the differences, which are the differences between the actual values ​​and the predicted values. The red horizontal dashed line in the chart represents the zero error line (Difference=0).

[0114] As can be seen from the figure, the difference data points exhibit a uniform and random distribution around the zero error line (i.e., good homoscedasticity), and are mainly concentrated within a small error band of ±2ms. No obvious funnel-shaped or curved trend appears in the figure, indicating that this model is an unbiased estimation model with extremely high robustness.

[0115] 3. Model testing and accuracy verification To verify the prediction accuracy and robustness of the above simulation model, this disclosure conducts a full-link inference performance test on the "Qwen3-30B-A3B" model. By comparing the simulation prediction data with the real-world operating data, a quantitative evaluation is performed from two core dimensions: "Tron Delay (TTFT)" and "Tron Inter-packet Delay (TPOT)".

[0116] Figure 10 This is a bar chart comparing the simulated value and the actual value of the inference time for the first character according to embodiments of this disclosure. For example... Figure 10 As shown, the horizontal axis represents different statistical quantiles (including mean, weighted time, 50th quantile, 95th quantile, etc.), and the vertical axis represents time (unit: ms). Green bars represent simulated predicted values, and blue bars represent actual running values.

[0117] As can be seen from the figure, the simulated values ​​are almost identical to the actual values ​​for the "average value" and "median (50th percentile)" indicators, which best represent the overall system performance. The average TTFT in the real environment is about 82ms, and the simulated prediction value is also stable at around 82ms, with the relative error between the two controlled within 1%.

[0118] Figure 11 This is a bar chart comparing the simulated and actual average room delay values ​​according to embodiments of this disclosure. Figure 11As shown, the horizontal axis represents the statistical quantile, and the vertical axis represents the single-step decoding time (unit: ms). The red bars represent the simulated predicted values, and the blue bars represent the actual running values.

[0119] Comparing the data from each group reveals that the height difference between the simulated predicted value (red bar) and the actual value (blue bar) remains consistently within a small range. For example, in terms of the average value, the simulated value is approximately 24ms, while the actual value is approximately 22ms, with an absolute time difference of less than 3ms. Based on the comprehensive analysis of all location data, the average prediction error rate of the simulation model in the decoding stage is significantly lower than 5%.

[0120] As can be seen from the above, the simulation method provided in this disclosure, through pre-filled modeling of "benchmark + correction + perturbation" and decoding modeling of "multivariable decoupling", can achieve pixel-level restoration of real hardware performance (error <5% or <3ms) in complex unknown load scenarios with extremely low time cost (minute-level testing), which fully meets the needs of large-scale AI cluster capacity planning and performance evaluation.

[0121] According to embodiments of this disclosure, this disclosure also provides a large-scale model simulation testing apparatus 100, with reference to... Figure 12 It includes: Module 110 is used to acquire inference performance data of the large model to be tested; The determination module 120 is used to determine a first mapping relationship in the pre-filling stage and a second mapping relationship in the decoding stage based on inference performance data. The first mapping relationship includes at least the correspondence between pre-filling time and the number of input tokens, and the second mapping relationship includes at least the correspondence between decoding time and the number of concurrent tokens. The simulation test module 130 is used to perform simulation reasoning on the large model under test using the first mapping relationship and the second mapping relationship to obtain simulation test results.

[0122] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0123] Figure 13 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0124] like Figure 13 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0125] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0126] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the simulation testing method for large models. For example, in some embodiments, the simulation testing method for large models can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the simulation testing method for large models described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the simulation testing method for large models by any other suitable means (e.g., by means of firmware).

[0127] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0129] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0130] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0131] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0132] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0133] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0134] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A simulation testing method for a large model, comprising: Obtain inference performance data for the large model to be tested; Based on inference performance data, a first mapping relationship for the pre-filling stage and a second mapping relationship for the decoding stage are determined. The first mapping relationship includes at least the correspondence between pre-filling time and the number of input tokens, and the second mapping relationship includes at least the correspondence between decoding time and the number of concurrent tokens. The simulation reasoning of the large model under test is carried out using the first and second mapping relationships to obtain the simulation test results.

2. The method according to claim 1, wherein the pre-filling time is a power function of the number of input tokens.

3. The method according to claim 1, wherein the first mapping relationship further includes: The functional relationship between pre-filling time and the number of concurrent requests.

4. The method according to claim 1, wherein the pre-filling time further includes: Random perturbation value; The range of the random perturbation value is determined based on the standard deviation between the inference performance data and the first mapping relationship.

5. The method according to any one of claims 1-4, wherein determining the pre-filling time using the first mapping relationship includes: The first time consumption item is determined based on the correspondence between the pre-filling time and the number of input tokens, and the number of requested input tokens. The second time consumption item is determined based on the correspondence between the pre-filling time and the number of concurrent requests; The pre-filling time is determined based on the first time-consuming item, the second time-consuming item, and the random disturbance value.

6. The method according to claim 1, wherein the second mapping relationship further comprises: The functional relationship between decoding time and the number of input tokens; and / or The functional relationship between decoding time and the number of output tokens.

7. The method according to claim 6, wherein determining the decoding time using the second mapping relationship includes: Calculate the first-order and second-order time components corresponding to the number of input tokens, the number of output tokens, and the number of concurrent transactions, respectively. Calculate the cross-time components corresponding to the product of each pair of the number of input tokens, the number of output tokens, and the number of concurrent requests; The decoding time is determined based on the first-order time consumption component, the second-order time consumption component, the cross-time consumption component, and the preset constant term.

8. The method according to claim 1, wherein obtaining the inference performance data of the large model to be tested includes: Based on the configuration parameters of the large model to be tested, an inference test environment is built; The number of input tokens is incremented according to a preset step size to obtain the pre-filled time data corresponding to different numbers of input tokens. For each concurrency level, change the combination of the number of input tokens and the number of output tokens to obtain decoding time data under different concurrency levels and different token combinations.

9. A simulation testing device for a large model, comprising: The acquisition module is used to acquire inference performance data of the large model to be tested; The determination module is used to determine a first mapping relationship in the pre-filling stage and a second mapping relationship in the decoding stage based on inference performance data. The first mapping relationship includes at least the correspondence between pre-filling time and the number of input tokens, and the second mapping relationship includes at least the correspondence between decoding time and the number of concurrent tokens. The simulation test module is used to perform simulation reasoning on the large model under test using the first and second mapping relationships to obtain simulation test results.

10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images