VLLM performance evaluation method, apparatus and device, medium and product

Abstract

The invention discloses a vLLM performance evaluation method and device, equipment, a medium and a product, and relates to the technical field of artificial intelligence, and the method comprises the steps: based on a target model, creating a target vLLM engine instance, and determining a target batch size corresponding to the target vLLM engine instance; wherein the target vLLM engine instance configures a target tensor parallel scale, a pre-allocated video memory resource upper limit, an input length and an output length; under the condition that the requests arrive uniformly, reasoning is carried out based on the target vLLM engine instance and the target batch size to obtain virtual state data corresponding to each request; constructing real state data based on the virtual state data; based on the real state data, configuring and reasoning the target vLLM engine instance to obtain target reasoning time consumption; and determining a performance evaluation result of the target vLLM engine instance based on the target reasoning time consumption. According to the method, performance evaluation can be efficiently realized.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a vLLM performance evaluation method, apparatus, device, medium and product. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have moved from the laboratory to large-scale commercial applications, covering multiple scenarios such as intelligent customer service, content generation, and code assistance. As a key support for the deployment of large language models, the performance of the inference system directly determines service response efficiency, deployment costs, and user experience. vLLM (Very Large Language Model Serving Engine), with its innovative Paged Attention memory management mechanism, continuous batching, and efficient CUDA (Compute Unified Device Architecture) implementation, outperforms traditional inference solutions in terms of throughput and latency optimization. It supports dense models and hybrid expert models with billions to hundreds of billions of parameters and is compatible with various quantization strategies such as FP16, BF16, and INT8. It has become the preferred open-source engine for enterprises deploying large language models, building a fully functional and vibrant inference service foundation.

[0003] Currently, the performance evaluation of vLLM inference services generally adopts the traditional approach of "deploying the service first, then conducting full load testing." This approach primarily aims to obtain performance data that closely resembles actual production conditions by recreating the real deployment environment and request traffic, providing a basis for hardware selection and parameter tuning decisions. While this traditional performance evaluation method can reproduce the real operating state to some extent, its inherent shortcomings become increasingly apparent under the demands of large-scale deployments. Issues such as lengthy testing cycles, strong resource exclusivity, and low efficiency make it unable to achieve efficient performance evaluation.

[0004] Therefore, there is an urgent need for a method that can efficiently perform performance evaluation. Summary of the Invention

[0005] This application provides a vLLM performance evaluation method, apparatus, device, medium, and product that can efficiently achieve performance evaluation.

[0006] To achieve the above objectives, this application adopts the following technical solution: Firstly, this application provides a vLLM performance evaluation method, including: Based on the target model, a target vLLM engine instance is created and the target batch size corresponding to the target vLLM engine instance is determined; wherein, the target vLLM engine instance is configured with the target tensor parallel scale, the upper limit of pre-allocated video memory resources, the input length and the output length; Under the condition that requests arrive evenly, inference is performed based on the target vLLM engine instance and the target batch size to obtain the virtual state data corresponding to each request; Construct real state data based on virtual state data; Based on real-world data, the target vLLM engine instance is configured and inferred to obtain the target inference time. Based on the target inference time, determine the performance evaluation results of the target vLLM engine instance.

[0007] In some possible implementations, assuming uniform request arrival, inference is performed based on the target vLLM engine instance and the target batch size to obtain virtual state data corresponding to each request, including: Under the condition that requests arrive evenly, a full lifecycle simulation is performed based on the target vLLM engine instance and the target batch size to determine the processing stage, the number of basic processing units, and the key-value cache occupancy status of each request. Based on the processing stage of each request, the number of processed request units, and the key-value cache occupancy status, determine the virtual state data corresponding to each request.

[0008] In some possible implementations, real state data is constructed based on virtual state data, including: With all requests in a running state, inference and reproduction are performed based on the virtual state data to obtain the real state data.

[0009] In some possible implementations, the target vLLM engine instance is configured and inferred based on real-world state data to obtain the target inference time, including: Based on real-world data, the target vLLM engine instance is configured and inferred, and the stage time corresponding to each stage is recorded; the stage time includes the pre-filling stage time, the decoding stage time, and the total time. Based on the time consumed in each stage, the target reasoning time is obtained.

[0010] In some possible implementations, the performance evaluation result of the target vLLM engine instance is determined based on the target inference time, including: Based on the decoding stage time and output length, determine the time consumption of the first request unit corresponding to the decoding stage; The target number of requests to be processed is determined based on the target batch size and total time. Based on the time consumed by the first request unit and the target number of requests processed, the performance evaluation result of the target vLLM engine instance is determined.

[0011] In some possible implementations, a target vLLM engine instance is created based on the target model, including: The target vLLM engine instance is initialized using virtual loading; where virtual loading skips the actual loading of the target model weights and directly constructs the computation graph and scheduling logic corresponding to the target vLLM engine instance.

[0012] Secondly, this application provides a vLLM performance evaluation apparatus, comprising: The instance creation module is used to create a target vLLM engine instance based on the target model and determine the target batch size corresponding to the target vLLM engine instance; wherein, the target vLLM engine instance is configured with the target tensor parallel scale, the upper limit of pre-allocated video memory resources, the input length and the output length; The inference module is used to perform inference based on the target vLLM engine instance and the target batch size to obtain virtual state data corresponding to each request when requests arrive evenly. The data construction module is used to construct real state data based on virtual state data; The configuration module is used to configure and infer the target vLLM engine instance based on real state data, and obtain the target inference time. The evaluation module is used to determine the performance evaluation results of the target vLLM engine instance based on the target inference time.

[0013] Thirdly, this application provides a computing device, including a memory and a processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of the first aspects.

[0014] Fourthly, this application provides a computer-readable storage medium for storing a computer program for performing the method as described in any one of the first aspects.

[0015] Fifthly, this application provides a computer program product comprising one or more computer instructions, wherein when the computer instructions are executed by a computer, the computer performs the method as described in any one of the first aspects.

[0016] As can be seen from the above technical solution, this application has at least the following beneficial effects: In this application, a target vLLM engine instance is created based on the target model, and the target batch size is determined. By using virtual loading, the actual loading of the target model weights is skipped, and an engine instance including a complete computation graph and scheduling logic is directly constructed. This significantly shortens the engine initialization time and avoids the substantial computational and time costs associated with model loading in traditional evaluations. Simultaneously, by configuring key parameters such as the target tensor parallel scale and pre-allocating the upper limit of GPU memory resources, the operating environment of the engine instance is ensured to be highly matched with the real deployment scenario, laying the foundation for the accuracy of subsequent evaluation results. The precise determination of the target batch size ensures that the engine operates within a reasonable load boundary during the evaluation process and avoids issues such as GPU memory overflow, enabling subsequent performance testing to be conducted in a stable and controllable environment, effectively improving the smoothness and reliability of the evaluation process.

[0017] Virtual state data is obtained through inference under a scenario of uniform request arrival. The entire request processing flow is simulated through a complete lifecycle simulation, eliminating the need for actual hardware to perform inference. This avoids the resource exclusivity problem of traditional load testing and can reproduce the processing states of different requests in a short time. By accurately capturing the processing stage, number of processed units, and KVCache occupancy status of each request, the generated virtual state data comprehensively and realistically reflects the engine's operating logic, providing high-quality foundational material for constructing real state data. Furthermore, the uniform request arrival scenario eliminates state fluctuations caused by sudden requests, enhancing the stability and representativeness of the virtual state data. This ensures that subsequent evaluation results are not affected by abnormal loads, improving the accuracy of performance evaluation.

[0018] By constructing real-world state data based on virtual state data and marking all requests as running states for inference reproduction, the target vLLM engine instance is directly brought into full-load operation, skipping the request queuing process in real-world scenarios and significantly shortening the evaluation cycle. The constructed real-world state data is entirely based on key information from the initial virtual state data, ensuring consistency with the real-world scenario while avoiding state noise caused by the dynamic arrival of real requests, making the engine's operating state more controllable and repeatable. This process achieves a seamless transition from virtual simulation to real-world operation, providing a stable and reliable environment for subsequent accurate measurement of inference time, effectively improving the accuracy and repeatability of time consumption data.

[0019] The engine is configured based on real-world data to obtain the target inference time. Precise engine configuration using this data ensures the engine's operational status perfectly matches the actual deployment, making the inference time measurement highly valuable. By breaking down the time into the record pre-filling stage, decoding stage, and total time, comprehensive time data is obtained, providing a foundation for deriving detailed performance metrics. Furthermore, averaging multiple inference runs effectively reduces random errors from single measurements, enhancing the stability of the target inference time and keeping the standard deviation low. This ensures the accuracy of the final performance evaluation results and avoids decision-making biases caused by inaccurate time data.

[0020] Performance evaluation results are determined based on the target inference time. Standardized calculations derive metrics such as first-token latency, inter-token latency, and requests processed per second, transforming abstract time-consuming data into intuitive and quantifiable performance evaluation results. This allows users to quickly understand the engine's operational efficiency. The evaluation results cover both latency metrics related to user experience and throughput metrics related to service capacity, comprehensively meeting users' diverse performance needs during deployment selection. Users can intuitively compare the adaptability of different configuration schemes based on these metrics, quickly making optimal deployment decisions. This completely breaks the traditional long chain of evaluation—"configuration, testing, and decision-making"—improving the agility of deployment decisions and reducing the technical threshold and resource costs of large-scale model service optimization. Ultimately, it enables efficient performance evaluation.

[0021] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0022] Figure 1 An application environment diagram for a vLLM performance evaluation method provided in this application embodiment; Figure 2 A flowchart illustrating a vLLM performance evaluation method provided in this application embodiment; Figure 3A structural diagram of a vLLM performance evaluation device provided in an embodiment of this application; Figure 4 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0023] The terms "first," "second," and "third," etc., used in this application specification and accompanying drawings are used to distinguish different objects, not to limit a specific order.

[0024] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0025] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first: vLLM, or Very Large Language Model Serving Engine, is a mainstream open-source inference engine in the industry. With its PagedAttention memory management mechanism, continuous batching, and efficient CUDA core implementation, it excels in throughput and latency optimization. It supports various model structures such as Dense Model and Hybrid Expert Model (MoE) and is compatible with various quantization strategies such as FP16 and INT8. It is an important foundation for the large-scale deployment of large language models (LLM).

[0026] The prefill and decode phases are two crucial stages in vLLM's request processing. The prefill phase computes the attention of all tokens in the input Prompt, generates the first output token, and builds the KVCache. The decode phase generates subsequent outputs (the second and subsequent tokens) token by token based on the built KVCache. The time taken in these two phases directly determines the latency performance of the inference service.

[0027] As large language models are deployed on a large scale, performance evaluation of vLLM inference services has become a crucial decision-making basis for hardware procurement, architecture design, and deployment selection. However, traditional evaluation methods still follow the path of "starting the service first, then load testing," which has significant efficiency bottlenecks. The testing cycle is lengthy: it requires a complete compilation and startup of the vLLM service, loading the model into the GPU memory, and then conducting full load testing based on tens of thousands of real dialogue data. A single evaluation takes several hours to several days, which seriously prolongs the decision-making cycle. Strong resource exclusivity: During the stress test, the GPU device is exclusively used throughout the process. In scenarios with tight computing power, it is impossible to conduct multiple sets of configuration experiments in parallel, which leads to the compression of the evaluation matrix and makes it difficult to discover potential better configurations. Deployment selection is inefficient: Users need to go through different tp-size configurations and perform complete load tests one by one to determine the optimal performance solution that fits their own input and output length requirements. The long chain of "configuration-testing-decision" greatly delays the model deployment process.

[0028] The main reason for these problems is that traditional methods rely on real service deployment and full request load testing, which fail to break away from the inherent logic of "actual operation and data collection". They cannot avoid the time and resource consumption caused by model loading, request accumulation and other processes, and can no longer meet the performance evaluation needs of scenarios such as rapid deployment and continuous integration.

[0029] In view of this, this application provides a vLLM performance evaluation method. In this method, by virtually loading and initializing the vLLM engine instance, simulating the request lifecycle to generate virtual state data, constructing the real running state, sampling the critical path time and extrapolating the performance indicators, the method can accurately output key performance data such as throughput and latency under different tensor parallel scales within minutes without fully deploying the inference service and conducting full load testing, thus efficiently supporting model deployment and selection decisions.

[0030] To make the technical solution of this application clearer and easier to understand, the application scenarios of the technical solution of this application are described below with reference to the accompanying drawings. Figure 1 As shown in the figure, this figure is a schematic diagram of an application scenario provided by an embodiment of this application.

[0031] In this application scenario, the terminal is operated by R&D or operations personnel. They input target model information (such as model structure, parameter size, and quantization strategy), business scenario configuration (such as input / output length distribution), and the range of tensor parallel scale (tp-size) to be evaluated through a visual interface. The terminal packages these configuration parameters and transmits them to the backend server via a communication network, triggering a performance evaluation task. Upon receiving the request, the server executes the evaluation method of this application based on its built-in vLLM engine framework: first, it quickly creates a target vLLM engine instance and calculates the target batch size using virtual loading; then, it generates virtual state data through request lifecycle simulation; after constructing a real full-load operating state, it samples the critical path latency; finally, it derives performance indicators such as first-token latency, inter-token latency, and requests processed per second. After the evaluation is completed, the server organizes the performance data corresponding to each tp-size into structured reports (such as performance comparison tables and trend charts) and feeds them back to the terminal. The terminal interface displays the complete evaluation results, allowing R&D personnel to intuitively compare the performance differences and cost adaptability of different configurations and quickly determine the optimal deployment solution. The entire interaction process does not require the terminal to participate in complex calculations; it only needs to complete parameter input and result viewing. The server then handles the evaluation logic, achieving an efficient and lightweight performance evaluation closed loop.

[0032] To make the technical solution of this application clearer and easier to understand, the following describes a vLLM performance evaluation method provided by an embodiment of this application, in conjunction with the above application scenarios. Figure 2 As shown, this figure is a flowchart of a vLLM performance evaluation method provided in an embodiment of this application. The vLLM performance evaluation method includes: S201. Based on the target model, create a target vLLM engine instance and determine the target batch size corresponding to the target vLLM engine instance.

[0033] The target vLLM engine instance is configured with target tensor parallel scale, pre-allocated video memory resource limit, input length, and output length.

[0034] One possible approach is to initialize the target vLLM engine instance using virtual loading; wherein, the virtual loading method skips the actual loading of the target model weights and directly constructs the computation graph and scheduling logic corresponding to the target vLLM engine instance.

[0035] The target model refers to the large language model (LLM) to be evaluated for performance. Its parameter size, model structure (such as dense model or hybrid expert model MoE), and quantization strategy (such as FP16, BF16, INT8, INT4, etc.) are clearly defined, and it is the object of performance evaluation of vLLM inference service.

[0036] The target vLLM engine instance is an engine instance created based on the vLLM (Very Large Language Model Large Scale Service Engine) framework, specifically designed for evaluating the performance of the target model. It includes a complete computation graph and scheduling logic, which can simulate the running mechanism of the target model in the inference service, and is configured with specific running parameters to match the evaluation scenario.

[0037] The target tensor parallel size (tp-size) is the configuration of the number of GPUs used to split model weights and computation tasks in the vLLM engine. It directly affects the parallel computing efficiency, memory usage and communication overhead of model inference, and is a configuration parameter that needs to be carefully examined in performance evaluation.

[0038] The pre-allocated video memory resource limit is the threshold of GPU video memory capacity reserved for the target vLLM engine instance. It is used to limit the amount of video memory used during engine operation, avoid video memory overflow, and determine the upper limit of request concurrency that the engine can handle.

[0039] Input length is the number of tokens (i.e., the length of prompt tokens) corresponding to the input prompt text received by the target model during inference, and it is one of the key factors affecting inference latency and memory usage.

[0040] Output length is the number of tokens (i.e., output token length) corresponding to the output content that needs to be generated during the inference of the target model. It is directly related to the computation time of the decoding stage and is an important basis for the calculation of performance indicators (such as inter-token latency).

[0041] The target batch size (max_batch_size) is the maximum number of requests that a target vLLM engine instance can handle simultaneously under full load. It is determined by the pre-allocated video memory resource limit and the single request KV Cache usage, and is an important parameter for measuring the throughput of the inference service.

[0042] Virtual loading is a lightweight initialization method supported by the vLLM engine. By enabling the --load-format="dummy" parameter, the actual loading process of the target model weight data is skipped, and only the computation graph and scheduling logic required for engine operation are quickly built, which greatly shortens the engine instance creation time.

[0043] The computation graph abstracts the reasoning process of the target model (such as attention computation, layer normalization, linear transformation and other operations) into ordered computation nodes and data flow, and is the foundation for the vLLM engine to perform parallel computation and task scheduling.

[0044] The scheduling logic is a set of rules in the vLLM engine used to manage the lifecycle of requests, allocate computing resources, and schedule the execution order of tasks. It includes important mechanisms such as continuous batching and PagedAttention memory management, which directly affect the throughput and latency performance of the inference service.

[0045] For example, in the performance evaluation process of vLLM inference services, the target model to be evaluated must first be clearly defined. Based on this model, a target vLLM engine instance dedicated to performance testing is created, and the maximum number of concurrent requests that this engine instance can handle (i.e., the target batch size) is determined. To ensure that the evaluation scenario is consistent with the actual deployment requirements, the target vLLM engine instance needs to be pre-configured with key operating parameters: including the target tensor parallel scale for splitting computation tasks, the upper limit of pre-allocated GPU memory resources to limit GPU memory usage, and the input length (number of tokens in the input prompt) and output length (number of tokens in the generated result) that match the real business scenario. The initialization of the target vLLM engine instance adopts a virtual loading method. This method does not require actually loading the massive weight data of the target model; it only requires constructing a complete computation graph and scheduling logic to complete the engine instance creation. This ensures the consistency between the engine's operating mechanism and the real deployment scenario, and significantly reduces the instance initialization time, laying the foundation for subsequent rapid performance evaluation. The determination of the target batch size depends on the calculation of the pre-allocated video memory resource limit and the KV cache usage per request, ensuring that the engine will not experience video memory overflow when running at full load, while maximizing the utilization of hardware resources.

[0046] For example, the `--load-format="dummy"` option in vLLM instantaneously initializes the `LLM_Engine`. This mechanism bypasses weight loading and directly creates an engine instance containing a complete computation graph and scheduling logic. By accessing the engine's `vllm_config.cache_config`, the pre-allocated `num_gpu_blocks` and `block_size` are obtained. Combined with the input / output length configuration, the KV cache usage for a single request is accurately calculated, thus deriving the maximum batch-size that can be handled under the current configuration. This value is automatically constrained by the default scheduling parameter `max_num_seqs=256`, ensuring that the simulation is performed within the real-world constraints.

[0047] S202. Under the condition that the requests arrive evenly, inference is performed based on the target vLLM engine instance and the target batch size to obtain the virtual state data corresponding to each request.

[0048] One possible approach is to perform a full lifecycle simulation based on the target vLLM engine instance and the target batch size, under the condition that requests arrive uniformly, to determine the processing stage, number of basic processing units, and key-value cache occupancy status of each request; and to determine the virtual state data corresponding to each request based on the processing stage, number of processed request units, and key-value cache occupancy status of each request.

[0049] Among them, uniform request arrival means that user requests for the inference service are received in an orderly manner at fixed time intervals, without concentrated bursts or long periods of idle time, simulating a relatively stable load scenario in real business, and ensuring the stability and representativeness of the simulation results.

[0050] Full lifecycle simulation is a lightweight discrete event simulator that simulates the entire process of each request, from accessing the engine, entering the processing queue, performing inference calculations, to completing the output. It does not rely on real hardware execution and only restores the request processing process through logical modeling.

[0051] The processing phase consists of two important stages in the vLLM engine's request handling: the prefill phase and the decode phase. The prefill phase calculates the attention of all tokens in the input Prompt at once and generates the first output token, while simultaneously building a key-value cache (KV Cache). The decode phase generates subsequent outputs token by token based on the built KV Cache.

[0052] The number of basic units processed, i.e. the number of tokens processed during the inference process, is an important indicator for measuring the progress of request processing. The pre-filling stage corresponds to the number of tokens in the input prompt, and the decoding stage corresponds to the number of output tokens generated.

[0053] Key-value cache occupancy status refers to the resource usage of the KV cache during request processing, including the number of GPU cache blocks used (num_gpu_blocks) and the size of the cache blocks (block_size), which directly affects the number of concurrent requests the engine can handle simultaneously.

[0054] Virtual state data is an abstract set of data obtained through simulation that can characterize the request processing state. It includes key information such as the request processing stage, the number of processed tokens, and the KV cache usage, and is used to construct the real running state in the future.

[0055] For example, in the vLLM performance evaluation process, it is necessary to simulate a stable load scenario where requests arrive evenly in real business scenarios. Based on the created target vLLM engine instance and the determined target batch size (maximum number of concurrent requests), the inference process simulation is carried out to obtain virtual state data corresponding to each request. In specific implementation, a lightweight discrete event simulator is first used to simulate the complete lifecycle of a request. Under the reasonable premise of non-simultaneous arrival of requests and no memory preemption, the entire process of each request from access to completion is simulated step by step in time, accurately tracking and determining the current processing stage (pre-filling stage or decoding stage), the number of processed tokens (number of basic processing units), and the occupancy status of the KV Cache for each request. Subsequently, these key information reflecting the real-time processing status of requests are integrated to form virtual state data that can comprehensively characterize the running state of requests, laying the foundation for constructing a real running environment and accurately measuring inference time. This process can efficiently reproduce the processing logic and state changes of requests in the engine without starting a real inference service or consuming GPU resources.

[0056] For example, a lightweight discrete event simulator can be built to model the complete lifecycle of a request from inception to completion. Based on the premise of "requests arriving at different times" and "no memory preemption," the simulator progresses step by step, tracking the prefill / decode stage of each request, the number of processed tokens, and the KV Cache occupancy status, thereby statistically analyzing the global state distribution of the engine. Key patterns are extracted: fully decoded states of different lengths and complete prefill states are used for subsequent sampling. Furthermore, the get_requests method, consistent with the official benchmark, can be called to generate virtual SequenceGroup data, with prompt tokens constructed based on random sampling.

[0057] S203. Construct real state data based on virtual state data.

[0058] One possible approach is to perform reasoning and reproduction based on virtual state data, given that all requests are in a running state, to obtain the real state data.

[0059] Among them, real state data is a set of state data that can accurately reproduce the real running scenario of the vLLM engine, including important information that is consistent with the actual inference process, such as request running status, processing progress, and resource consumption. It is the basis for subsequent accurate measurement of inference time.

[0060] The RUNNING state is an important processing state for requests in the vLLM engine. It means that the request has entered the execution queue and is being scheduled and processed by the engine, rather than being in a pending state in the waiting queue or a terminated state that has been completed.

[0061] Inference and reproduction are based on important information from virtual state data. They recreate the specific scenarios and states of request processing in the target vLLM engine instance, allowing the engine to enter a load state consistent with actual operation without waiting for real requests to accumulate.

[0062] For example, in the vLLM performance evaluation process, constructing realistic state data is a crucial step connecting virtual simulation and real performance measurement. This primarily involves recreating a state environment consistent with the engine's actual operation based on the previously obtained virtual state data. Specifically, all pending requests are first uniformly set to the RUNNING state, simulating a scenario where the engine is running at full load. This bypasses the process in real inference services where requests gradually enter the execution queue from the waiting queue, directly placing the engine in a peak load state for efficient processing. Subsequently, based on information recorded in the virtual state data, such as the processing stages of each request, the number of processed tokens, and the KV cache usage, inference is reproduced in the target vLLM engine instance. This accurately recreates the specific state and resource consumption of each request during real operation, ultimately obtaining realistic state data that characterizes the engine's actual operating scenario. This process ensures consistency between the state data and the actual inference scenario while avoiding the time consumption and state fluctuations caused by the dynamic arrival of real requests, providing a stable and reliable foundation for subsequent accurate measurement of inference time.

[0063] For example, to achieve full load, the scheduler logic is bypassed, and all requests are marked as RUNNING and injected into the execution queue. By adjusting the SchedulerMetadata, specific state patterns obtained from the simulation (such as the position of the prefill step and the progress of each request in the batch) can be accurately reproduced, so that the engine can present the target running state in a single execution without waiting for the accumulation of real requests.

[0064] S204. Based on real-state data, configure and infer the target vLLM engine instance to obtain the target inference time.

[0065] One possible approach is to configure and infer a target vLLM engine instance based on real-world state data, and record the stage time corresponding to each stage; wherein, the stage time includes the pre-filling stage time, the decoding stage time, and the total time; and obtain the target inference time based on the stage time corresponding to each stage.

[0066] For example, based on the decoding phase time and output length, the time of the first request unit corresponding to the decoding phase is determined; based on the target batch size and total time, the target number of processing requests is determined; based on the first request unit time and the target number of processing requests, the performance evaluation result of the target vLLM engine instance is determined.

[0067] Among them, the target inference time refers to the total time cost for the target vLLM engine instance to complete the inference task in a real running state. It is the basic data for computing performance evaluation indicators, including key time dimension information of the entire inference process.

[0068] Stage time is time data broken down into important processes of request processing by the vLLM engine. Specifically, it includes pre-filling stage time, decoding stage time, and total time, which correspond to different key stages of the inference process.

[0069] The prefill phase time is the total time the engine spends in the prefill phase. This phase requires calculating the attention of all tokens in the input prompt, generating the first output token, and building the KV Cache all at once. It is the initial step in the inference process.

[0070] The decoding stage time is the total time the engine spends in the decoding stage. This stage is based on the KV Cache built in the pre-filling stage and generates subsequent outputs (the second and subsequent tokens) token by token, which directly affects the generation efficiency of the output content.

[0071] Total time is the total time it takes for the engine to complete a single reasoning process, which is the sum of the time spent in the pre-filling stage and the time spent in the decoding stage, reflecting the overall time cost of the reasoning task.

[0072] The first request unit time is the unit token time in the decoding phase, which is an indicator of the efficiency of the decoding phase. It is calculated by the ratio of the decoding phase time to the output length.

[0073] The target number of requests to be processed is the total number of requests that the target vLLM engine instance can handle within the time range corresponding to the total time consumption, which reflects the engine's concurrent processing capability.

[0074] For example, based on the previously constructed real-state data, the target vLLM engine instance is brought into a running state consistent with the actual deployment and performs inference, ultimately obtaining the accurate target inference time. In specific implementation, firstly, based on the information recorded in the real-state data, such as the request processing stage and KV Cache occupancy status, the target vLLM engine instance is precisely configured to ensure that the engine's running environment is completely matched with the real business scenario; then, the inference process is started, and the time of each stage after being broken down is recorded synchronously, including the pre-filling stage time (the time to process the input Prompt and build the KV Cache), the decoding stage time (the time to generate the output per token), and the total time covering the entire process. Based on the time consumed in these stages, the target inference time can be directly summarized, and important performance parameters can be further derived: the time consumed in the first request unit (unit token generation time) corresponding to the decoding stage can be calculated by the ratio of the decoding stage time to the output length; the number of target processing requests (the total number of requests that can be processed per unit time) can be determined by the ratio of the target batch size (maximum number of concurrent requests of the engine) to the total time consumed; finally, by combining the time consumed in the first request unit and the target number of processing requests, the performance evaluation results of the target vLLM engine instance can be comprehensively and accurately determined, providing data support for deployment selection.

[0075] For example, after placing the engine in a specific state configuration, perform forward propagation N times (e.g., N=10), including warm-up rounds, record the time taken for each step, and take the arithmetic average to obtain a stable latency benchmark. State injection avoids dynamic arrival noise, has high measurement repeatability, and the standard deviation is usually controlled within 5%. This step replaces tens of thousands of request traversals with precise sampling, compressing the time measurement to dozens of core calls.

[0076] S205. Based on the target inference time, determine the performance evaluation result of the target vLLM engine instance.

[0077] Among them, the performance evaluation results refer to a set of important indicators that can comprehensively reflect the running efficiency of the target vLLM engine instance, including parameters such as Time To First Token (TTFT), Time Per Output Token (TPOT), and Requests Per Second (RPS). These are important bases for judging whether the engine meets business requirements under a specific configuration.

[0078] First Token Delay (TTFT) is the total time from requesting access to the engine to generating the first output token. It corresponds to the cumulative time spent in the pre-filling phase and directly affects the user's initial response experience.

[0079] Inter-Token Latency (TPOT) is the average time taken to generate a single output token during the decoding phase. It reflects the efficiency of subsequent token generation and determines the smoothness in long text output scenarios.

[0080] Requests per second (RPS) is the total number of requests that a target vLLM engine instance can process per unit of time. It is an important indicator for measuring engine throughput and is directly related to deployment costs and service capacity.

[0081] For example, based on the target inference time obtained in the early stage (including the time spent in the pre-filling stage, the time spent in the decoding stage, and the total time spent), a performance evaluation result that can quantify the engine's operating efficiency is derived through standardized calculations. Specifically, firstly, the cumulative time spent in the pre-filling stage is used as the first token latency (TTFT), which intuitively reflects the waiting time from when the user initiates a request to when they receive the first response; then, the ratio of the decoding stage time spent to the output length is used to calculate the token-to-token latency (TPOT), reflecting the generation efficiency of each subsequent output token; finally, the number of requests processed per second (RPS) is derived by combining the target batch size (the engine's maximum concurrent requests) and the total time spent, quantifying the engine's throughput level. These metrics together constitute the performance evaluation result, which includes latency metrics related to user experience as well as throughput metrics related to service capacity. Based on these results, users can intuitively judge the suitability of current configurations such as tensor parallel scale (tp-size) by combining their own business's input and output length distribution and performance requirements, and thus quickly make optimal deployment decisions.

[0082] For example, multiple target vLLM engine instances can be generated for different parameter combinations—specifically covering various combinations of different target models (including parameter size, model structure, and quantization strategy differences), different target tensor parallel scales (tp-size), different input lengths (number of tokens in the input prompt), and different output lengths (number of tokens in the generated result). The performance evaluation process of this application is executed for each target vLLM engine instance, obtaining performance metrics such as First Token Latency (TTFT), Inter-Token Latency (TPOT), and Requests Per Second (RPS) for each instance. Furthermore, the parameter configurations of all instances are integrated with the corresponding performance evaluation results to generate a structured performance evaluation summary table, clearly presenting the performance differences under different configuration schemes. Users can directly and quickly filter out the optimal deployment scheme in terms of performance and cost based on their own business's input / output length distribution, concurrent request volume requirements, and cost budget by looking up the table, significantly improving decision-making efficiency.

[0083] Based on the above, the vLLM performance evaluation method creates a target vLLM engine instance based on the target model and determines the target batch size. By using virtual loading, it skips the actual loading of the target model weights and directly constructs an engine instance including a complete computation graph and scheduling logic. This significantly shortens the engine initialization time and avoids the substantial computational and time costs associated with model loading in traditional evaluations. Simultaneously, by configuring key parameters such as the target tensor parallel scale and pre-allocating the upper limit of GPU memory resources, it ensures that the engine instance's operating environment highly matches the real deployment scenario, laying the foundation for the accuracy of subsequent evaluation results. The precise determination of the target batch size ensures that the engine operates within reasonable load boundaries during the evaluation process and avoids issues such as GPU memory overflow, enabling subsequent performance testing to be conducted in a stable and controllable environment, effectively improving the smoothness and reliability of the evaluation process.

[0084] Virtual state data is obtained through inference under a scenario of uniform request arrival. The entire request processing flow is simulated through a complete lifecycle simulation, eliminating the need for actual hardware to perform inference. This avoids the resource exclusivity problem of traditional load testing and can reproduce the processing states of different requests in a short time. By accurately capturing the processing stage, number of processed units, and KVCache occupancy status of each request, the generated virtual state data comprehensively and realistically reflects the engine's operating logic, providing high-quality foundational material for constructing real state data. Furthermore, the uniform request arrival scenario eliminates state fluctuations caused by sudden requests, enhancing the stability and representativeness of the virtual state data. This ensures that subsequent evaluation results are not affected by abnormal loads, improving the accuracy of performance evaluation.

[0085] By constructing real-world state data based on virtual state data and marking all requests as running states for inference reproduction, the target vLLM engine instance is directly brought into full-load operation, skipping the request queuing process in real-world scenarios and significantly shortening the evaluation cycle. The constructed real-world state data is entirely based on key information from the initial virtual state data, ensuring consistency with the real-world scenario while avoiding state noise caused by the dynamic arrival of real requests, making the engine's operating state more controllable and repeatable. This process achieves a seamless transition from virtual simulation to real-world operation, providing a stable and reliable environment for subsequent accurate measurement of inference time, effectively improving the accuracy and repeatability of time consumption data.

[0086] The engine is configured based on real-world data to obtain the target inference time. Precise engine configuration using this data ensures the engine's operational status perfectly matches the actual deployment, making the inference time measurement highly valuable. By breaking down the time into the record pre-filling stage, decoding stage, and total time, comprehensive time data is obtained, providing a foundation for deriving detailed performance metrics. Furthermore, averaging multiple inference runs effectively reduces random errors from single measurements, enhancing the stability of the target inference time and keeping the standard deviation low. This ensures the accuracy of the final performance evaluation results and avoids decision-making biases caused by inaccurate time data.

[0087] Performance evaluation results are determined based on the target inference time. Standardized calculations derive metrics such as first-token latency, inter-token latency, and requests processed per second, transforming abstract time-consuming data into intuitive and quantifiable performance evaluation results. This allows users to quickly understand the engine's operational efficiency. The evaluation results cover both latency metrics related to user experience and throughput metrics related to service capacity, comprehensively meeting users' diverse performance needs during deployment selection. Users can intuitively compare the adaptability of different configuration schemes based on these metrics, quickly making optimal deployment decisions. This completely breaks the traditional long chain of evaluation—"configuration, testing, and decision-making"—improving the agility of deployment decisions and reducing the technical threshold and resource costs of large-scale model service optimization. Ultimately, it enables efficient performance evaluation.

[0088] The above text combined Figures 1 to 2 The vLLM performance evaluation method provided in the embodiments of this application has been described in detail. The apparatus and equipment provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0089] This application also provides a vLLM performance evaluation device, such as... Figure 3 As shown in the figure, this is a schematic diagram of a vLLM performance evaluation device provided in an embodiment of this application. The device includes: The instance creation module 301 is used to create a target vLLM engine instance based on the target model and determine the target batch size corresponding to the target vLLM engine instance; wherein, the target vLLM engine instance is configured with the target tensor parallel scale, the upper limit of pre-allocated video memory resources, the input length and the output length. Inference module 302 is used to perform inference based on the target vLLM engine instance and the target batch size when the requests arrive evenly, and obtain the virtual state data corresponding to each request. The data construction module 303 is used to construct real state data based on virtual state data; Configuration module 304 is used to configure and infer the target vLLM engine instance based on real state data, and obtain the target inference time. Evaluation module 305 is used to determine the performance evaluation results of the target vLLM engine instance based on the target inference time.

[0090] In some possible implementations, the inference module 302 specifically includes: Under the condition that requests arrive evenly, a full lifecycle simulation is performed based on the target vLLM engine instance and the target batch size to determine the processing stage, the number of basic processing units, and the key-value cache occupancy status of each request. Based on the processing stage of each request, the number of processed request units, and the key-value cache occupancy status, determine the virtual state data corresponding to each request.

[0091] In some possible implementations, the data construction module 303 specifically includes: With all requests in a running state, inference and reproduction are performed based on the virtual state data to obtain the real state data.

[0092] In some possible implementations, configuration module 304 specifically includes: Based on real-world data, the target vLLM engine instance is configured and inferred, and the stage time corresponding to each stage is recorded. The stage time includes the pre-filling stage time, the decoding stage time, and the total time. Based on the stage time corresponding to each stage, the target inference time is obtained.

[0093] In some possible implementations, configuration module 304 specifically includes: Based on the decoding stage time and output length, determine the time consumption of the first request unit corresponding to the decoding stage; The target number of requests to be processed is determined based on the target batch size and total time. Based on the time consumed by the first request unit and the target number of requests processed, the performance evaluation result of the target vLLM engine instance is determined.

[0094] In some possible implementations, the instance creation module 301 specifically includes: The target vLLM engine instance is initialized using virtual loading; where virtual loading skips the actual loading of the target model weights and directly constructs the computation graph and scheduling logic corresponding to the target vLLM engine instance.

[0095] The vLLM performance evaluation apparatus according to the embodiments of this application can correspondingly execute the method described in the embodiments of this application, and the other operations and / or functions of each module / unit of the vLLM performance evaluation apparatus are respectively for implementing Figure 2 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.

[0096] This application also provides a computing device. For example... Figure 4 As shown in the figure, this is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 400 includes a bus 401, a processor 402, a communication interface 403, and a memory 404. The processor 402, the memory 404, and the communication interface 403 communicate with each other via the bus 401.

[0097] Bus 401 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0098] Processor 402 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0099] Communication interface 403 is used for communication with external devices.

[0100] Memory 404 may include volatile memory, such as random access memory (RAM). Memory 404 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0101] The memory 404 stores executable code, and the processor 402 executes the executable code to perform the aforementioned vLLM performance evaluation method.

[0102] Specifically, in achieving Figure 3 In the case of the illustrated embodiment, and Figure 3 When the modules or units of the vLLM performance evaluation device described in the embodiments are implemented in software, the following steps are performed: Figure 3The software or program code required for the functions of each module / unit can be partially or entirely stored in memory 404. Processor 402 executes the program code corresponding to each unit stored in memory 404 and performs the aforementioned vLLM performance evaluation method.

[0103] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the vLLM performance evaluation method described above.

[0104] This application also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0105] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0106] When the computer program product is executed by a computer, the computer performs any of the aforementioned vLLM performance evaluation methods. The computer program product can be a software installation package; when any of the aforementioned vLLM performance evaluation methods is required, the computer program product can be downloaded and executed on the computer.

[0107] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0108] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A vLLM performance evaluation method, characterized in that, The method includes: Based on the target model, a target vLLM engine instance is created and the target batch size corresponding to the target vLLM engine instance is determined; wherein, the target vLLM engine instance is configured with target tensor parallel scale, pre-allocated video memory resource limit, input length and output length; Under the condition that the requests arrive evenly, inference is performed based on the target vLLM engine instance and the target batch size to obtain the virtual state data corresponding to each request; Based on the virtual state data, construct the real state data; Based on the real state data, the target vLLM engine instance is configured and inferred to obtain the target inference time; Based on the target inference time, the performance evaluation result of the target vLLM engine instance is determined.

2. The method according to claim 1, characterized in that, Under the condition of uniform request arrival, inference is performed based on the target vLLM engine instance and the target batch size to obtain virtual state data corresponding to each request, including: Under the condition that requests arrive evenly, a complete lifecycle simulation is performed based on the target vLLM engine instance and the target batch size to determine the processing stage, the number of basic processing units, and the key-value cache occupancy status of each request. Based on the processing stage of each request, the number of processed request units, and the key-value cache occupancy status, determine the virtual state data corresponding to each request.

3. The method according to claim 1, characterized in that, The construction of real state data based on the virtual state data includes: With all requests in a running state, inference and reproduction are performed based on the virtual state data to obtain the real state data.

4. The method according to claim 1, characterized in that, The process of configuring and inferring the target vLLM engine instance based on the real-state data to obtain the target inference time includes: Based on the real state data, the target vLLM engine instance is configured and inferred, and the stage time corresponding to each stage is recorded; wherein, the stage time includes the pre-filling stage time, the decoding stage time, and the total time. Based on the time consumed in each stage, the target reasoning time is obtained.

5. The method according to claim 4, characterized in that, The process of determining the performance evaluation result of the target vLLM engine instance based on the target inference time includes: Based on the decoding stage time and output length, determine the time consumption of the first request unit corresponding to the decoding stage; The target number of requests to be processed is determined based on the target batch size and total time. The performance evaluation result of the target vLLM engine instance is determined based on the time consumed by the first request unit and the target number of requests processed.

6. The method according to claim 1, characterized in that, The creation of a target vLLM engine instance based on the target model includes: The target vLLM engine instance is initialized using a virtual loading method; wherein, the virtual loading method skips the actual loading of the target model weights and directly constructs the computation graph and scheduling logic corresponding to the target vLLM engine instance.

7. A vLLM performance evaluation device, characterized in that, The device includes: The instance creation module is used to create a target vLLM engine instance based on the target model and determine the target batch size corresponding to the target vLLM engine instance; wherein, the target vLLM engine instance is configured with target tensor parallel scale, pre-allocated video memory resource limit, input length and output length; The inference module is used to perform inference based on the target vLLM engine instance and the target batch size when the requests arrive evenly, and obtain the virtual state data corresponding to each request. A data construction module is used to construct real state data based on the virtual state data; The configuration module is used to configure and infer the target vLLM engine instance based on the real state data, and obtain the target inference time. The evaluation module is used to determine the performance evaluation result of the target vLLM engine instance based on the target inference time.

8. A computing device, characterized in that, Including memory and processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes one or more computer instructions, which, when executed by a computer, perform the method as described in any one of claims 1 to 6.

Images