Code interpretation large model parameter tuning method based on vllm deployment

Abstract

The application provides a code reading large model parameter optimization method based on vllm deployment, which is implemented by a computer, first, the core viewpoints of a paper are combined, an experimental group is designed, a benchmark experimental group and an experimental control group are designed, experimental record data contents are used, experimental results are obtained, and mathematical analysis is combined to analyze the obtained data, then, suspected problem data is investigated and surveyed to obtain the best parameter configuration of the code reading large model relying on the vllm deployment, the core viewpoints of the paper can be combined to design reasonable benchmark experimental groups and experimental control groups, experimental record data contents are used, and mathematical analysis is combined to obtain the best parameter practice of the code reading large model relying on the vllm deployment. The application can ensure that the parameter optimization result of the code reading large model deployed on the vllm has a complete evidence chain, and has enough confidence to apply the optimized parameters in a real production environment.

Description

Technical Field

[0001] This invention relates to the field of large code interpretation models, and in particular to a method for parameter tuning of large code interpretation models based on VLLM deployment. Background Technology

[0002] Code interpretation of large models is an emerging field, and VLLM (Virtual Large Language Model) is one of the infrastructures for deploying enterprise-level large models.

[0003] Typically, code is a private file within an enterprise, while publicly deployed code models are generally shared. Code models include public-network code models and local code models. Public-network code models can be deployed on the public network (such as chatgpt), while local code models can be deployed locally within the enterprise. In practical applications, VLLM inference frameworks often incorporate PagedAttention. PagedAttention allows storing consecutive keys and values ​​in non-contiguous memory spaces. In PagedAttention, memory waste only occurs in the last block of the sequence, amounting to less than 4%, significantly improving inference speed. For fast inference, VLLM inference frameworks are typically used when deploying local code models. When deploying local code models, multiple different code models can be integrated to meet various code inference needs, such as integrating codegeex, stracoder, wizardcoder, and other code models.

[0004] However, for large-scale code interpretation models deployed on VLLM to achieve optimal performance during the inference phase, some key parameters of VLLM need to be tuned. For deploying large-scale code interpretation models on VLLM, there is a lack of best practice references on how to set the main parameters, whether to turn the switches on or off, and what the specific values ​​should be. Currently, there is a lack of theoretical research on large-scale code interpretation models, and there is a lack of experimental procedures and demonstrations on how to tune VLLM parameters. We can only see some suggested values ​​in the VLLM manual, and even the official VLLM manual only provides suggested values ​​without including the complete derivation process, making it difficult to prove whether these suggested values ​​are reasonable or applicable to large-scale code interpretation models.

[0005] Therefore, in actual engineering practice, the lack of optimal configuration of the corresponding parameters will prevent the large model inference from achieving its best performance.

[0006] Existing technical solutions do not provide a rigorous proof process for optimizing the parameters of large models on VLLM through code interpretation; they only offer some suggested values, and there is no guarantee that these suggested values ​​will be applicable to real-world scenarios.

[0007] For the emerging field of interpreting large-scale models based on VLLM deployments, there is no complete causal chain from theory to experimental design, data analysis, troubleshooting of potentially problematic data, and finally, determining the optimal parameter configuration. Therefore, even the configurations provided in the official manual may not be the optimal parameter configurations for direct application in a production environment; they can only serve as reference values. There is a lack of experimental design and data analysis techniques.

[0008] However, there is an urgent need in existing technologies for an inventive method that combines relevant technical papers to design corresponding benchmark and control group experiments, record the experimental process and results, and investigate suspected erroneous data in the experiments. This would allow for a rigorous determination of the optimal parameter configuration practices and proof of their accuracy. Furthermore, when configuring parameters, specific scenarios must be considered. For example, some scenarios prioritize the time taken for the first token during inference (TTFT), while others prioritize the average time taken for subsequent tokens (ITL). The optimal configuration should be tailored to the specific scenario. Summary of the Invention

[0009] In view of the technical problems existing in the prior art, the present invention provides a method for parameter tuning of a large code interpretation model based on VLLM deployment. This method is implemented by a computer and can combine the core viewpoints of the paper to design reasonable benchmark experimental groups and experimental control groups, record data through experiments, and combine mathematical analysis to obtain the optimal parameter practice of a large code interpretation model based on VLLM deployment.

[0010] Specifically, according to a first aspect of the present invention, a method for parameter optimization of a large-scale code interpretation model based on VLLM deployment is provided. This method is implemented by a computer. First, based on the core viewpoints of the paper, an experimental group is designed, including a baseline experimental group and an experimental control group. Data is recorded experimentally to obtain experimental results. Mathematical analysis is then used to analyze the obtained data. Next, data suspected of having problems is investigated and assessed to obtain the optimal parameter configuration for the large-scale code interpretation model based on VLLM deployment.

[0011] In the experimental design, multiple graphics cards, such as four A100 graphics cards, were used. The model used was a finely tuned qwen2-72b-instruct large language model, deployed based on the fastchat+vllm inference engine. Through the model controller, for each computing node in the target large model inference cluster, the location of the inference request initiation, the location of the inference request completion response, the location of the first token returned, and the total inference time were determined, and the average time for non-first tokens was calculated.

[0012] As described above, configure the controller address property in the AppSettings settings of the App to point to the model controller address, and complete the registration with the controller in the base class BaseModelWorker.

[0013] As described above, in the base class BaseModelWorker, the worker generate() method is called as follows:

[0014] 1. Obtain several parameters:

[0015] prompt,request_id,temperature,top_p,top_k,presence_penalty,frequency_penaltyuse_beam_search,request object;

[0016] 2. Construct a SampleParams object;

[0017] 3. Call the AsyncLLMEngine,generate(context,sample_params,request_id) method.

[0018] As described above, the worker mentioned is the FastChat vllm-worker, not the worker in the vllm framework.

[0019] As described above, the Prometheus client library is embedded into the application code of the inference engine (fastchat+ vllm) to record metric data related to inference requests. Through comprehensive analysis of these specific metric data, key performance indicators of inference performance data for each computing node are obtained.

[0020] As described above, parameter tuning can be performed in a serial manner with 20 transactions. Each transaction calls a certain number of large models, and the next transaction waits until the previous transaction has been processed before proceeding.

[0021] As described above, the method of interpreting and tuning key parameters of a large model—PrefixCaching—deployed on VLLM is used in the experimental design.

[0022] Baseline group: 20 transactions executed serially, with --enable-prefix-caching disabled;

[0023] Control group: 20 transactions were executed serially with --enable-prefix-caching enabled.

[0024] As described above, when Prefix Caching is enabled, for two independent requests, if request 2's request token sequence is mostly similar to that of request 1 and shares the same System Prompt, then request 2 reuses the physical blocks that request 1 has already generated during the prefill phase.

[0025] As described above, when reusing physical blocks, the Prefix Caching is completely consistent starting from the first token, the sequence calculated starting from the first token is consistent, and the corresponding hash values ​​are consistent.

[0026] As described above, when evaluating the rate of the first token, the max_new_token parameter is set to 1 so that vllm generates only 1 token; when evaluating the rate of subsequent tokens, the token generation rate is statistically analyzed using the fastchat client.

[0027] As described above, in the experimental group design, the constraint of the experimental group is that the value of max_num_batched_tokens is greater than the value of max-num-seqs.

[0028] The code-interpretation large-scale model parameter tuning method based on VLLM deployment according to this invention employs rigorous paper interpretation, experimental group design, meticulous analysis and derivation processes, and the design of additional experimental groups to verify suspected erroneous data. This ensures that the parameter tuning results of the code-interpretation large-scale model deployed on VLLM have a complete chain of evidence. The tuned parameters are corroborated by evaluation data combined with real-world environments, thus providing sufficient confidence to apply these tuned parameters in real production environments. Attached Figure Description

[0029] Figure 1 This diagram shows the overall deployment architecture of the experimental group of a code interpretation method for large model parameter tuning based on vllm deployment according to an embodiment of the present invention.

[0030] Figure 2 Show Figure 1A graph showing the experimental group of the method involved regarding the first token time-to-TFT (TTFT);

[0031] Figure 3 Show Figure 1 A graph showing the rate of first token generation for the experimental group of the methods involved;

[0032] Figure 4 Show Figure 1 The experimental group of the methods involved provides a table showing the number of input tokens and the number of output tokens for each round.

[0033] Figure 5 Show Figure 1 The execution time graph for the additional experimental group 1 and round 9 of the method involved;

[0034] Figure 6 Show Figure 1 The graph shows the input / output token count statistics for experimental groups 2, 3, and 4 of the methods involved in round 9. Detailed Implementation

[0035] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Those skilled in the art will understand that this description is exemplary and the present invention is not limited to these specific embodiments.

[0036] Figure 1 This diagram illustrates the overall deployment architecture of the experimental group in the code interpretation and large model parameter tuning method based on VLLM deployment involved in this invention. In this embodiment, the inference engine uses FastChat + VLLM.

[0037] (I) [Code Interpretation: The Basic Environment for Hyperparameter Tuning of Large Models]

[0038] The experiment used four A100 (80G) graphics cards and ran a fine-tuned qwen2-72b-instruct large language model, deployed based on the fastchat+vllm inference engine.

[0039] like Figure 1As shown, firstly, it is determined whether an inference request exists on the requesting side. It can be understood that an inference request refers to a computational task performed on a large model, typically including input data and the expected output result. The inference and proof method in this embodiment of the invention operates when an inference request exists. If an inference request exists, the model controller determines the inference request initiation position, the inference request completion response position, the position where the first token is returned, and the total inference time for each computational node in the target large model inference cluster. The inference request initiation position refers to the code location where the user or system issues the inference request. This position can be a specific function call or a key point in the program execution flow, marking the start of the inference process. The inference request completion response position refers to the code location where the inference result is returned. This position is usually pre-defined by the developer when writing the program to ensure that the inference result can be correctly received and used. It could be a function return value, a variable assignment operation, or any other code point that can receive data. The position where the first token is returned refers to the position where the first output unit is returned, where the token typically refers to the smallest unit of model output (divided according to actual conditions, not specifically limited here). Total inference time refers to the total time consumed from the inference request being initiated until all inference results are returned. Specifically, the controller address property in the AppSettings is configured to point to the model controller address, and registration with the controller is completed in the base class BaseModelWorker. Calling the worker generate() method can be done as follows: 1. Obtain several parameters, such as...

[0040] The functions include prompt, request_id, temperature, top_p, top_k, presence_penalty, frequency_penalty, use_beam_search, and the request object. 2. Construct a SampleParams object. 3. Call the AsyncLLMEngine, generate(context, sample_params, request_id) method.

[0041] It's important to note that the worker mentioned above is the vllm-worker for FastChat, not the worker within the VLLM framework. The Prometheus client library is embedded in the application code of inference engines (such as VLLM and FastChat) to record metrics data related to inference requests. Through comprehensive analysis of these specific metrics, key performance indicators such as inference performance data for each compute node can be obtained.

[0042] During parameter tuning, there will be a method of executing 20 transactions serially. Each transaction will call a certain number of large models, and the next transaction must wait for the previous transaction to be processed before proceeding.

[0043] (II) [Code Interpretation and Key Parameter Tuning for Large Models Deployed on VLLM - Prefix Caching]

[0044] Baseline group: 20 transactions executed serially. `--enable-prefix-caching` is disabled.

[0045] Control group: 20 transactions were executed serially (same as the baseline group), with --enable-prefix-caching enabled.

[0046] For the baseline group, it takes 63 minutes to execute without enabling --enable-prefix-caching; for the control group, it takes 69 minutes to execute with --enable-prefix-caching enabled.

[0047] This shows that enabling Prefix Caching did not improve overall efficiency; in fact, it slowed things down. This is because enabling Prefix Caching in VLLM only affects the performance of the prefill stage, but not the decode stage. This is because VLLM's CachedBlockAllocator manages physical blocks using Prefix Caching.

[0048] When Prefix Caching is enabled, for two independent requests, if request 2's token sequence is largely similar to request 1's (e.g., sharing the same System Prompt), then request 2 can reuse the physical blocks already generated by request 1 during the prefill phase. This Prefix Caching must ensure complete consistency starting from the first token; there can be no misalignment. If there is a misalignment, the sequence calculated from the first token will be different, resulting in inconsistent hash values ​​and preventing the reuse of physical blocks.

[0049] The most important task in the prefill phase is processing requests and generating the first token. Therefore, this experiment examines whether enabling prefix caching affects the time required to generate the first token.

[0050] From a log perspective, it's difficult to capture the time taken to generate the first token. There's a workaround: since the VMLM is accessed via FastChat, this access method is compatible with Hugging Faces. In Hugging Faces, setting `max_new_tokens` to 1 ensures the large model outputs at most one token; no further tokens are generated after the first one. In this case, the token generation time from the FastChat client side represents the time taken to generate the first token.

[0051] The vllm version used here is 0.6.1.post2, which already has prefix-caching capabilities (reducing latency and improving throughput). The corresponding experimental data is as follows:

[0052]

[0053] In statistics, to calculate the expected value (i.e., the mean), for multiple consecutive independent events, the highest and lowest samples need to be removed, and then the mean of the remaining samples is calculated. The highest score is 20.37%, and the lowest score is -29.05%. Therefore, enabling Prefix Caching reduces the average time for generating the first token by 9.8% compared to not enabling Prefix Caching, meaning that Prefix Caching is effective in reducing the time for generating the first token in the Prefill stage.

[0054] Calculation method: (13.9%+8.7%+0.35%+15.99%+13.38%+6.54%) / 6=9.8%.

[0055] (III) [Code Interpretation and Key Parameter Tuning for Large Models Deployed on VLLM - Chunked Prefill]

[0056] (i) [Paper References]

[0057] For information on chunked prefill for VLLM, please refer to the following paper:

[0058] The first reference paper is "SARATHI: Efficient LLM Inference by PiggybackingDecodes with Chunked Prefills," available at: https: / / arxiv.org / pdf / 2308.16369. It outlines several core concepts: SARATHI's core algorithm is decode-maximal batching, which determines the maximum number of tokens that can be processed in each batch. Then, the "decode-maximalbatching" algorithm determines the ratio of prefill tokens to decode tokens within that batch. Specifically, the algorithm prioritizes adding sequences requiring decoding to the batch until the reserved key-value buffer space for decoding is insufficient. Finally, based on the remaining token budget in the batch, the sequences requiring prefill are split according to chunk size, and the corresponding prefill tokens are added to the batch.

[0059] Another relevant paper is "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve," which outlines several key points (paper link: https: / / arxiv.org / pdf / 2403.02310):

[0060] Key Takeaway 1: The Prefill stage is compute-bounded. As the Sequence Length increases, the token processing time remains almost constant between 128 and 512 seconds. However, beyond 512 seconds, the processing time increases dramatically. The Decode stage, on the other hand, is memory-bounded. The token processing time remains almost constant regardless of batch size. The paper points out that, taking a single A100 card as an example, as the Sequence Length increases, the overall trend of token processing time is an increase, but this change is not constant. There is a critical value: Sequence length = 512. Within 512 seconds, the increase in token processing time is not significant, but beyond 512 seconds, the token processing time changes drastically. For Decode, however, the token processing time remains almost constant across different batch sizes.

[0061] Key Takeaway 2: The optimal balance point for computational resources across four A100 GPUs is 512 tokens, representing the compute bounds in the prefill phase and the memory bounds in the decode phase. The prefill phase is computationally intensive, processing all input tokens in parallel, easily maximizing GPU utilization. The decode phase is I / O intensive, generating tokens autoregressively and frequently reading the key-value cache, resulting in high I / O overhead but low GPU overhead. Therefore, to fully utilize computing resources, a chunked-prefill technique is used, combining requests in the prefill and decode phases into a single batch for simultaneous computation. This is the Sarathi-Serve algorithm described in the paper. Regarding the optimal balance point, the paper determined the value to be 512 through multiple benchmark tests.

[0062] These two papers are included here by reference.

[0063] (ii) [Explanation of the vllm manual]

[0064] In VLLM, the Scheduler algorithm typically prioritizes prefill requests over decode requests. Therefore, if the prompt in the prefill request is large, the Scheduler has less time allocated for decoding, thus slowing down the average rate of generating non-first tokens. VLLM therefore includes chunked prefill capabilities. These chunks are split into smaller chunks, which are then scheduled together with decode requests as a batch. This increases the frequency of decoding (thus improving the generation rate of non-first tokens), but at the expense of prefill (leading to increased latency for the first token).

[0065] The core challenge here is the size of the chunked block setting. If the setting is too large, it's equivalent to insufficient prefill chunking, so not much prefill is sacrificed (in extreme cases, it becomes so large that prefill is not chunked, which is equivalent to not enabling chunked prefill). Therefore, decoding doesn't fully benefit, resulting in a lower average token rate and a lower latency for the first token. If the setting is too small, it's equivalent to very fine prefill chunking, so the decoder has more scheduling frequency, resulting in a higher average token rate, but also a higher latency for the first token.

[0066] The `schedule_chunked_prefill()` function is quite complex. Chunked prefill breaks down prefill requests into chunks and sends them together with decode requests. The scheduling strategy is as follows: a. Schedule as many decode requests as possible; b. Schedule chunked and incomplete prefill requests; c. Schedule requests in the swapped queue; d. Schedule new prefill requests. Since decode requests are primarily used to generate tokens other than the first token, this scheduling strategy improves the rate of generating these other tokens.

[0067] This scheduling strategy results in higher GPU utilization because it places prefill and decode requests in the same batch. It improves token latency because decode requests are no longer blocked by prefill requests. In the default scheduling strategy, prefill requests are scheduled with higher priority than decode requests. Now, by treating them equally, decode requests are processed more promptly, thus increasing the average rate of token generation.

[0068] The VLLM manual states that optimization can be achieved by changing `max_num_batched_tokens`. The default value is 512, which achieves the best ITL (Inter-Process Time, not the first token generation average) on the A100 benchmark. Smaller `max_num_batched_tokens` will result in a better ITL because fewer prefill requests are added, thus interrupting the processing of decode requests. Larger `max_num_batched_tokens` will result in a better FTT (Third-First Time) because more prefill requests are included in the batch. This study uses an experimental design to progressively increase the value of `max_num_batched_tokens` and observe the actual effects to determine the optimal value.

[0069] (iii) [Experimental group design]

[0070] The rate of the first token is evaluated by setting the max_new_token parameter to 1, so that vllm generates only 1 token; the rate of subsequent tokens is evaluated by using the fastchat client to perform token generation rate statistics.

[0071] The constraint for the experimental group is that the value of `max_num_batched_tokens` must be greater than the value of `max-num-seqs`. This is because before inference, LLM performs a calculation, dividing these two values ​​by `max_num_batched_tokens / max_num_seqs` to obtain the number of tokens for each sequence, which is then used in other calculations. Therefore, the value of `max_num_batched_tokens` must be greater than `max_num_seqs` to ensure the result is an integer greater than 1. If `max_num_batched_tokens` is less than `max_num_seqs`, VLLM will report an error.

[0072] Therefore, in the experimental design, four A100 cards were used, with tensor parallelism. The max-num-seqs value was set to 8.

[0073] The control group had chunked-prefill disabled.

[0074] Experiment 1 was conducted with enabled-chunked-prefill enabled and max_num_batched_tokens set to 512.

[0075] Experiment 2 involves enabling enabled-chunked-prefill and setting max_num_batched_tokens to 256.

[0076] Experiment 3 involves enabling enabled-chunked-prefill and setting max_num_batched_tokens to 128.

[0077] Experiment 4 involves enabling enabled-chunked-prefill and setting max_num_batched_tokens to 32.

[0078] (iv) [Observe TTFT]

[0079] The time taken to generate the first token (TTFT), in seconds.

[0080]

[0081] Plot the table as a curve, as shown below. Figure 2 As shown, the horizontal axis represents the number of model calls, and the vertical axis represents the time taken for the first token (TTFT).

[0082] From the perspective of the time taken to acquire the first token, the control group (i.e., without chunked-prefill) had the shortest time, as shown by the orange line (i.e., control group 0). For experimental groups 1, 2, 3, and 4, the time taken increased as max_num_batched_token increased, with the longest time taken for the first token when max_num_batched_token was 32, as shown by the red line (i.e., experimental group 4).

[0083] Convert this graph into a first-level token generation rate graph, with units of tokens / second, as follows:

[0084]

[0085] Plot the table as a curve, as shown below. Figure 3 As shown, the horizontal axis represents the number of model calls, and the vertical axis represents the rate at which the first token is generated.

[0086] Looking at the generation rate of the first token, the control group 0 (i.e. without chunked-prefill) had the fastest rate. However, after chunked-prefill was enabled, the rate slowed down as the value of the max_num_batched_token parameter gradually decreased. When set to the lowest value of this experimental group (max_num_batched_token=32), the generation rate of the first token was the slowest.

[0087] (v) [Total observation time]

[0088] The total time for each experiment is calculated. The experimental group with the shortest total time in each round is indicated in yellow. As you can see, the control group is no longer always the shortest. This time is determined by multiple factors, such as the length of the input token and the length of the output token.

[0089]

[0090] Originally, the time taken for the first token was the shortest in the control group (i.e., without chunked prefill), but here we can see that the total experimental time is no longer always the shortest in the control group. This is because:

[0091] Total time for each experiment = Time spent on the first token (depending on the length of the token entered in the Prompt) + Average time spent on non-first tokens × Number of tokens.

[0092] Enabling chunked prefill speeds up the decoding phase, especially when generating a large number of tokens, resulting in even greater time savings. A table showing the input and output token counts for each round (e.g.) Figure 4 As shown in the figure, it can be seen that:

[0093] For the five call model rounds "1, 2, 6, 7, 8", the number of input tokens far exceeds the number of output tokens. In this case, the control group (without chunked prefill enabled) performs best. Because the control group does not enable the chunked prefill feature, the VLLM scheduler prioritizes processing prefill requests. Prefill is used to generate the first token, thus significantly reducing the time spent generating the first token.

[0094] However, the difference in the number of input and output tokens is not as significant for "3, 4, 5". Therefore, the total time will be determined by both the prefill and decode phases. As `max_num_batched_token` decreases, the total time of the prefill phase will increase, but the average token latency of the decode phase will decrease. Therefore, the data from any experimental group may be optimal (i.e., have the lowest time).

[0095] The details regarding "9, 10, 11" will be explained later.

[0096] (vi) [Observe the ITL (interval between the generation of other tokens besides the first token)]

[0097] We only consider the case where Chunked Prefill is enabled, i.e., the gray area. From the perspective of ITL (ITL represents the time interval between the generation of other tokens besides the first token), this perspective can be represented by a token generation rate graph. The list of average token generation rates is as follows, in units of tokens / s. The tokens with the best ITL performance, i.e., the fastest average generation rate, are marked in dark.

[0098]

[0099] This table shows that the best ITL performance was achieved with 6 votes for Experiment 1 and 2 votes for Experiment 2. Furthermore, the best performance in Experiment 2 was very close to that in Experiment 1. Therefore, it can be concluded that Experiment 1 has the best ITL. Experiment 1 had Chunked Prefill enabled and max_num_batch_tokens=512, which matches the statement in the vllm user manual that max_num_batch_tokens=512 optimizes the ITL of the A100 graphics card.

[0100] (iiv) [Looking at rounds 9, 10, and 11 of the experiment]

[0101] As shown in the table below: A closer look at rounds 9-11 of the experiment reveals that the total time spent in experimental groups 2 and 4 was relatively short, but the total time spent in experimental group 3 was particularly long.

[0102]

[0103] The experimental data shows that for round 9, experimental group 2 took only 7.92 seconds and experimental group 4 took only 11.97 seconds, but experimental group 3 took a whopping 82.49 seconds, almost 8-9 times longer; for round 10, experimental group 2 took only 3.11 seconds and experimental group 4 took only 4.01 seconds, but experimental group 3 took a whopping 20.83 seconds, almost 5-6 times longer; for round 11, experimental group 2 took only 3.31 seconds and experimental group 4 took only 4.06 seconds, but experimental group 3 took a whopping 18.73 seconds, almost 4-5 times longer.

[0104] From Experiment 2 to Experiment 4, the value of `max_num_batched_tokens` decreases progressively: 256 for Experiment 2, 128 for Experiment 3, and 32 for Experiment 4. Therefore, following this pattern, the total time for Experiment 3 in round 9 should be between 7.92 seconds and 11.97 seconds; for Experiment 3 in round 10, it should be between 3.11 seconds and 4.01 seconds; and for Experiment 3 in round 11, it should be between 3.31 seconds and 4.06 seconds.

[0105] To identify whether the problem in experimental group 3 was a common problem or a unique problem, three additional experiments were designed.

[0106] [Additional Experiment 1] Run only Experiment 3, still in the original rounds, and check the overall time consumption, especially to determine whether calling round 9 still takes more than 80 seconds instead of about 10 seconds.

[0107] [Additional Experiment 2] Only run Experiment 3, but shuffle the rounds and let round 9 run first, and see the overall time consumption.

[0108]

Additional Experiment 3

[0109] Through these three additional experiments, we will focus on observing two phenomena: (a) confirming whether only experimental group 3 has problems, or whether any experimental group may have problems. (b) if experimental group 3 has problems, then we will further confirm whether the particularly time-consuming bad cases always occur in round 9, or whether they may occur in other rounds.

[0110] The three additional experiments can help us identify problem areas and research directions:

[0111] a. If experimental group 3 is consistently the slowest, and the slowest time is always round 9, then focus on analyzing the input and output tokens of that round.

[0112] b. If experimental group 3 is consistently the slowest, but not necessarily in round 9, then the focus should be on analyzing the rationality of the `max_num_batched_tokens=128` configuration parameter. For example, are there any checks in the vllm program that prevent this parameter from functioning correctly?

[0113] c. If other experimental groups may also be slow, but once experimental group 3 is given priority, it is no longer slow, then the focus should be on analyzing the GPU resource contention problem.

[0114] Experimental Procedure - Additional Experiment 1

[0115] Observing the results of [Additional Experiment 1], without changing the order, it can be found that the execution time of round 9 is still as high as 83.48 seconds (e.g. Figure 5 (As shown).

[0116] Observe the results of [Additional Experiment 2] by executing round 9 separately and checking the time taken. After running it three times, we found that each round took about 83 seconds.

[0117] Meanwhile, run round 9 using the parameters of experimental group 2 and experimental group 4 respectively:

[0118] Experiment 2 was configured with max_num_batched_tokens=256. After running it three times, it was found that each run took about 8 seconds.

[0119] Experiment 4 was configured with max_num_batched_tokens=32. After running it three times, it was found that each run took about 12 seconds.

[0120] Observing the results of [Additional Experiment 3], it can be found that even if Experiment 3 is executed first, it still takes 82.63 seconds to complete round 9.

[0121] The above three additional experiments fully illustrate the following fact: In Experiment 3 (max_num_batched_tokens=128), for round 9, each execution takes 82 to 83 seconds.

[0122] Therefore, we need to try to see how the number of input / output tokens in round 9 differs from other experimental groups.

[0123] Figure 6 A statistical table showing the number of input / output tokens for experimental groups 2, 3, and 4 in round 9 is provided.

[0124] The number of input tokens was the same for all groups, 4089. However, Experiment 3 had a significantly higher output token count, reaching 1100. In contrast, Experiments 2 and 4 each had only 78 output tokens, a difference of nearly 14 times. Because the tokens are generated autoregressively during the inference phase, a larger number of tokens results in greater execution time. This explains why Experiment 3's execution time in round 9 was as high as 82.49 seconds, while Experiments 2 and 4's execution times in round 9 were only 7.92 and 11.92 seconds, respectively.

[0125] Similarly, Experiment 3 output more tokens in rounds 10 and 11 than Experiments 2 and 4. In round 10, Experiment 3 output 368 tokens, while Experiment 2 and Experiment 4 both output 76 tokens. In round 11, Experiment 3 output 365 tokens, while Experiment 2 and Experiment 4 output 76 and 75 tokens respectively. The difference in number is not as large as in round 9, so Experiment 3 took approximately 4-6 times longer in rounds 10 and 11 than Experiments 2 and 4.

[0126] Therefore, experimental group 3 is normal data; it's just that it has a particularly large number of output tokens, which causes it to take longer than other experimental groups.

[0127] [Experimental Group Summary]

[0128] For the first token generation time (TFTT), disabling chunked prefill is significantly better than enabling it. In other words, disabling chunked prefill is more suitable for scenarios requiring rapid generation of the first token.

[0129] From the perspective of total time consumption, if the input token is much larger than the output token, then even if chunked prefill is enabled, the benefit to the decoding stage is not significant, and it is still optimal not to enable chunked prefill.

[0130] To maximize the ITL (interval between non-first tokens) effect when chunked prefill is enabled, max_num_batched_tokens should be set to 512.

[0131] The code-interpretation large-scale model parameter tuning method based on VLLM deployment according to this invention employs rigorous paper interpretation, experimental group design, meticulous analysis and derivation processes, and the design of additional experimental groups to verify suspected erroneous data. This ensures that the parameter tuning results of the code-interpretation large-scale model deployed on VLLM have a complete chain of evidence. The tuned parameters are corroborated by evaluation data combined with real-world environments, thus providing sufficient confidence to apply these tuned parameters in real production environments.

[0132] The present invention has been described in detail above. Those skilled in the art will understand that various modifications and changes can be made to it. As long as they do not depart from the spirit and purpose of the present invention, all such modifications and changes should fall within the protection scope of the present invention, which is defined by the appended claims.

Claims

1. A method for parameter tuning of a code-interpretation large model based on vllm deployment, the method being implemented by a computer and characterized in that, First, the experimental group design is carried out, the benchmark experimental group and the experimental control group are designed, the experimental record data content is obtained, the experimental results are obtained, and the data obtained are analyzed by combining mathematical analysis, then, the suspected problem data are investigated and surveyed, the suspected error data are demonstrated by designing multiple additional experimental groups, so as to obtain the best parameter configuration of the code reading large model relying on vllm deployment, wherein, the parameter tuning is for the performance parameters of the image prefill stage and the decode stage, specifically including Prefix Caching and Chunked Prefill; the parameter tuning is in the mode of 20 transaction serial execution, each transaction calls a certain number of large models, and the next transaction is performed after the previous transaction is processed; In the experimental group design, multiple graphics cards are used, the model uses the fine-tuned qwen2-72b-instruct large language model, the deployment is based on fastchat+vllm inference engine, through the model controller controller, for each computing node in the target large model inference cluster, the inference request initiation position, the inference request completion reply position, the first token return position and the inference total time are determined, and the average time consumption of non-first token is calculated; when evaluating the rate of the first token, set the max_new_token parameter to 1, so that vllm generates only 1 token; when evaluating the rate of non-first token, the token generation rate is counted through the fastchat client.

2. The method of claim 1, wherein, The controller address property of AppSettings in the App is configured to point to the model controller address, and the registration to the controller controller is completed.

3. The method of claim 2, wherein, When the workergenerate() method is called to complete the registration to the controller controller, the method is: (1) obtain a plurality of parameters; (2) construct a SamplingParams object; (3) call the AsyncLLMEngine, generate(context, sample_params, request_id) method.

4. The method of claim 3, wherein, The worker in the above worker generate() method is the vllm-worker of fastchat, not the worker in the vllm framework.

5. The method of claim 4, wherein, The Prometheus client library is embedded in the application code of the inference engine (fastchat+vllm) to record the specific indicator data related to the inference request, and through comprehensive analysis of these specific indicator data, the key performance indicators of the inference performance data of each computing node are obtained.

6. The method of claim 1, wherein, For the key parameter tuning of the code reading large model deployed on vllm-Prefix Caching, in the experimental group design, Benchmark group: 20 transaction serial execution, without starting --enable-prefix-caching prefix cache; Control group: 20 transactions were executed serially with --enable-prefix-caching enabled.

7. The method of claim 6, wherein, When Prefix Caching is enabled, for two independent requests, if request 2 shares the same System Prompt as request 1 with its request token sequence, then request 2 will reuse the physical blocks that request 1 has already generated during the prefill phase.

8. The method of claim 7, wherein, When reusing physical blocks, Prefix Caching is completely consistent starting from the first token, the sequence calculated starting from the first token is consistent, and the corresponding hash values ​​are consistent.

Images