Code interpretation large model parameter tuning method based on vllm deployment

Abstract

The invention provides a vllm deployment-based code interpretation large model parameter tuning method, which is implemented by a computer, and comprises the following steps of: firstly, designing an experimental group by combining a core viewpoint of a thesis, designing a reference experimental group and an experimental control group, recording data contents by using an experiment to obtain an experimental result, and combining mathematical analysis to obtain a mathematical analysis result; the method comprises the following steps of: analyzing obtained data, and then checking and surveying data with suspected problems so as to obtain optimal parameter configuration of a code interpretation large model deployed by relying on vllm, and by combining a core viewpoint of a thesis, designing a reasonable benchmark experiment group and an experiment control group, recording data content by using an experiment and combining mathematical analysis, determining the optimal parameter configuration of the large model. And obtaining the optimal parameter practice for interpreting the large model by relying on the codes deployed by the vllm. According to the method, it can be ensured that a complete evidence chain exists in the result of parameter adjustment and optimization of code deployment on the vllm to interpret the large model, and sufficient confidence is provided for applying the adjusted and optimized parameters in the real production environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of code interpretation large models, in particular to a code interpretation large model parameter optimization method based on vllm deployment. BACKGROUND

[0002] Code interpretation large models are a new field, and vllm (Virtual Large Language Model) is one of the infrastructures for deploying enterprise-level large models.

[0003] Generally, codes are private files within an enterprise, and code models deployed on public networks are generally shared. Code models include public code models and local code models. Public code models can be code models deployed on public networks (such as chatgpt), and local code models can be code models deployed in an enterprise. The VLLM inference framework often introduces PagedAttention when used in practice. PagedAttention allows storing continuous keys and values in non-continuous memory space. In PagedAttention, memory waste only occurs in the last block of the sequence, wasting less than 4%, greatly improving the inference speed. In order to achieve fast inference, the VLLM inference framework is usually used when deploying local code models. When deploying local code models, multiple different code models can be integrated to meet various code inference needs, such as integrating codegeex, stracoder, wizardcoder, and other code models.

[0004] However, in order for the code interpretation large model based on vllm deployment to perform optimally in the inference stage, some key parameters of vllm need to be optimized. For code interpretation large models deployed on vllm, how should the main parameters be set, should the switch be turned on or off, and what are the specific numerical values? These topics lack corresponding best practice references on the market. Currently, there is a lack of theoretical research on code interpretation large models, and there is also a lack of experimental process and argumentation on how to optimize the parameters of vllm. Only some reference suggested values are provided in the vllm manual. Even the official manual of vllm is only a suggested value, but it does not include the complete derivation process of the numerical value, making it difficult to prove whether these reference suggested values are reasonable and suitable for code interpretation large models.

[0005] Therefore, this may cause the best performance of large model inference to be unable to be achieved in actual engineering practice due to the lack of optimal configuration of corresponding parameters.

[0006] The existing technical solution does not give a rigorous proof process for the best parameter tuning of the code reading large model on the vllm, but only has some recommended values, which cannot guarantee that these recommended values are applicable to the real scene.

[0007] For the emerging field of code reading large model based on vllm deployment, there is no complete causal chain from theory to experimental group design, to data analysis, to suspected problem data investigation, to the final best parameter configuration. Therefore, even the configuration given by the official manual cannot be directly used as the best parameter configuration in the scene-oriented production environment, but can only be used as a reference configuration value, lacking similar experimental group design skills and data analysis skills.

[0008] However, there is an urgent need for an invention method in the prior art to combine related technical papers, design corresponding benchmark groups and control group experiments, record the completion process and experimental results of the experiments, and investigate suspected error data in the experiments, so as to very rigorously obtain the best parameter configuration practice and prove its accuracy. And when configuring parameters, specific scenarios also need to be considered, such as some scenarios that value the time of the first Token (TTFT) during inference, and some scenarios that value the average time of non-first Token (ITL) during inference. The combination of scenarios is optimized. SUMMARY

[0009] In view of the technical problems existing in the prior art, the present application provides a code reading large model parameter tuning method based on vllm deployment, which is implemented by a computer and can combine the core points of papers, design reasonable benchmark experimental groups and experimental control groups, record data content with experiments, and combine mathematical analysis to obtain the best parameter practice of the code reading large model relying on vllm deployment.

[0010] Specifically, according to the first aspect of the present application, a code reading large model parameter tuning method based on vllm deployment is provided, which is implemented by a computer. First, the core points of papers are combined to design experimental groups, design benchmark experimental groups and experimental control groups, record data content with experiments, obtain experimental results, and combine mathematical analysis to analyze the obtained data. Then, the data suspected to have problems is investigated and investigated to obtain the best parameter configuration of the code reading large model relying on vllm deployment, wherein, In the experimental group design, multiple graphics cards such as 4 A100 graphics cards are used, the running model is the fine-tuned qwen2-72b-instruct large language model, the deployment is based on the fastchat+vllm inference engine, through the model controller controller, for each computing node in the target large model inference cluster, determine the inference request initiation position, inference request completion reply position, first token return position and inference total time, and calculate the average time consumption of non-first token.

[0011] As described above, the method configures the controller address property of AppSettings pointing to the model controller address in the App, and registers with the controller controller in the base class BaseModelWorker.

[0012] As described above, in the base class BaseModelWorker, the worker generate() method is called: 1. Obtain several parameters: prompt, request_id, temperature, top_p, top_k, presence_penalty, frequency_penaltyuse_beam_search, request object; 2. Construct the SamplingParams object; 3. Call the AsyncLLMEngine, generate(context, sample_params, request_id) method.

[0013] As described above, the worker is the vllm-worker of fastchat, not the worker in the vllm framework.

[0014] As described above, the Prometheus client library is embedded in the application code of the inference engine (fastchat+ vllm) to record the metric data related to the inference request. Through comprehensive analysis of these specific metric data, the key performance indicators of the inference performance data of each computing node are obtained.

[0015] As described above, the parameter tuning has a 20-transaction serial execution mode, each transaction calls a certain number of large models, and the next transaction is performed after the previous transaction is processed.

[0016] As described above, for the code deployed on vllm, the large model key parameter tuning-PrefixCaching is interpreted, and in the experimental group design, Baseline: 20 transactions serially executed, without prefix caching enabled. Control: 20 transactions serially executed, with prefix caching enabled.

[0017] As described above, when prefix caching is enabled, for two independent requests, if request 2 has a token sequence that is mostly similar to request 1, and shares the same system prompt, request 2 reuses the physical block that request 1 has already generated in the prefill phase.

[0018] As described above, when reusing the physical block, prefix caching is identical from the first token, the sequence calculated from the first token is identical, and the corresponding hash value is identical.

[0019] As described above, when evaluating the rate of the first token, the max_new_token parameter is set to 1, so that the vllm generates only 1 token; when evaluating the rate of non-first tokens, the fastchat client is used to perform token generation rate statistics.

[0020] As described above, in the experimental group design, the constraint of the experimental group is that the value of max_num_batched_tokens is greater than the value of max-num-seqs.

[0021] According to the vllm-based code reading large model parameter optimization method for deployment, the rigorous paper reading, experimental group design, rigorous analysis and derivation process, and the design of additional experimental groups to demonstrate suspected error data can ensure that the results of the vllm-based code reading large model parameter optimization have a complete evidence chain. The optimized parameters have real environment evaluation data as evidence, so that there is enough confidence to apply these optimized parameters in the real production environment. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 An overall deployment architecture diagram of the experimental group of the vllm-based code reading large model parameter optimization method according to an embodiment of the present application is shown. Figure 2 A curve diagram of the experimental group of the method related to the time taken for the first token (TTFT) is shown. Figure 1 A curve diagram of the experimental group of the method related to the time taken for the first token (TTFT) is shown. Figure 3 A curve diagram of the experimental group of the method related to the time taken for the first token (TTFT) is shown. Figure 1Graph of the experimental group of the method involved about the first token generation rate; Figure 4 Show Figure 1 The input token number and output token number table for each round of the experimental group of the method involved; Figure 5 Show Figure 1 The execution time running chart of the additional experimental group 1 of the method involved, round 9; Figure 6 Show Figure 1 The input / output token number statistics chart of the experimental group 2, experimental group 3 and experimental group 4 of the method involved, round 9. DETAILED DESCRIPTION

[0023] The present application will be described in detail below with reference to the accompanying drawings in conjunction with the specific embodiments, and those skilled in the art will understand that the description is exemplary and the present application is not limited only to the specific embodiments.

[0024] Figure 1 To show the overall deployment architecture diagram of the experimental group in the code reading large model parameter tuning method based on vllm deployment involved in the present application. In this embodiment, the inference engine adopts fastchat+vllm.

[0025] (I)

Code reading large model hyperparameter tuning basic environment

[0026] As Figure 1As shown, first, it is judged whether there is a reasoning request on the request side. It can be understood that the reasoning request refers to a computing task performed on a large model, usually including input data and expected output results. The reasoning method of the embodiment of the present application is performed in the case of a reasoning request. If there is a reasoning request, through the model controller controller, for each computing node in the target large model reasoning cluster, the reasoning request initiation position, the reasoning request completion reply position, the first token return position and the total reasoning time are determined. Among them, the reasoning request initiation position refers to the code position where the user or system issues the reasoning request. This position can be a specific function call, or a key point in the program execution flow, which marks the beginning of the reasoning process. The reasoning request completion reply position refers to the code position where the reasoning result is returned. This position is usually set by the developer when writing the program to ensure that the reasoning result can be correctly received and used. It can be a function return value, a variable assignment operation, or any other code point that can receive data. The first token return position usually refers to the smallest unit of model output (divided according to actual conditions, not limited here). The first token return position refers to the position where the first output unit is returned. The total reasoning time refers to the total time consumed from the beginning of the reasoning request initiation to the final reasoning result being returned. Among them, the controller address property of the AppSettings in the App is configured to point to the model controller address, and the registration to the controller controller is completed in the base class BaseModelWorker. Among them, calling the worker generate() method can be, for example, 1. Obtain several parameters such as prompt, request_id, temperature, top_p, top_k, presence_penalty, frequency_penalty, use_beam_search, request object, etc. 2. Construct the SamplingParams object 3. Call the AsyncLLMEngine, generate(context, sample_params, request_id) method.

[0027] It is worth noting that the above-mentioned worker is the vllm-worker of fastchat, not the worker in the vllm framework. The Prometheus client library is embedded in the application code of the reasoning engine (such as vllm, fastchat, etc.) to record the index data related to the reasoning request. Through comprehensive analysis of these specific index data, the reasoning performance data and other key performance indicators of each computing node can be obtained.​

[0028] Parameter tuning will have 20 transactions serially executed, each transaction will call a certain number of large models, the next transaction must wait for the previous transaction to be processed.

[0029] (II)

Code interpretation large model key parameter tuning deployed on vllm-Prefix Caching

[0030] Control group: 20 transactions serially executed (same as benchmark group), open --enable-prefix-caching prefix cache.

[0031] For the benchmark group, when --enable-prefix-caching prefix cache is not opened, it needs to execute for 63 minutes; for the control group, when --enable-prefix-caching prefix cache is opened, it needs to execute for 69 minutes.

[0032] From here, we can see that opening Prefix Caching does not improve overall efficiency, but rather slows down. This is because the opening of Prefix Caching in vllm only affects the performance of the prefill stage, but does not affect the performance of the decode stage. Because the CachedBlockAllocator in vllm is managed by physical blocks according to the way of Prefix Caching.

[0033] When Prefix Caching is opened, for two independent requests, if request 2 requests a token sequence similar to request 1, such as sharing the same System Prompt, request 2 can reuse the physical blocks generated in the prefill stage of request 1. This Prefix Caching must be completely consistent from the first token, and cannot be misaligned. If misaligned, the sequence calculated from the first token is not the same, so the corresponding hash value is not consistent, and the physical block cannot be reused.

[0034] The most important work of the prefill stage is to process requests and generate the first token. Therefore, through experiments, we can see whether the opening of prefix caching has an impact on the time consumption of generating the first token.

[0035] From the log perspective, it is difficult to capture the time-consuming of the first token. There is a trick way, because it is accessed through the fastchat way. The access method is compatible with the writing of Hugging Face. In the writing of Hugging Face, set max_new_tokens to 1, that is, let the large model output at most one token, and after generating the first token, it will not generate other tokens. In this case, the token time-consuming from the fastchat client side is the time-consuming of the first token.

[0036] Here, the version of vllm used is 0.6.1.post2, which already has prefix-caching capability (reducing latency and improving throughput). The corresponding experimental data is as follows:

[0037] In statistics, to calculate the expectation (that is, the average), for continuous multiple independent events, the highest sample and the lowest sample need to be removed and then the average of the other samples is calculated. The highest score is 20.37%, and the lowest score is -29.05%. Therefore, the average percentage of the first token time-consuming reduction of the Prefix Caching enabled compared to the Prefix Caching not enabled is 9.8%, that is, the Prefix Caching is effective in reducing the time-consuming of the first token in the Prefill stage.

[0038] Calculation method: (13.9% + 8.7% + 0.35% + 15.99% + 13.38% + 6.54%) / 6 = 9.8%.

[0039] III)

Code deployed on vllm Key parameter tuning of large model-Chunked Prefill

[0040] Another reference paper is "Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve," which discusses several key points (paper link: https: / / arxiv.org / pdf / 2403.02310): Key point 1: The prefilled stage is compute-bounded, and as the sequence length increases, the token processing time remains almost constant in the 128-512 interval, but after exceeding 512, the processing time increases dramatically. The decode stage is memory-bounded, and the token processing time remains almost constant for different batch sizes. The paper points out that, using a single A100 card as an example, as the sequence length increases, the token processing time overall trend is larger, but the change is not constant. There is a critical value sequence length = 512. Within 512, the token processing time growth trend is not obvious, but after 512, the token processing time changes greatly. For decode, the token processing time remains almost constant for different batch sizes.

[0041] Core Insight 2: The compute bound of prefill stage and the memory bound of decode stage, the power balance point of 4 A100 cards is 512 tokens. The prefill stage is compute intensive, it processes all tokens in parallel, so it can easily fill the GPU utilization. The decode stage is I / O intensive. It generates tokens recursively, and needs to read kv cache frequently during this period, so its I / O overhead is large, but its GPU overhead is small. Therefore, in order to fully utilize the computing resources, use chunked-prefill technology to put requests in the prefill stage and requests in the decode stage in a batch to calculate at the same time. This is the Sarathi-Serve algorithm in the paper. The optimal value of the balance point is 512, which is measured by multiple benchmark tests in the paper.

[0042] The two papers are included in this paper by reference.

[0043] (ii) [vllm manual] The Scheduler algorithm in vllm generally prioritizes prefill requests and then schedules decode requests. So if the prompt in prefill is large, the Scheduler will have less time to allocate to decode, so the average rate of generating non-first tokens will slow down. Therefore, vllm has chunked prefill capability, which divides prefill requests into small chunks, and these chunks and decode requests are scheduled as a batch. This increases the frequency of decode (so the rate of generating non-first tokens will increase), while sacrificing prefill to some extent (so the first token delay will increase).

[0044] The core problem here is the size of the chunked block setting. If it is set to be large, it means that the prefill block is not fine enough, so it does not sacrifice prefill too much (in the extreme case, it is particularly large, which means that prefill is not blocked, which is equivalent to not enabling chunked prefill), so decode does not benefit enough, the average token rate will be small, and the first token delay will be small; if it is set to be small, it means that the prefill block is divided into small blocks, so the decoder has more scheduling frequency, so the average token rate will be large, but the first token delay will be large.

[0045] The schedule_chunked_prefill() logic is complex. Chunked prefill breaks up the prefill request into chunks and sends them together with the decode request. The scheduling strategy is: a. Schedule as many decoding requests as possible; b. Schedule chunked and incomplete prefill requests; c. Schedule requests in the swapped queue; d. Schedule new prefill requests. The decoding requests are mainly used to generate other tokens besides the first token, so this scheduling strategy can improve the rate of generating other tokens besides the first token.

[0046] This scheduling strategy can bring higher GPU utilization because it puts prefill and decode requests in the same batch. It improves the token latency because the decode request is not hindered by the prefill request. In the default scheduling strategy, the priority of scheduling prefill requests is higher than that of decode requests. Now both are treated equally, so the decode request will be better handled in time, thereby improving the average rate of token generation.

[0047] The vllm manual points out that tuning can be done by changing max_num_batched_tokens. The default value is 512, and this 512 achieves the best ITL (average time to generate non-first token) on the A100 benchmark test. Smaller max_num_batched_tokens can achieve better ITL because fewer prefill requests are added to interrupt the processing of decode requests. Larger max_num_batched_tokens can achieve better TFTT because more prefill requests are put into the batch. Here, through the design of the experimental group, the value of max_num_batched_tokens is gradually increased, and the actual effect is observed, so as to determine the best value.

[0048] iii) [Experimental group design] The rate of the first token is still evaluated by setting the max_new_token parameter to 1, so that vllm only generates 1 token; the rate of non-first token is still evaluated by the fastchat client to count the token generation rate.

[0049] The constraint for the experimental group is that the value of max_num_batched_tokens must be greater than the value of max_num_seqs. Because the LLM will perform a calculation before inference, the two values will be divided max_num_batched_tokens / max_num_seqs to obtain the number of tokens corresponding to each sequence, and then participate in other operations. Therefore, the value of max_num_batched_tokens must be greater than max_num_seqs, so that the result of division is an integer greater than 1. If max_num_batched_tokens is less than max_num_seqs, vllm will report an error.

[0050] Therefore, in the experimental group design, 4 A100 cards are used for tensor parallelism. max_num_seqs is set to 8.

[0051] The control group does not enable chunked-prefill.

[0052] Experiment 1 enables enabled-chunked-prefill and sets max_num_batched_tokens to 512.

[0053] Experiment 2 enables enabled-chunked-prefill and sets max_num_batched_tokens to 256.

[0054] Experiment 3 enables enabled-chunked-prefill and sets max_num_batched_tokens to 128.

[0055] Experiment 4 enables enabled-chunked-prefill and sets max_num_batched_tokens to 32.

[0056] (iv) [Observation of TTFT] The time consumption of generating the first token (TTFT) is in seconds.

[0057]

[0058] The table is drawn into a curve as shown in Figure 2 The horizontal axis represents the model call round, and the vertical axis represents the first token time consumption (TTFT).

[0059] From the perspective of the time consumption of the first Token, the control group (i.e., without opening chunked-prefill) has the shortest time consumption, as shown by the orange fold line (i.e., control group 0). The experimental groups 1, 2, 3, and 4 have increasing time consumption as the value of max_num_batched_token increases, and the first Token has the longest time consumption when max_num_batched_token is 32, as shown by the red fold line (i.e., experimental group 3).

[0060] Convert this figure into a first Token generation rate chart, with units of tokens / s, as follows:

[0061] Draw the curve as shown in Figure 3 , where the horizontal axis represents the call model round, and the vertical axis represents the first Token generation rate.

[0062] From the perspective of the generation rate of the first Token, the control group 0 (i.e., without opening chunked-prefill) has the fastest rate, and after opening chunked-prefill, the rate will gradually decrease as the value of the max_num_batched_token parameter decreases. When set to the lowest value of this experimental group (max_num_batched_token = 32), the first Token generation rate is the slowest.

[0063] (v) [Observation of total time consumption] Statistical total time consumption of each experiment. The experimental group with the shortest total time consumption of each round is represented in yellow. It can be seen that it is no longer the control group that is the shortest every time. This time consumption is determined by multiple factors, such as input token length, output token length, etc.

[0064]

[0065] Originally, the time consumption of the first token was the shortest in the control group (i.e., without opening chunked prefill), but here it can be seen that the total time consumption of the experiment is not always the shortest in the control group. This is because: The total time consumption of each experiment = first token time consumption (depends on the input token length in the Prompt) + average time consumption of non-first token × token number. After opening chunked prefill, the decode stage becomes faster, especially when the number of tokens generated by decode is large, which saves more time. From the input token number and output token number comparison table (as shown in Figure 4 ), it can be seen that: For #1, #2, #6, #7, #8, the input tokens are much more than the output tokens. The control group (without chunked prefill) performs the best. Because the control group does not turn on the chunked prefill feature, the vllm scheduler will give priority to the prefill request, and the prefill is used to generate the first token, so it greatly saves the time of generating the first token.

[0066] But for #3, #4, #5, the gap between the number of input tokens and output tokens is not so large. Therefore, the total time consumption is determined by the prefill phase and the decode phase. As max_num_batched_token continues to decrease, the total time consumption of the prefill phase will increase, but the average token delay of the decode phase will decrease. Therefore, the data of any experimental group may be the best (i.e. the lowest time consumption).

[0067] For #9, #10, #11, which will be described in detail later.

[0068] (vi) [Observing ITL (Interval Time Length of Non-first token)] Only look at the case where Chunked Prefill is turned on, that is, the gray part. From the perspective of ITL (ITL represents the time interval between the generation of other tokens other than the first token), this perspective can be represented by a token generation rate chart. The average token generation rate list is as follows, in tokens / s. The best ITL performance, that is, the fastest average token generation rate, is identified by dark color.

[0069]

[0070] This table shows that the best ITL performance has 6 votes for experimental group 1 and 2 votes for experimental group 2. Moreover, the best of experimental group 2 is not much different from that of experimental group 1. Therefore, it can be determined that experimental group 1 is the best for ITL. Experimental group 1 is to turn on Chunked Prefill and max_num_batch_tokens=512, which matches the statement in the vllm manual that max_num_batch_tokens=512 can make the ITL of A100 graphics card reach the best.

[0071] (iiv) [Looking at the #9, #10, #11 three rounds of experiments] As shown in the following table: look at the 9-11 of the experiment for three rounds. Found that the total time of the experimental group 2 and the experimental group 4 is relatively short, but the total time of the experimental group 3 is particularly long.

[0072]

[0073] The experimental data can be seen that for round 9, the time of experimental group 2 is only 7.92 seconds, the time of experimental group 4 is only 11.97 seconds, but the time of experimental group 3 is as high as 82.49 seconds, almost 8-9 times; round 10, the time of experimental group 2 is only 3.11 seconds, the time of experimental group 4 is only 4.01 seconds, but the time of experimental group 3 is as high as 20.83 seconds, almost 5-6 times; round 11, the time of experimental group 2 is only 3.31 seconds, the time of experimental group 4 is only 4.06 seconds, but the time of experimental group 3 is as high as 18.73 seconds, almost 4-5 times.

[0074] The value of max_num_batched_tokens of experimental group 2 to experimental group 4 is decreasing, respectively, experimental group 2 is 256, experimental group 3 is 128, and experimental group 4 is 32. Then according to the rule, the total time of experimental group 3 of round 9 should be between [7.92 seconds, 11.97 seconds], the total time of experimental group 3 of round 10 should be between [3.11 seconds, 4.01 seconds], and the total time of experimental group 3 of round 11 should be between [3.31 seconds, 4.06 seconds].

[0075] In order to identify whether experimental group 3 is a common problem or an individual problem, three additional experiments are designed.

[0076]

Additional experiment 1

[0077]

Additional experiment 2

[0078]

Additional experiment 3

[0079] Through these three additional experiments, two phenomena are observed: (a) confirm whether only experimental group 3 has a problem, or any experimental group may have a problem. (b) If experimental group 3 has a problem, then confirm again whether the badcase with particularly long time always occurs in round 9, or it may occur in other rounds.

[0080] Three additional experiments can help us identify the problem points and research directions: a. If each time is particularly slow in experimental group 3, and each time is slow in round 9, then focus on analyzing the input and output tokens of this round.

[0081] b. If each time is particularly slow in experimental group 3, but not necessarily slow in round 9, then focus on analyzing the rationality of the configuration parameter max_num_batched_tokens=128. For example, does the vllm program have some judgment that prevents this parameter from functioning properly?

[0082] c. If other experimental groups may also be slow, but once experimental group 3 is given priority, it is not slow, then focus on analyzing the resource competition problem of the GPU.

[0083] Experimental process - additional experiment 1 Observing the results of

additional experiment 1

[0084] Observing the results of

additional experiment 2

[0085] At the same time, run round 9 with the parameters of experimental group 2 and experimental group 4 respectively: Experimental group 2 is max_num_batched_tokens=256 configuration, running three times, we find that each time takes about 8 seconds.

[0086] Experimental group 4 is max_num_batched_tokens=32 configuration, running three times, we find that each time takes about 12 seconds.

[0087] Observing the results of

additional experiment 3

[0088] The above three additional experiments fully demonstrate the following facts: that is, experimental 3 (max_num_batched_tokens=128), for round 9, each execution takes 82 to 83 seconds.

[0089] Therefore, we need to try to see what is different about the input / output token number of round 9 and other experimental groups.

[0090] Figure 6 The table shows the input / output token number statistics of experimental group 2, experimental group 3, and experimental group 4 in round 9.

[0091] The input token numbers are all the same, 4089 tokens. But the output token number of experiment group 3 is particularly large, up to 1100. While the output token numbers of experiment group 2 and experiment group 4 are only 78, which is nearly 14 times different. Because the token is generated by autoregression in the reasoning stage, the more the token number, the more time-consuming, which is the main reason why the execution time of experiment group 3 in round 9 is as high as 82.49 seconds, while the execution time of experiment group 2 and experiment group 4 in round 9 is only 7.92 and 11.92 seconds respectively.

[0092] Similarly, the output token number of experiment group 3 in round 10 and round 11 is also more than experiment 2 and experiment 4. In round 10, the output token number of experiment group 3 is 368, and the output token number of experiment group 2 and experiment group 4 is 76; in round 11, the output token number of experiment group 3 is 365, while the output token number of experiment group 2 and experiment group 4 is 76 and 75 respectively. The difference is not as much as round 9, so the time-consuming of experiment group 3 in round 10 and round 11 is also nearly 4-6 times of experiment group 2 and experiment group 4.

[0093] Therefore, experiment group 3 is normal data, only its output token is particularly large, which leads to longer time-consuming than other experimental groups.

[0094] [Experiment group summary] For the first token time-consuming (TFTT), it is obviously better not to open chunked prefill than to open chunked prefill. That is, not opening chunked prefill is more suitable for scenarios that require fast generation of the first token.

[0095] From the perspective of total time-consuming, if the input token is much larger than the output token, even if chunked prefill is opened, the benefit of the decode stage is not obvious, and it is still better not to open chunked prefill.

[0096] For opening chunked prefill, to make the ITL (output delay between non-first tokens) effect reach the peak, max_num_batched_tokens should be set to 512.

[0097] According to the code reading large model parameter optimization method based on the vllm deployment of the application, the rigorous paper reading, the experimental group design, the rigorous analysis and derivation process, and the design of additional experimental groups to demonstrate suspected error data can ensure that the result of the parameter optimization of the code reading large model deployed on the vllm has a complete evidence chain. The optimized parameters have the evaluation data combined with the real environment as evidence, so that there is enough confidence to apply the optimized parameters in the real production environment.

[0098] The application is described in detail above, and those skilled in the art can make various modifications and changes, as long as they do not deviate from the purpose and spirit of the application. These modifications and changes shall fall within the protection scope of the application, and the protection scope of the application is defined by the appended claims.

Claims

1. A method for tuning parameters of a large model based on code interpretation of vlllm deployment, the method being implemented by a computer and characterized in that: First, based on the core ideas of the paper, we conducted experimental group design, designed the benchmark experimental group and the experimental control group, used the experiment to record the data content, obtained the experimental results, and combined mathematical analysis to analyze the obtained data. Then, we investigated and surveyed the data suspected of having problems to obtain the optimal parameter configuration of the code interpretation model deployed based on vlllm. In the experimental group design, multiple graphics cards are used, and the running model adopts a large language model. The deployment is based on the FastChat+VLLM inference engine. Through the model controller, for each computing node in the target large model inference cluster, the inference request initiation location, the inference request completion reply location, the first token return location and the total inference time are determined, and the average time consumed by non-first tokens is calculated.

2. The method according to claim 1, wherein In the App Settings, configure the controller address property to point to the model controller address and complete the registration with the controller.

3. The method according to claim 2, wherein When registering with the controller, call the workergenerate() method: (1) Obtain several parameters; (2) Construct a SamplingParams object; (3) Call AsyncLLMEngine, generate(context, sample_params, request_id) method.

4. The method according to claim 2 or 3, wherein: The worker in the call to worker generate() is the vllm-worker of fastchat, not the worker in the vllm framework.

5. The method according to claim 4, wherein The Prometheus client library is embedded in the application code of the inference engine (fastchat + vllm) to record indicator data related to inference requests. Through comprehensive analysis of these specific indicator data, the key performance indicators of the inference performance data of each computing node are obtained.

6. The method according to claim 1, wherein When tuning parameters, 20 transactions are executed serially. Each transaction calls a certain number of large models, and the next transaction will not be carried out until the previous transaction is processed.

7. The method according to claim 6, wherein Interpretation of key parameter tuning of large models for code deployed on vllm - Prefix Caching, when designing the experimental group, Benchmark group: 20 transactions executed serially, without --enable-prefix-caching prefix caching enabled; Control group: 20 transactions were executed serially, with --enable-prefix-caching enabled.

8. The method according to claim 7, wherein When Prefix Caching is enabled, for two independent requests, if the request token sequence of request 2 is mostly similar to that of request 1 and they share the same SystemPrompt, request 2 reuses the physical block generated by request 1 in the prefill phase.

9. The method according to claim 8, wherein When reusing physical blocks, Prefix Caching is completely consistent starting from the first token, the sequence calculated starting from the first token is consistent, and the corresponding hash values ​​are consistent.

10. The method according to claim 8 or 9, characterized in that When evaluating the rate of the first token, set the max_new_token parameter to 1 so that vllm only generates one token. When evaluating the rate of non-first tokens, use the fastchat client to calculate the token generation rate.

Images