Configuration parameter optimization method and optimization device of large language model inference engine

Abstract

This application discloses a method and apparatus for optimizing configuration parameters of a large language model inference engine. After determining the configuration parameters of a predetermined server cluster and the historical load tracing data of the large language model inference engine, this application determines the target load tracing data for the cluster based on the historical load tracing data. Then, in each cycle, the values ​​of the configuration parameters are updated based on the performance evaluation results of the cluster in the previous cycle until the iteration terminates, thereby determining the final values ​​of the configuration parameters. In this embodiment, the target load tracing data is determined based on the matching degree with the load feature vector of the predetermined server cluster. By pre-playing back the target load tracing data and recording the playback parameters, in subsequent configuration parameter adjustments, only a portion of the target load tracing data needs to be played back based on the playback parameters. Therefore, this embodiment can effectively improve the performance of the inference engine by optimizing the configuration parameters, while reducing the consumption of computing resources for parameter optimization.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a method and apparatus for optimizing configuration parameters of a large language model inference engine. Background Technology

[0002] With the continuous development of computer technology, large model inference services are becoming increasingly popular in daily life, work, and other fields. In practical applications, large model inference services are usually deployed to different server clusters according to workload type to form dedicated clusters suitable for different workload types, such as chat clusters, coder clusters, and thinking clusters.

[0003] To improve the inference performance of large language model inference engines across different server clusters, different configuration parameters are typically set for the engine on each cluster. Existing optimization methods for these parameters are usually based on grid search or manual tuning, but these methods lack universality and it is difficult to guarantee the impact of configuration parameter optimization on the inference performance of the large language model inference engine. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method and apparatus for optimizing configuration parameters of a large language model inference engine. The method updates the values ​​of configuration parameters iteratively based on the performance evaluation results of the server cluster in the previous period, and stops replaying when the load feature vector of the replayed load tracing data matches the baseline load feature vector. This effectively improves the performance of the inference engine, reduces the consumption of computing resources by parameter optimization, and enhances the universality of parameter optimization.

[0005] In a first aspect, embodiments of the present invention provide a method for optimizing configuration parameters of a large language model inference engine, the method comprising: The parameters to be adjusted for the predetermined server cluster and the historical load tracing data of the large language model inference engine are determined, wherein the historical load tracing data is determined based on multiple historical requests and the arrival time of each historical request; The target load tracing data of the predetermined server cluster is determined based on the historical load tracing data. The second load feature vector of the target load tracing data is matched with the first load feature vector corresponding to the predetermined server cluster. The first load feature vector includes the request length feature, key-value cache hit feature, and request arrival feature corresponding to the historical load tracing data. The second load feature vector includes the request length feature, key-value cache hit feature, and request arrival feature corresponding to the target load tracing data. The target load tracing data is continuously replayed, and the third load feature vector of the replayed tracing data is calculated until the third load feature vector matches the first load feature vector. The third load feature vector includes the request length feature, the key-value cache hit feature, and the request arrival feature corresponding to the replayed tracing data. The replay parameters when the matching condition is met are recorded. In each cycle, the value of the parameter to be adjusted is updated based on the performance evaluation results of the predetermined server cluster in the previous cycle, until the preset iteration termination condition is met. The value of the parameter to be adjusted in the last cycle is determined as the target value corresponding to the parameter to be adjusted; The step of updating the value of the parameter to be adjusted in each period based on the performance evaluation results of the predetermined server cluster in the previous period includes: According to the value of the parameter to be adjusted in the current period, the corresponding part of the target load tracking data is replayed according to the predetermined replay parameters, and the performance parameters of the large language model inference engine are recorded. The performance evaluation result of the predetermined server cluster in the current period is determined based on the performance parameters recorded at the end of the playback. The value of the parameter to be adjusted in the next period is updated based on the value of the parameter to be adjusted in the current period and the performance evaluation result.

[0006] Optionally, determining the target load tracing data for the predetermined server cluster based on the historical load tracing data includes: The historical load tracing data is sampled to obtain multiple candidate load tracing data. The candidate load tracing data that matches the fourth load feature vector with the first load feature vector is determined as the target load tracing data. The fourth load feature vector includes the request length feature, the key-value cache hit feature, and the request arrival feature corresponding to the candidate load tracing data.

[0007] Optionally, updating the value of the parameter to be adjusted in the next period based on the value of the parameter to be adjusted in the current period and the performance evaluation result includes: Based on the deployment status of the predetermined server cluster, the value of the parameter to be adjusted in the current period, the performance evaluation result of the previous period, and the optimization rules of the parameter to be adjusted, an optimization scheme for the parameter to be adjusted is generated based on a large language model. The optimization scheme is used to update the value of the parameter to be adjusted in the next period.

[0008] Optionally, the request length feature is determined based on the statistical value of the request length of each of the historical requests, the key-value cache hit feature is determined based on the statistical value of the key-value cache hit length and the statistical value of the key-value cache hit rate of each of the historical requests, and the request arrival feature is determined based on the statistical value of the average arrival rate per graphics processor, the statistical value of the arrival time interval, and the statistical value of the arrival count per second of each of the historical requests.

[0009] Optionally, the key-value cache hit feature corresponding to the replayed tracking data is obtained by bias sampling of the key-value cache hit rate corresponding to the replayed tracking data through a sliding window.

[0010] Optionally, the preset iteration termination condition is specifically: The iteration period of the parameter to be adjusted reaches a preset period threshold.

[0011] Secondly, embodiments of the present invention provide a configuration parameter optimization device for a large language model inference engine, the device comprising: The first data determination unit is used to determine the parameters to be adjusted for the predetermined server cluster and the historical load tracking data of the large language model inference engine. The historical load tracking data is determined based on multiple historical requests and the arrival time of each historical request. The second data determination unit is used to determine the target load tracing data of the predetermined server cluster based on the historical load tracing data. The second load feature vector of the target load tracing data is matched with the first load feature vector corresponding to the predetermined server cluster. The first load feature vector includes the request length feature, key-value cache hit feature, and request arrival feature corresponding to the historical load tracing data. The second load feature vector includes the request length feature, key-value cache hit feature, and request arrival feature corresponding to the target load tracing data. The replay parameter determination unit is used to continuously replay the target load tracing data and calculate the third load feature vector of the replayed tracing data until the third load feature vector matches the first load feature vector. The third load feature vector includes the request length feature, the key-value cache hit feature and the request arrival feature corresponding to the replayed tracing data. The replay parameters are recorded when the matching condition is met. The parameter update unit is used to update the value of the parameter to be adjusted in each cycle according to the performance evaluation result of the predetermined server cluster in the previous cycle, until the preset iteration termination condition is met. The parameter determination unit is used to determine the value of the parameter to be adjusted in the last cycle as the target value corresponding to the parameter to be adjusted. The parameter update unit is further used for: According to the value of the parameter to be adjusted in the current period, the corresponding part of the target load tracking data is replayed according to the predetermined replay parameters, and the performance parameters of the large language model inference engine are recorded. The performance evaluation result of the predetermined server cluster in the current period is determined based on the performance parameters recorded at the end of the playback. The value of the parameter to be adjusted in the next period is updated based on the value of the parameter to be adjusted in the current period and the performance evaluation result.

[0012] Thirdly, embodiments of the present invention provide an electronic device including a memory and a processor, the memory being used to store one or more computer program instructions, the processor including a network chip, wherein the one or more computer program instructions are executed by the network chip to implement the method as described in any one of the first aspects.

[0013] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any one of the first aspects.

[0014] Fifthly, embodiments of the present invention provide a computer program product comprising a computer program / instruction, which, when executed by a processor, implements the method as described in any one of the first aspects.

[0015] In this embodiment of the invention, after determining the configuration parameters of a predetermined server cluster and the historical load tracing data of the inference engine for a large language model, the target load tracing data for the cluster is determined based on the historical load tracing data. Then, in each cycle, the values ​​of the configuration parameters are updated based on the performance evaluation results of the cluster in the previous cycle until the iteration terminates, thereby determining the final values ​​of the configuration parameters. In this embodiment, the target load tracing data is determined based on the matching degree with the load feature vector of the predetermined server cluster. The target load tracing data is pre-replayed, and replay parameters are recorded when the replayed portion of the load feature vector matches the aforementioned load feature vector. During subsequent configuration parameter adjustments, only a portion of the target load tracing data needs to be replayed based on the replay parameters. Therefore, this embodiment of the invention can effectively improve the performance of the inference engine by optimizing configuration parameters, while reducing the consumption of computational resources for parameter optimization, thereby improving the universality of parameter optimization. Attached Figure Description

[0016] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a flowchart of the configuration parameter optimization method for the large language model inference engine according to an embodiment of the present invention; Figure 2 This is a flowchart of the configuration parameter optimization method for the large language model inference engine according to an embodiment of the present invention; Figure 3 This is a flowchart of the configuration parameter optimization method for the large language model inference engine according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the playback process of target load tracking data according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the configuration parameter optimization device for the large language model inference engine according to an embodiment of the present invention; Figure 6 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0017] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0018] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0019] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0020] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0021] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0022] Existing configuration parameter optimization methods mainly achieve the following: 1. Grid search-based optimization. However, grid search exhaustively searches within the adjustable range of configuration parameters, thus consuming excessive computational resources and making it difficult to guarantee the impact of configuration parameter optimization on the inference performance of large language model inference engines. 2. Manual parameter tuning. However, manual tuning relies heavily on the engineer's experience, lacks universality, and the engineer's experience has a significant impact on the optimization results, again making it difficult to guarantee the impact of configuration parameter optimization on the inference performance of large language model inference engines. 3. SCOOT (Service Level Objective-Oriented Performance Tuning for LLMInference Engines), a performance tuning system for large model inference engines based on service level objective constraints. SCOOT uses a Gaussian Process Surrogate Model to fit the mapping relationship between configuration parameters and the inference performance of the large language model inference engine, and selects a set of candidate configuration parameters from the value space through Bayesian optimization. However, this approach may generate a large number of invalid configuration parameters, waste computing resources, reduce the optimization efficiency of configuration parameters, and make it difficult to guarantee the reliability of configuration parameter optimization.

[0023] To address the aforementioned issues, this invention proposes a method and apparatus for optimizing configuration parameters of a large language model inference engine. This method iteratively updates the values ​​of configuration parameters based on the performance evaluation results of the server cluster in the previous period, and stops replaying when the load feature vector of the replayed load tracing data matches the baseline load feature vector. This effectively improves the performance of the inference engine while reducing the computational resource consumption of parameter optimization, thereby enhancing the universality of parameter optimization. The following is a description of the method implementation. Figure 1 This is a flowchart of the configuration parameter optimization method for the large language model inference engine according to an embodiment of the present invention. Figure 1 As shown, the method in this embodiment may include the following steps: Step S100: Determine the parameters to be adjusted for the predetermined server cluster and the historical load tracking data of the large language model inference engine.

[0024] In this step, a predetermined server cluster can be determined through various methods, and historical load tracing data of the large language model inference engine deployed on that server cluster can be obtained. Simultaneously, the configuration parameters of the server cluster can be acquired as parameters to be adjusted. In this embodiment, the historical load tracing data is the actual trace data generated by the large language model inference engine during online operation. It is determined based on multiple historical requests and the arrival time of each historical request, and may further include multiple historical requests and all downstream call requests corresponding to those requests. The configuration parameters may include at least one of the following: tensor parallelism of the inference engine, block pre-filling (including total number of switches, block granularity parameters, long request threshold parameters, etc.), long pre-filling threshold, and memory utilization parameters.

[0025] In the following description, the load feature vector refers to a vector used to characterize the workload features of the inference engine of a large language model. The first load feature vector refers to the load feature vector of historical load tracing data, which includes the request length feature, key-value cache hit feature, and request arrival feature corresponding to the historical load tracing data.

[0026] The second load feature vector refers to the load feature vector of the target load tracing data, which includes the request length feature, the key-value cache hit feature, and the request arrival feature corresponding to the target load tracing data.

[0027] The third load feature vector refers to the load feature vector of the replayed trace data, which includes the request length feature, the key-value cache hit feature, and the request arrival feature corresponding to the replayed trace data.

[0028] The fourth load feature vector refers to the load feature vector of the candidate load tracing data, which includes the request length feature, the key-value cache hit feature, and the request arrival feature of the candidate load tracing data.

[0029] In practical applications, the inference performance of a large language model's inference engine is primarily determined by three workload characteristics. The first dimension is request length. Request length characterizes the length of the input and output tokens for a single request. It determines the pre-fill computation and decoding iterations required by the inference engine, thus influencing the computational cost per request. Under the same service level objectives (e.g., single inference time less than or equal to 200 milliseconds, processor usage not exceeding limits), a longer request length results in higher computational cost per request, leading to poorer inference engine performance. The second dimension is key-value cache hit rate. Key-value cache hit rate characterizes the degree of prefix sharing between different requests. It determines the key-value cache reuse rate. Under the same service level objectives, a lower key-value cache reuse rate results in more pre-fill operations, higher computational and memory overhead, and consequently, poorer inference engine performance. The third feature dimension is the request arrival feature, which reflects the arrival rate and burstiness of different requests. The request arrival feature determines the batching behavior of the inference engine's continuous batching scheduler. Under the condition that the service level target remains unchanged, the greater the deviation of the request arrival rate from the reasonable arrival rate range that the inference engine can handle, the greater the difference in the arrival time distribution, the higher the difficulty in forming a suitable number of request batches, the lower the utilization rate of computing resources in batching, and therefore the worse the performance of the inference engine. The higher the burstiness of the requests, the higher the consumption of GPU memory by batching, and therefore the worse the performance of the inference engine.

[0030] The workloads of different server clusters vary, while the workload of the same server cluster is usually relatively stable over a period of time. Therefore, in some embodiments, load tracing data for each server cluster in the current period can be obtained at a predetermined period, and the load feature vector of the load tracing data can be obtained. Then, it can be determined whether the load feature vector of the current period matches a first load feature vector used as a benchmark. For any server cluster, if the two load feature vectors match, it indicates that the workload of the server cluster is relatively stable; if the two load feature vectors do not match, it indicates that the workload of the server cluster has changed, and therefore the server cluster can be identified as a predetermined server cluster.

[0031] The load feature vector can be determined based on the request length characteristics, key-value cache hit characteristics, and request arrival characteristics of each request in the load tracing data. To improve the ability of the load feature vector to characterize the workload of the server cluster, in some embodiments, the request length characteristic can be determined based on the statistical value of the request length of each request in the load tracing data, the key-value cache hit characteristic can be determined based on the statistical value of the key-value cache hit length and the key-value cache hit rate of each request, and the request arrival characteristic can be determined based on the statistical value of the average arrival rate per GPU, the statistical value of the arrival time interval, and the statistical value of the arrival count per second for each request.

[0032] The key-value cache hit rate can be determined by the ratio of the number of terms that a single request hits the key-value cache (i.e., the key-value cache hit length) to the total number of terms that need to be pre-filled; the average arrival rate per graphics processor can be determined by the ratio of the total number of requests arriving at the predetermined server cluster per unit time to the number of graphics processors. In some embodiments, the sub-vector corresponding to the request length feature of workload W This can be expressed by the following formula:

[0033] in, Indicates the request length. This represents the mean. Represents the 95th percentile. This represents the coefficient of variation. Wherein, the coefficient of variation... It can be calculated based on the ratio of the standard deviation to the mean.

[0034] Subvectors corresponding to the key-value cache hit features of workload W This can be expressed by the following formula:

[0035] in, This indicates the key-value cache hit length for each request. This indicates the overall key-value cache hit rate for each request.

[0036] The subvector corresponding to the arrival feature of the workload W This can be expressed by the following formula:

[0037] Where λ represents the average arrival rate of a single graphics processor. Indicates the time interval between requests. This represents the Fano factor for arrivals per second. The Fano factor for arrivals per second is used to characterize the suddenness that the coefficient of variation cannot reflect, and it can be calculated based on the ratio of the variance to the mean of the arrivals per second.

[0038] The load feature vector can be obtained by concatenating the subvectors corresponding to the request length feature, the key-value cache hit feature, and the request arrival feature. In other words, the load feature vector of workload W. This can be expressed by the following formula:

[0039] This embodiment analyzes various statistical values. The change allows the load feature vector to reflect the relative differences between statistical values, thereby balancing the importance of a small number of statistical changes (such as the number of morphemes changing from 1,000 to 2,000) with a large number of statistical changes (such as the number of morphemes changing from 32,000 to 42,000).

[0040] The first load feature vector can be determined in several ways. For example, it can be determined based on the load tracing data of the predetermined server cluster in the previous period. It's easy to understand that if the load feature vector of the current period does not match the first load feature vector, the load feature vector of the current period can be updated to the first load feature vector.

[0041] Matching the load feature vector of the current period with the load feature vector of the previous period means that the similarity between the load feature vector of the current period and the load feature vector of the previous period meets a preset similarity condition. The preset similarity condition can be determined according to the similarity calculation method. For example, if the similarity calculation method is cosine similarity, then the preset similarity condition can be set to the cosine similarity between the load feature vector of the current period and the load feature vector of the previous period being greater than (or greater than or equal to) a preset similarity threshold.

[0042] The similarity of load feature vectors for different workloads can be calculated using the following formula:

[0043] Where W and W' represent different workloads, This represents the normalized load feature vector. This represents the L2 norm (also known as the Euclidean norm).

[0044] Step S200: Determine the target load tracing data for the predetermined server cluster based on historical load tracing data.

[0045] In this embodiment, historical load tracing data is used to perform performance evaluation of the inference engine. In order to improve the performance evaluation efficiency of the inference engine, reduce the performance evaluation cost of the inference engine, and reduce the negative impact of noisy data on the evaluation results of the inference engine, the historical load tracing data can be sampled to obtain the target load tracing data of the predetermined server cluster.

[0046] Figure 2 This is a flowchart of the configuration parameter optimization method for the large language model inference engine according to an embodiment of the present invention. Figure 2 As shown, in an optional implementation of this embodiment, step S200 may include the following steps: Step S210: Sample the historical load tracing data to obtain multiple candidate load tracing data.

[0047] In this step, historical load tracing data can be sampled in various ways to obtain multiple samples as candidate load tracing data. Optionally, historical load tracing data can be truncated within a predetermined time window, and each truncated segment of historical load tracing data can be designated as a candidate load tracing data.

[0048] Step S220: The candidate load tracking data that matches the fourth load feature vector with the first load feature vector is determined as the target load tracking data.

[0049] In this step, the fourth load feature vector of each candidate load tracing data can be calculated, and it can be determined whether each fourth load feature vector matches the first load feature vector. For any candidate load tracing data, if the corresponding fourth load feature vector matches the first load feature vector, the candidate load tracing data can be determined as the target load tracing data.

[0050] In this embodiment, the calculation method of the fourth load feature vector can refer to the calculation method of the load feature vector described above, and will not be repeated here.

[0051] Step S300: Continuously replay the target load tracing data and calculate the third load feature vector of the replayed tracing data until the third load feature vector matches the first load feature vector. The third load feature vector includes the request length feature, the key-value cache hit feature, and the request arrival feature corresponding to the replayed tracing data. Record the replay parameters when the matching condition is met.

[0052] In practical applications, the actual workload of the inference engine can be simulated by replaying load trace data, thereby enabling performance evaluation of the inference engine. The reliability of the performance evaluation depends on the similarity in feature space between the load feature vector of the replayed load trace data and the load feature vector used as a benchmark. In other words, if the similarity between the load feature vector of the replayed load trace data and the load feature vector used as a benchmark meets a preset similarity condition, the accurate performance evaluation of the inference engine can be achieved simply by replaying the load trace data for a short period of time.

[0053] Therefore, in this step, the value of the parameter to be adjusted in the current cycle can be determined in each cycle, and the target load tracking data can be continuously replayed under this value. During the replay of the target load tracking data, the third load feature vector of the replayed target load tracking data (i.e., the replayed tracking data) is calculated. Then, when the third load feature vector of the replayed target tracking data matches the first load feature vector, the replay of the target load tracking data is stopped in the current cycle. At the same time, the relevant parameters of the replayed target tracking data, such as time and the part involved, are recorded when the above matching condition is met.

[0054] After recording the playback parameters, the target load tracking data can be replayed in subsequent sessions with the same time length and the parts involved, thus enabling accurate evaluation by the inference engine.

[0055] It should be understood that this step does not require online execution in a production environment. Instead, it can be performed in a CPU-simulated environment. There's no need to record performance data; instead, the corresponding third load feature vector is obtained and matched against the first load feature vector. After determining the playback parameters, subsequent online playback of the corresponding data in a production environment can be used to refine the parameter conditions.

[0056] Figure 4 This is a schematic diagram illustrating the playback process of target load tracking data according to an embodiment of the present invention. Figure 4As shown, data 41 is the target load tracing data corresponding to the predetermined server cluster. At time t1, the target load tracing data that has been replayed in the Tth period can be determined as data 42, and the third load feature vector of data 42, i.e., feature vector 43, is calculated. Then, the similarity between feature vector 43 and the first load feature vector, i.e., feature vector 45, can be calculated, and it is determined that the similarity is less than θ, i.e., the preset similarity condition is not met, so data 41 continues to be replayed. At time t2, the target load tracing data that has been replayed can be determined as data 42', and the third load feature vector of data 42', i.e., feature vector 43', is calculated. Then, the similarity between feature vector 43' and feature vector 45 can be calculated, and it is determined that the similarity is greater than or equal to θ, so the replay can be stopped. At the same time, the replay parameters are determined or a portion of the target load tracing data that has been replayed is recorded based on the target load tracing data that has already been replayed.

[0057] It should be understood that Figure 4 This is merely an illustrative diagram. In this embodiment, continuous playback can be performed from multiple different time points of the target load tracking data to select more suitable playback parameters for use in subsequent step S400. In step S400, the values ​​of the parameters to be adjusted are updated in each cycle based on the performance evaluation results of the predetermined server cluster in the previous cycle until the preset iteration termination condition is met.

[0058] In practical applications, the performance of the inference engine is affected by different configuration parameter values. Therefore, in this step, the values ​​of the parameters to be adjusted can be updated through multiple iterations based on the performance evaluation results of the predetermined server cluster under different configuration parameter values, so as to optimize the configuration parameters.

[0059] In this embodiment, the preset iteration termination condition can be set according to actual needs. For example, it can be set to the performance evaluation result of a predetermined server cluster reaching a service level target. In some embodiments, to avoid excessive consumption of computing resources and improve the optimization efficiency of configuration parameters, the preset iteration termination condition can be set to the iteration period of the parameter to be adjusted reaching a preset period threshold. The preset period threshold can be set to a small value, such as 10 times.

[0060] Figure 3 This is a flowchart of the configuration parameter optimization method for the large language model inference engine according to an embodiment of the present invention. Figure 3 As shown, in an optional implementation of this embodiment, step S300 may include the following steps: Step S310: According to the value of the parameter to be adjusted in the current period, replay the corresponding part of the target load tracking data according to the predetermined replay parameters, and record the performance parameters of the large language model inference engine.

[0061] The performance parameters of the inference engine may include at least one of the following: Peak Sustainable Request Rate (PSR), Time To First Token (TTFT), and Time Per Output Token (TPOT).

[0062] In some embodiments, to further improve the reliability of the inference engine's performance evaluation, the decision to stop replaying the target load tracking data can be based on the rate of change of the performance parameters. Optionally, when the rate of change of the performance parameters at the current performance evaluation time meets a preset rate of change condition, such as being less than (or less than or equal to) a preset rate of change threshold, it can be determined that the change of the performance parameters tends to be stable, and the replay of the target load tracking data can be stopped when the third load feature vector at the current performance evaluation time matches the first load feature vector.

[0063] In practical applications, the performance evaluation results of cache-intensive workloads can be reduced due to prefix overlap between missing requests, thereby diminishing the optimization effect of configuration parameters. Therefore, in some embodiments, a sliding window can be used to bias-sample the key-value cache hit rate corresponding to the replay trace data, thus obtaining the key-value cache hit characteristics corresponding to the replay trace data. Bias sampling is a method that adjusts the sampling frequency of data (such as key-value cache hit rate) within a time window based on the importance of the data; the higher the importance of the data, the higher the sampling weight and the higher the probability of it being sampled. Therefore, this method can restore the prefix overlap between requests to its true level, effectively compensating for the reduced optimization effect of configuration parameters caused by prefix overlap between missing requests.

[0064] Step S320: Determine the performance evaluation result of the predetermined server cluster in the current period based on the performance parameters recorded at the end of the playback.

[0065] After the target load tracing data replay is complete, the performance parameters recorded at the end of the replay can be used as the performance evaluation results of the predetermined server cluster in the current period. This method effectively reduces the replay time of the target load tracing data, thereby significantly improving the efficiency of configuration parameter optimization.

[0066] Step S330: Update the value of the parameter to be adjusted in the next period based on the value of the parameter to be adjusted in the current period and the performance evaluation results.

[0067] In this step, the values ​​of the configuration parameters in the current period and the performance evaluation results can be used as optimization guidelines for the configuration parameters to obtain their values ​​in the next period. Optionally, in the first period, the values ​​of the parameters to be adjusted set by the system can be obtained as the values ​​of the parameters to be adjusted in the current period.

[0068] In some embodiments, an optimization scheme for the parameter to be adjusted can be generated based on a large language model according to the deployment status of the predetermined server cluster, the performance evaluation results of the previous period, the value of the parameter to be adjusted in the current period, and the optimization rules, so as to update the value of the parameter to be adjusted in the next period through the optimization scheme.

[0069] The deployment status of the pre-defined server cluster may include at least one of the following: operating environment parameters, hardware parameters, identifier of the deployment model, workload, and service level objectives of the pre-defined server cluster.

[0070] Optimization rules are used to map the performance bottleneck state of a predetermined server cluster to the configuration parameter combination most likely to solve the problem, and to define protection conditions for other system degradation problems caused by changes in configuration parameter values. For example, when the configuration parameter is tensor parallelism, the optimization rules may include: probing the pre-filling effect of adjacent tensor parallelism values ​​in the active execution environment; based on the pre-filling-latency slope, queue latency growth, and high tensor parallelism queuing inflection point, combined with information such as the actual pre-filling length per request, prefix hit rate, burst pre-filling load of a single graphics processor, number of active replicas, and service level target gap of first packet latency, projecting each tensor parallelism value onto a unified boundary for comparison and decision-making.

[0071] In practical applications, the request length characteristics of the pre-defined server cluster affect the values ​​of configuration parameters such as tensor parallelism, block pre-filling, and long pre-filling threshold; the key-value cache hit characteristics of the pre-defined server cluster affect the values ​​of configuration parameters such as tensor parallelism, memory utilization, block size, and whether prefix caching is enabled; and the request arrival characteristics affect the values ​​of configuration parameters such as scheduling delay factor (DP), maximum sequence number (MS), and maximum batch tokens (MBT). Therefore, in some embodiments, optimization rules can be determined based on the mapping relationship between workload characteristic types and configuration parameter types.

[0072] The optimization scheme for the parameters to be adjusted can include the performance bottleneck analysis results of the predetermined server cluster, the parameters to be adjusted in the current round, and their corresponding values. Therefore, by obtaining the optimization schemes for the parameters to be adjusted in each round, it is possible to accurately understand the optimization direction and reasons for the configuration parameters, effectively improving the transparency and interpretability of the optimization process.

[0073] Step S500: Determine the value of the parameter to be adjusted in the last cycle as the target value corresponding to the parameter to be adjusted.

[0074] After the preset iteration termination condition is met, the parameter to be adjusted will no longer be updated. Therefore, the value of the parameter to be adjusted in the last cycle can be determined as the target value of the parameter to be adjusted, thereby completing the optimization process of the configuration parameter.

[0075] In this embodiment of the invention, after determining the configuration parameters of a predetermined server cluster and the historical load tracing data of the inference engine for a large language model, the target load tracing data for the cluster is determined based on the historical load tracing data. Then, in each cycle, the values ​​of the configuration parameters are updated based on the performance evaluation results of the cluster in the previous cycle until the iteration terminates, thereby determining the final values ​​of the configuration parameters. In this embodiment, the target load tracing data is determined based on the matching degree with the load feature vector of the predetermined server cluster. The target load tracing data is pre-replayed, and replay parameters are recorded when the replayed portion of the load feature vector matches the aforementioned load feature vector. During subsequent configuration parameter adjustments, only a portion of the target load tracing data needs to be replayed based on the replay parameters. Therefore, this embodiment of the invention can effectively improve the performance of the inference engine by optimizing configuration parameters, while reducing the consumption of computational resources for parameter optimization, thereby improving the universality of parameter optimization.

[0076] Figure 5 This is a schematic diagram of the configuration parameter optimization device for the large language model inference engine according to an embodiment of the present invention. Figure 5 As shown, the configuration parameter optimization device of the large language model inference engine in this embodiment includes a first data determination unit 501, a second data determination unit 502, a playback parameter determination unit 503, a parameter update unit 504, and a parameter determination unit 505.

[0077] The first data determination unit 501 is used to determine the parameters to be adjusted for the predetermined server cluster and the historical load tracing data of the large language model inference engine. The historical load tracing data is determined based on multiple historical requests and the arrival time of each historical request. The second data determination unit 502 is used to determine the target load tracing data of the predetermined server cluster based on the historical load tracing data. The second load feature vector of the target load tracing data matches the first load feature vector corresponding to the predetermined server cluster. The first load feature vector includes the request length feature, key-value cache hit feature, and request arrival feature corresponding to the historical load tracing data. The second load feature vector includes the request length feature, key-value cache hit feature, and request arrival feature corresponding to the target load tracing data. The replay parameter determination unit 503 is used to continuously replay the target load tracing data and calculate the third load feature vector of the replayed tracing data until the third load feature vector matches the first load feature vector. The third load feature vector includes the request length feature, the key-value cache hit feature, and the request arrival feature corresponding to the replayed tracing data. The replay parameter is recorded when the matching condition is met. The parameter update unit 504 is used to update the value of the parameter to be adjusted in each period according to the performance evaluation result of the predetermined server cluster in the previous period until the preset iteration termination condition is met. The parameter determination unit 505 is used to determine the value of the parameter to be adjusted in the last period as the target value corresponding to the parameter to be adjusted. The parameter update unit 504 is further used for: Based on the value of the parameter to be adjusted in the current period, the corresponding part of the target load tracing data is replayed according to the pre-determined replay parameters, and the performance parameters of the large language model inference engine are recorded; the performance evaluation result of the predetermined server cluster in the current period is determined based on the performance parameters recorded at the end of the replay; the value of the parameter to be adjusted in the next period is updated based on the value of the parameter to be adjusted in the current period and the performance evaluation result.

[0078] In this embodiment of the invention, after determining the configuration parameters of a predetermined server cluster and the historical load tracing data of the inference engine for a large language model, the target load tracing data for the cluster is determined based on the historical load tracing data. Then, in each cycle, the values ​​of the configuration parameters are updated based on the performance evaluation results of the cluster in the previous cycle until the iteration terminates, thereby determining the final values ​​of the configuration parameters. In this embodiment, the target load tracing data is determined based on the matching degree with the load feature vector of the predetermined server cluster. The target load tracing data is pre-replayed, and replay parameters are recorded when the replayed portion of the load feature vector matches the aforementioned load feature vector. During subsequent configuration parameter adjustments, only a portion of the target load tracing data needs to be replayed based on the replay parameters. Therefore, this embodiment of the invention can effectively improve the performance of the inference engine by optimizing configuration parameters, while reducing the consumption of computational resources for parameter optimization, thereby improving the universality of parameter optimization.

[0079] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of the present invention. In this embodiment, the electronic device 6 includes a server, a terminal, etc. Figure 6 As shown, the electronic device 6 includes at least one processor 601; a memory 602 communicatively connected to at least one processor 601; and a communication component 603 communicatively connected to the processor 601, the communication component 603 receiving and sending data under the control of the processor 601; wherein the memory 602 stores instructions executable by at least one processor 601, the instructions being executed by at least one processor 601 to implement the configuration parameter optimization method of the large language model inference engine described above.

[0080] Specifically, the electronic device includes: one or more processors 601 and a memory 602. Figure 6 Taking a processor 601 as an example, the processor 601 and the memory 602 can be connected via a bus or other means. Figure 6 Taking a bus connection as an example, memory 602, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Processor 601 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions, and modules stored in memory 602, thereby implementing the configuration parameter optimization method for the aforementioned large language model inference engine.

[0081] Memory 602 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; the data storage area may store an option list, etc. Furthermore, memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, memory 602 may optionally include memory remotely located relative to processor 601, and these remote memories may be connected to external devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0082] One or more modules are stored in memory 602, and when executed by one or more processors 601, they execute the configuration parameter optimization method of the large language model inference engine in any of the above method embodiments.

[0083] The above-mentioned products can perform the methods provided in the embodiments of this application, and have the corresponding functional modules and beneficial effects of performing the methods. For technical details not described in detail in this embodiment, please refer to the methods provided in the embodiments of this application.

[0084] In this embodiment of the invention, after determining the configuration parameters of a predetermined server cluster and the historical load tracing data of the inference engine for a large language model, the target load tracing data for the cluster is determined based on the historical load tracing data. Then, in each cycle, the values ​​of the configuration parameters are updated based on the performance evaluation results of the cluster in the previous cycle until the iteration terminates, thereby determining the final values ​​of the configuration parameters. Thus, this embodiment of the invention can effectively improve the performance of the inference engine by optimizing configuration parameters, while reducing the consumption of computing resources for parameter optimization, thereby improving the universality of parameter optimization.

[0085] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0086] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0087] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for optimizing configuration parameters of a large language model inference engine, characterized in that, The method includes: The parameters to be adjusted for the predetermined server cluster and the historical load tracing data of the large language model inference engine are determined, wherein the historical load tracing data is determined based on multiple historical requests and the arrival time of each historical request; The target load tracing data of the predetermined server cluster is determined based on the historical load tracing data. The second load feature vector of the target load tracing data is matched with the first load feature vector corresponding to the predetermined server cluster. The first load feature vector includes the request length feature, key-value cache hit feature, and request arrival feature corresponding to the historical load tracing data. The second load feature vector includes the request length feature, key-value cache hit feature, and request arrival feature corresponding to the target load tracing data. The target load tracing data is continuously replayed, and the third load feature vector of the replayed tracing data is calculated until the third load feature vector matches the first load feature vector. The third load feature vector includes the request length feature, the key-value cache hit feature, and the request arrival feature corresponding to the replayed tracing data. The replay parameters when the matching condition is met are recorded. In each cycle, the value of the parameter to be adjusted is updated based on the performance evaluation results of the predetermined server cluster in the previous cycle, until the preset iteration termination condition is met. The value of the parameter to be adjusted in the last cycle is determined as the target value corresponding to the parameter to be adjusted; The step of updating the value of the parameter to be adjusted in each period based on the performance evaluation results of the predetermined server cluster in the previous period includes: According to the value of the parameter to be adjusted in the current period, the corresponding part of the target load tracking data is replayed according to the predetermined replay parameters, and the performance parameters of the large language model inference engine are recorded. The performance evaluation result of the predetermined server cluster in the current period is determined based on the performance parameters recorded at the end of the playback. The value of the parameter to be adjusted in the next period is updated based on the value of the parameter to be adjusted in the current period and the performance evaluation result.

2. The method according to claim 1, characterized in that, Determining the target load tracing data for the predetermined server cluster based on the historical load tracing data includes: The historical load tracing data is sampled to obtain multiple candidate load tracing data. The candidate load tracing data that matches the fourth load feature vector with the first load feature vector is determined as the target load tracing data. The fourth load feature vector includes the request length feature, the key-value cache hit feature, and the request arrival feature corresponding to the candidate load tracing data.

3. The method according to claim 1, characterized in that, The step of updating the value of the parameter to be adjusted in the next period based on the value of the parameter to be adjusted in the current period and the performance evaluation result includes: Based on the deployment status of the predetermined server cluster, the value of the parameter to be adjusted in the current period, the performance evaluation result of the previous period, and the optimization rules of the parameter to be adjusted, an optimization scheme for the parameter to be adjusted is generated based on a large language model. The optimization scheme is used to update the value of the parameter to be adjusted in the next period.

4. The method according to claim 1, characterized in that, The request length feature is determined based on the statistical value of the request length of each of the historical requests. The key-value cache hit feature is determined based on the statistical value of the key-value cache hit length and the statistical value of the key-value cache hit rate of each of the historical requests. The request arrival feature is determined based on the statistical value of the average arrival rate per graphics processor, the statistical value of the arrival time interval, and the statistical value of the arrival count per second of each of the historical requests.

5. The method according to claim 1, characterized in that, The key-value cache hit features corresponding to the replayed tracking data are obtained by bias sampling of the key-value cache hit rate corresponding to the replayed tracking data through a sliding window.

6. The method according to claim 1, characterized in that, The preset iteration termination condition is specifically as follows: The iteration period of the parameter to be adjusted reaches a preset period threshold.

7. A configuration parameter optimization device for a large language model inference engine, characterized in that, The device includes: The first data determination unit is used to determine the parameters to be adjusted for the predetermined server cluster and the historical load tracking data of the large language model inference engine. The historical load tracking data is determined based on multiple historical requests and the arrival time of each historical request. The second data determination unit is used to determine the target load tracing data of the predetermined server cluster based on the historical load tracing data. The second load feature vector of the target load tracing data is matched with the first load feature vector corresponding to the predetermined server cluster. The first load feature vector includes the request length feature, key-value cache hit feature, and request arrival feature corresponding to the historical load tracing data. The second load feature vector includes the request length feature, key-value cache hit feature, and request arrival feature corresponding to the target load tracing data. The replay parameter determination unit is used to continuously replay the target load tracing data and calculate the third load feature vector of the replayed tracing data until the third load feature vector matches the first load feature vector. The third load feature vector includes the request length feature, the key-value cache hit feature and the request arrival feature corresponding to the replayed tracing data. The replay parameters are recorded when the matching condition is met. The parameter update unit is used to update the value of the parameter to be adjusted in each cycle according to the performance evaluation result of the predetermined server cluster in the previous cycle, until the preset iteration termination condition is met. The parameter determination unit is used to determine the value of the parameter to be adjusted in the last cycle as the target value corresponding to the parameter to be adjusted. The parameter update unit is further used for: According to the value of the parameter to be adjusted in the current period, the corresponding part of the target load tracking data is replayed according to the predetermined replay parameters, and the performance parameters of the large language model inference engine are recorded. The performance evaluation result of the predetermined server cluster in the current period is determined based on the performance parameters recorded at the end of the playback. The value of the parameter to be adjusted in the next period is updated based on the value of the parameter to be adjusted in the current period and the performance evaluation result.

8. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, and the processor includes a network chip, wherein the one or more computer program instructions are executed by the network chip to implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program / instruction that, when executed by a processor, implements the method as described in any one of claims 1-6.

Images