Parameter tuning method and device for large model inference engine
Through Bayesian optimization algorithm and feasibility prediction model, combined with multi-objective optimization and parallel evaluation, the problems of complex parameter configuration and large tuning overhead of large-model inference engine are solved, and performance optimization and cost reduction are achieved.
Patent Information
- Application Number
- CN202411068588.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-12
AI Technical Summary
The parameter configuration of the big model inference engine is complex, which leads to performance tuning difficulties caused by different optimization goals, and there are problems of hidden constraints and high-tuning overhead.
The Bayesian optimization algorithm (BO algorithm) is used to combine multi-objective optimization and feasibility prediction model, and the parameter configuration is optimized through sampling and proxy models, adapt to different optimization goals, and evaluate parameter groups in parallel to reduce tuning time.
Accurately and efficiently determine the best performance or close to the optimal configuration parameters, reduce the computing resources and time-consuming of the inference engine processing requests, reduce the average cost, avoid hidden constraints, and significantly reduce tuning time.
Smart Images

Figure CN118940843A_ABST
Abstract
Description
Technical Field
[0001] One or more embodiments of the present specification relate to the field of computer technology, and in particular, to a parameter tuning method and device for a large model inference engine. Background Art
[0002] Large models mainly refer to large language models (LLMs). Due to their outstanding performance in many prediction tasks, they have been used by many service providers (or service provider applications) in order to provide users with high-quality services.
[0003] However, large models have a huge number of parameters, which can reach hundreds of billions or even billions, which leads to very high inference costs. For example, in order to accommodate such a large number of parameters, a high-end GPU with large video memory or multiple high-end GPUs is required, which brings very high equipment purchase and operation and maintenance costs. For another example, due to LLM's unique autoregressive generation mechanism and key-value pair caching mechanism, the GPU's computing efficiency is low during LLM inference.
[0004] In response to this, the industry proposes to use a large model inference engine (hereinafter referred to as an inference engine) to optimize the model inference process, so that under the same computing resources, the large model can be used to process more inference requests initiated by the service application, thereby reducing the average cost of each request. Specifically, the inference engine is a software system specifically used to load and run large model inference, such as Figure 1 As shown in the figure, the upper-layer application initiates an inference request to the inference engine, and the large model inference engine schedules the large model to process the inference request. For example, the content is extracted from the inference request as the input of the large model, and the output of the large model is used as the response data or processing result of the inference request, and then fed back to the upper-layer application. For example, in the process of scheduling the large model, the inference engine can use continuous batching, paged attention, and chunked prefill to improve the inference speed and throughput of the LLM model.
[0005] Large model inference engines usually have some parameters that need to be configured, such as whether to enable paging attention, whether to enable block padding, block size, maximum number of characters to be processed in batches, etc. The configuration of these parameters directly affects the model inference performance.
[0006] Therefore, a parameter tuning solution is needed to determine the optimal or near-optimal parameter configuration, thereby improving or even maximizing the performance of the inference engine and effectively reducing the processing cost of inference requests. Summary of the invention
[0007] The embodiments of this specification describe a parameter tuning method and device for a large model inference engine, which can determine an optimal or near-optimal parameter configuration.
[0008] According to a first aspect, a parameter tuning method for a large model inference engine is provided, wherein the large model inference engine is used to run a target large model to process an inference request initiated by a target application. The tuning method comprises:
[0009] A plurality of observation samples are obtained, wherein each observation sample includes a set of configuration parameters, and after configuring the large model inference engine according to the set of parameters, the engine performance index is determined by replaying the historical inference request of the target application. The observation data in the BO algorithm is initialized using the plurality of observation samples, and the objective function in the BO algorithm is defined as the mapping relationship between the configuration parameters and the engine performance index, so as to execute the BO algorithm and obtain several sets of configuration parameters that optimize the engine performance index.
[0010] In some embodiments, obtaining multiple observation samples includes: obtaining the request load of the target application, including model input and output information corresponding to historical reasoning requests; sampling multiple groups of configuration parameters from the parameter space, configuring the large model reasoning engine according to each group of configuration parameters and replaying the request load, thereby determining the corresponding engine performance indicators according to each operation data of the large model reasoning engine; and constructing the multiple observation samples based on the each group of configuration parameters and their corresponding engine performance indicators.
[0011] Furthermore, in some specific embodiments, sampling multiple groups of configuration parameters from the parameter space includes: adopting a quasi-Monte Carlo algorithm based on a Sobol sequence to sample the multiple groups of parameter configurations from the parameter space.
[0012] In some embodiments, the type of engine performance indicator is provided by the target application.
[0013] In some embodiments, the target application is an offline application, and the engine performance indicator includes a throughput indicator; or, the target application is an online non-interactive application, and the engine performance indicator includes a tail delay; or, the target application is an online interactive application, and the engine performance indicator includes a first word time TTFT and a per-word time TPOT.
[0014] In some embodiments, executing the BO algorithm involves multiple rounds of iterative updates, wherein any round of iterative updates includes: training a proxy model for fitting the objective function based on current observation data. Determining new configuration parameters to be evaluated with the goal of maximizing the function value of a preset acquisition function; wherein the input of the acquisition function is the output of the proxy model. The large model inference engine is configured according to the new configuration parameters and the historical inference requests are replayed, thereby obtaining corresponding engine performance indicators based on the operating data. The observation data is updated using observation samples constructed by the new configuration parameters and the corresponding engine performance indicators.
[0015] Further, in some specific embodiments, the indicator type of the engine performance indicator is 1, and the number of the acquisition functions is multiple; wherein, with the goal of maximizing the function value of a preset acquisition function, determining the new configuration parameters to be evaluated includes: with maximizing multiple function values corresponding to the multiple acquisition functions as multiple goals, using a multi-objective optimization algorithm to obtain a Pareto frontier solution set; based on the Pareto frontier solution set, determining the new configuration function to be evaluated.
[0016] In some other specific embodiments, there are multiple indicator types of the engine performance indicator, and the acquisition function is expected hypervolume improvement EHVI.
[0017] In some further specific embodiments, before executing the BO algorithm, the method further includes: obtaining a plurality of feasibility samples, wherein each sample includes a set of configuration parameters, and after configuring the large model inference engine according to the set of configuration parameters, a feasibility label determined by replaying the historical inference request, wherein the label indicates whether an abnormality occurs in the engine operation.
[0018] Any round of iterative updating further includes: training a prediction model based on current feasibility data, the feasibility data initially including the plurality of feasibility samples; wherein the prediction model is used to limit the feasibility probability of the new configuration parameter to be greater than or equal to a probability threshold. Determining a feasibility label corresponding to the new configuration parameter based on the operating data, and updating the feasibility data using a feasibility sample constructed from the new configuration parameter and the corresponding feasibility label.
[0019] Furthermore, in some examples, any round of iterative updating also includes: when the feasibility samples determined in this round include infeasible samples, increasing the current probability threshold; or, when the feasibility samples determined in consecutive scheduled rounds including this round are all feasible samples, lowering the current probability threshold.
[0020] On the other hand, in some further specific embodiments, configuring the large model reasoning engine according to the new configuration parameters and replaying the historical reasoning requests includes: selecting k groups of configuration parameters from the new configuration parameters for parallel evaluation according to a given degree of parallelism k.
[0021] According to the second aspect, a parameter tuning device for a large model inference engine is provided, wherein the large model inference engine is used to run a target large model to process inference requests initiated by a target application. The tuning device comprises: a sample acquisition module, configured to acquire a plurality of observation samples, wherein each observation sample comprises a set of configuration parameters, and an engine performance index determined by replaying the historical inference requests of the target application after configuring the large model inference engine according to the set of parameters. An algorithm execution module, configured to initialize the observation data in the BO algorithm using the plurality of observation samples, and to define the objective function in the BO algorithm as a mapping relationship between configuration parameters and engine performance indexes, thereby executing the BO algorithm to obtain a plurality of sets of configuration parameters that optimize the engine performance indexes.
[0022] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method provided in the first aspect.
[0023] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method provided in the first aspect is implemented.
[0024] In summary, by adopting the above-mentioned method and device disclosed in the embodiments of this specification, and using the BO algorithm to adapt to the optimization goals of different types of applications, the configuration parameters with optimal or near-optimal performance can be accurately and efficiently determined, thereby reducing the computing resources and time consumed by the inference engine to process requests and reducing the average cost. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0026] Figure 1 This is a schematic diagram of the application scenario of the large model reasoning engine;
[0027] Figure 2 This is a diagram of the implementation architecture of the solution for optimizing the performance of the inference engine based on the BO algorithm disclosed in the embodiments of this specification;
[0028] Figure 3 This is one of the process step diagrams of the parameter tuning method of the large model inference engine disclosed in the embodiments of this specification;
[0029] Figure 4A The hypervolume HV is shown when the number of objective functions is 2 and the number of sampling points in the existing solution set is 3;
[0030] Figure 4B Indicates based on Figure 4A Hypervolume improved HVI after adding new sampling points x4;
[0031] Figure 5 This is a diagram of the implementation architecture of the solution for optimizing the performance of the inference engine based on the BO algorithm and the feasibility prediction model disclosed in the embodiments of this specification;
[0032] Figure 6 This is a second schematic diagram of the process steps of the parameter tuning method of the large model inference engine disclosed in the embodiment of this specification;
[0033] Figure 7 This is a functional structural diagram of the parameter tuning device of the large model inference engine disclosed in the embodiments of this specification. DETAILED DESCRIPTION
[0034] The solution provided in this specification is described below in conjunction with the accompanying drawings.
[0035] The embodiments of this specification disclose a parameter tuning scheme for a large model inference engine. It should be understood that the performance tuning of the inference engine is achieved through parameter tuning, so performance tuning and parameter tuning can be used interchangeably in this article.
[0036] To help understand, the inventive concept of the parameter tuning solution is first introduced below. Specifically, the applicant has found through observation and research that the important problems currently faced by the inference engine parameter tuning include (other problems will be introduced in turn later):
[0037] Problem 1: Complex optimization goals. Different upper-layer applications focus on different optimization goals. For example, for offline recommendation applications, focus on throughput indicators; for online classification applications, focus on tail latency; for interactive applications, focus on first-word time (Time-To-First-Token, TTFT) and per-word time (Time-Per-Output-Token, TPOT).
[0038] To solve the above problem 1, it is proposed to use Bayesian Optimization (BO) in the parameter tuning scheme to adapt to different optimization objectives. To help understand, the BO algorithm is briefly introduced first.
[0039] The BO algorithm is a strategy for global optimization problems, especially for black-box functions with high computational costs. The basic idea is to approximate the objective function (i.e., black-box function) by constructing a proxy model, and use the proxy model to iteratively select sample points, thereby using the information of these sample points and Bayesian inference to update the proxy model, and then gradually approach the global optimal solution of the objective function. The BO algorithm includes the following steps:
[0040] 1) Define the optimization problem:
[0041] Determine the objective function f(x), which is the function you want to optimize (maximize or minimize), where x is the decision variable.
[0042] Determine the domain of the decision variable x.
[0043] 2) Select the proxy model:
[0044] Select a probability model to approximate the target function f(x). The probability model is used to capture the function value distribution of the target function.
[0045] 3) Select the acquisition function:
[0046] The acquisition function is used to determine the next sampling point based on the prediction of the proxy model, or to use the output of the proxy model as input. In the determination process, exploration and exploitation are balanced. Exploration refers to sampling in areas with high uncertainty, and exploitation refers to sampling in areas with the best prediction.
[0047] Commonly used acquisition functions include: Expected Improvement (EI), Upper Confidence Bound (UCB), Probability of Improvement (PI), etc.
[0048] 4) Initialization:
[0049] Select initial data points and evaluate the objective function at the initial data points.
[0050] 5) Iterative optimization:
[0051] ① Update the proxy model: Update the proxy model using all the data points collected so far.
[0052] ②Maximize the acquisition function: Find the point x that maximizes the acquisition function next , that is, determine the next point to be evaluated.
[0053] ③Evaluate the objective function:next Evaluate the objective function f(x next ).
[0054] ④ Update the data set: Update the new data point (x next ,f(x next )) is added to the dataset.
[0055] 6) Termination conditions:
[0056] Repeat step 5) until a termination condition is met, such as reaching a preset number of iterations, a time limit, or the value of the acquisition function no longer increases significantly.
[0057] 7) Output the optimal solution:
[0058] Returns the optimal value of the objective function f(x) and the corresponding decision variable x at all evaluation points.
[0059] The above is a brief introduction to the BO algorithm. In the parameter tuning solution disclosed in the embodiments of this specification, the BO algorithm is introduced to adapt to different optimization objectives.
[0060] Figure 2 This is a diagram of the implementation architecture of the solution for performance tuning of the inference engine based on the BO algorithm disclosed in the embodiment of this specification, which illustrates the following steps a to g:
[0061] Step a, obtaining the inference request load (or simply request load) of the service party application side and the defined performance optimization target, and sampling multiple groups of configuration parameters in the parameter space.
[0062] Step b, starting the large model inference engine and replaying the request load according to the sampled configuration parameters, and collecting performance indicators corresponding to the performance optimization goals from the running results of the inference engine, so as to initialize the observation data using the configuration parameter-performance indicator pair.
[0063] In step c, a proxy model is constructed for each performance optimization objective.
[0064] In step d, based on the predictions of the surrogate model, the parameter configuration is evaluated using the acquisition function.
[0065] Step e: running the solver based on the acquisition function to obtain multiple groups of configuration parameters to be observed.
[0066] Step f, start the inference engine according to the configuration parameters to be observed and replay the request load.
[0067] Step g, collecting performance indicators based on the running results of the inference engine, and adding the newly added configuration parameter-performance indicator pairs to the observed data.
[0068] Steps c to g are executed in a loop until a termination condition is met, for example, the number of samples in the observed data reaches a given threshold.
[0069] Next, the specific implementation steps of the above tuning solution are introduced in combination with more drawings and embodiments.
[0070] Figure 3 This is one of the process step diagrams of the parameter tuning method of the large model inference engine disclosed in the embodiment of this specification. The large model inference engine is used to run the target large model to process the inference request initiated by the target application. The target large model refers to any large model running on the large model inference engine. The target application refers to any service application that initiates an inference request to the large model inference engine. The service application may include an application client and a server.
[0071] It should be understood that the execution subject of the tuning method can be any device, platform, server or equipment cluster with computing and processing capabilities. Figure 3 As shown, the tuning method includes the following steps:
[0072] Step S310, obtaining multiple observation samples, wherein each observation sample includes a set of configuration parameters, and an engine performance indicator determined by replaying historical reasoning requests of a target application after configuring a large model reasoning engine according to the set of configuration parameters.
[0073] To help understanding, the configuration parameters and engine performance indicators are introduced first, and then the implementation method of this step is introduced.
[0074] The parameter configuration of the inference engine will affect the utilization of computing resources and the efficiency of data processing, thereby affecting the inference performance. The following Table 1 shows the parameters that may need to be configured.
[0075] Table 1
[0076]
[0077]
[0078] Table 1 provides an exemplary introduction to the types of configuration parameters in the inference engine.
[0079] Regarding the engine performance index (or simply performance index), it reflects the performance of the inference engine when processing inference requests. Generally, different applications focus on different optimization goals. In one implementation, the type of performance index is provided by the target application, or customized. In another implementation, the mapping relationship between the application type and the performance index is pre-configured in the inference engine. This mapping relationship can be established based on expert experience, so that the corresponding performance indicator type can be determined according to the type of target application. The following Table 2 shows the performance indicators that different types of applications are concerned about.
[0080] Table 2
[0081]
[0082] The above introduces the configuration parameters and performance indicators of the inference engine in combination with Table 1 and Table 2 respectively.
[0083] The implementation of this step can be achieved by executing the following sub-steps S311-S314:
[0084] Step S311, obtaining the request load of the target application, including the model input and output information corresponding to the historical reasoning request. It can be understood that the number of historical reasoning requests is generally multiple.
[0085] Regarding the input and output information of the large model, in one embodiment, the original model input and model output may be included. In another embodiment, considering possible privacy protection demands, the original input and output content may not be included, but only the size information of the input and output may be included. For example, when the original input and output of the model are both text, the input and output information in the request payload may be the length of the input and output text respectively. For another example, when the original output also includes an image, the output information may also include the image size instead of the original image.
[0086] From the above, we can obtain the request load of the target application.
[0087] Before, after, or simultaneously with executing step S311, step S312 may be executed to sample multiple groups of configuration parameters from the parameter space.
[0088] Referring to the above introduction to the BO algorithm, each set of configuration parameters and parameter space sampled in this step corresponds to the decision variable x and its domain in the BO algorithm. It should be understood that the multiple spatial dimensions of the parameter space correspond to the multiple parameters to be configured, where each parameter has a corresponding range of values, for example, it may be a set of discrete values (such as Boolean values), or for example, it may be a continuous range of values. Therefore, a point sampled from the parameter space corresponds to a set of configuration parameters.
[0089] For the execution of this step, in one implementation, random sampling may be performed in the parameter space to obtain a predetermined number of groups of configuration parameters.
[0090] In another embodiment, in order to improve the availability of sampling points, it is proposed that a quasi-Monte Carlo algorithm based on the Sobol sequence can be used to uniformly sample in the parameter space, thereby obtaining multiple sets of parameter configurations. Specifically, the following steps are included:
[0091] 1) Define the optimization problem in the quasi-Monte Carlo algorithm.
[0092] The optimization problem includes at least one objective function, where the functional form of each objective function can be customized, which receives a set of configuration parameters as input and returns a scalar value of the engine performance indicator. The goal of the optimization problem is to find the parameter configuration that makes the objective function optimal (minimum or maximum) by adjusting the parameters.
[0093] 2) Generate the Sobol sequence and transform it into parameter space.
[0094] It should be understood that the Sobol sequence is a low-discrepancy sequence, and the sequence points (or sequence elements, sampling points) are evenly distributed in the unit hypercube [0,1] d In which d is the dimension, uniform sampling in high-dimensional space can be achieved. After generating the Sobol sequence, each sampling point can be linearly transformed into the parameter space.
[0095] 3) Traverse all the transformed sampling points and calculate the objective function value corresponding to each sampling point.
[0096] 4) According to the objective function value, select some sampling points with better function values as the sampled multiple groups of configuration parameters.
[0097] Thus, by adopting the above implementation, uniform sampling can be achieved in the parameter space.
[0098] As described above, by executing step S312, multiple groups of configuration parameters can be sampled from the parameter space.
[0099] Based on the request load obtained above and the multiple groups of configuration parameters sampled, step S313 can be executed to configure the large model inference engine according to each group of configuration parameters and replay the request load, thereby determining the corresponding engine performance indicators based on the running data of each time of the large model inference engine.
[0100] It should be understood that each replay of the request load is performed in the same hardware environment by calling the same large model. In addition, when calling the target large model according to the request load, if the request load does not include the original input and output content, at this time, the content matching the input information in the request load can be fabricated as the model input. For example, the text content of the corresponding length can be fabricated as the model input according to the text length indicated by the input information. For another example, the large model output can also be set to match the output information in the request load (such as the length of the output text).
[0101] After each replay request load, the engine performance index (the index value) can be collected according to the corresponding running result generated by the inference engine.
[0102] From the above, the performance indicators corresponding to each group of configuration parameters can be obtained. It should be understood that if there is a situation where the performance indicators cannot be collected due to an operation error, the corresponding configuration parameters will be discarded.
[0103] Step S314: construct multiple observation samples based on each group of configuration parameters and the corresponding engine performance indicators.
[0104] As described above, by executing sub-steps S311 - S314 in step S310 , multiple observation samples can be constructed.
[0105] Next, in step S320, the observation data in the BO algorithm is initialized using the multiple observation samples, and the objective function in the BO algorithm is defined as a mapping relationship between configuration parameters and engine performance indicators, so that the BO algorithm is executed to obtain several groups of configuration parameters that optimize the engine performance indicators.
[0106] It should be understood that the optimization target of the target application, that is, the engine performance index, can be one or more, and accordingly, the objective function defined in this step can also be one or more. In addition, when the optimization target is one, Single-Objective Bayesian Optimization (SOBO) is specifically adopted; when the optimization target is multiple, Multi-Objective Bayesian Optimization (MOBO) is specifically adopted. For the case where the optimization target is one or more, please refer to Table 2.
[0107] The execution of the BO algorithm in this step involves multiple rounds of iterative updates, wherein any round of iterative updates includes the following sub-steps S321-S324:
[0108] Step S321, training the proxy model based on the current observation data.
[0109] It should be understood that the current observation data is the observation data updated after the previous round of iterations, and its initial data is the above-mentioned multiple observation samples. The following mainly takes the Gaussian Process (GP) model as an example of the proxy model for relevant introduction. The GP model has the advantage of being parameter-free, does not need to assume the specific form of the objective function, and can flexibly adapt to different types of data and complex functional relationships. In addition, each objective function corresponds to a GP model, and the following mainly takes a single GP model as an example for explanation.
[0110] The GP model assumes that the probability distribution of the target function f(x) follows a Gaussian distribution, and thus calculates the mean function and variance function of f(x), namely μ(f(x)) and σ, through the current observation data. 2 (f(x)). These two functions can predict any set of configuration parameters x and predict the mean and variance at that point. The larger the predicted mean, the higher the expected value of the objective function value at that point and the better the performance; the larger the predicted variance, the higher the uncertainty of the objective function value at that point.
[0111] Step S322, determining new configuration parameters to be evaluated with the goal of maximizing the function value of a preset acquisition function; wherein the input of the acquisition function is the output of the proxy model.
[0112] It can be understood that the output of the proxy model is used as the input of the acquisition function, and the two constitute a composite function. The input of the composite function is the input of the proxy model, that is, the configuration parameters, and the output is the function value of the acquisition function. Therefore, one or more sets of configuration parameters that maximize the value of the acquisition function can be solved and classified as new configuration parameters to be evaluated.
[0113] Continuing with the example of implementing the proxy model as a GP model, in this step, the acquisition function is based on the mean μ(f(x)) and variance σ predicted by the GP model 2 (f(x)) computes a score for a configuration point x, and points with high scores are suggested as the next points to evaluate the objective function. Different acquisition functions balance exploration (visiting areas of high uncertainty) and exploitation (searching areas known to perform well) in different ways.
[0114] In the SOBO algorithm, the acquisition functions used may include:
[0115] 1) Upper Confidence Bound (UCB), UCB tends to select a point with high exploration and high utilization. Its calculation formula is:
[0116] UCB(f(x))=μ(f(x))+βσ(f(x)) (1)
[0117] in, Represents standard deviation.
[0118] 2) Expected Improvement (EI). EI not only considers the probability of improvement, but also the size of the expected improvement, and gives priority to the point that maximizes the expected gain. Its calculation formula is:
[0119]
[0120] Among them, x+ is the point with the largest known objective function value so far, and ξ is a parameter that encourages exploration.
[0121] 3) Probability of Improvement (PI), PI selects the point that is most likely to exceed the currently known best observation value, and its calculation formula is:
[0122] PI(x)=P(f(x)+ξ>f(x + )) (3)
[0123] Among them, ξ is a parameter that encourages exploration.
[0124] Generally, any of the above acquisition functions can be selected, and an optimization algorithm such as evolutionary algorithm or random search can be used as a solver to maximize the acquisition function value to solve the new configuration parameter value to be evaluated. However, considering that in practical applications, if only a single acquisition function is selected, its performance in the tuning task may be poor, and it is difficult to determine the optimal acquisition function in advance.
[0125] Therefore, the applicant proposes to select two or more acquisition functions at the same time, for example, the three acquisition functions of UCB, EI and PI at the same time, and use a multi-objective optimization algorithm as a solver in this step to solve the Pareto Frontier (Pareto Frontier) of multiple acquisition functions, or the Pareto optimal frontier. The Pareto frontier is a solution set used to represent that cannot be further improved under given conditions in a multi-objective optimization problem. Specifically, the improvement of these solution sets on one objective will lead to the degradation of another objective.
[0126] For example, assuming that UCB, EI and PI are selected as acquisition functions at the same time, and the solved Pareto front includes one set of configuration parameters x1 and another set of configuration parameters x2, then the possible situation is that x1 is better than x2 in UCB and EI, and worse than x2 in PI. The impossible situation is that x1 is worse than x2 in UCB, EI and PI.
[0127] The above introduces the selection and solution of the acquisition function in the SOBO algorithm.
[0128] In the MOBO algorithm, the expected hypervolume improvement (EHVI) can be used as a collection function to identify and balance promising parameter configurations for conflicting objectives. EHVI evaluates the quality of a new sampling point based on the potential increase it may bring to the hypervolume of the existing Pareto solution set. The expression of EHVI is:
[0129]
[0130] Where HV(·) represents the hypervolume function and x represents the set of existing Pareto solutions.
[0131] The hypervolume function HV(·) is calculated by measuring the volume of the space between the sampling point and the reference point r, which is usually set as the lower bound of all objective functions. Figure 4A , which shows that when there are only two optimization objectives f1 and f2, HV(·) is calculated by calculating the sum of the areas of the rectangles formed between each point x1, x2 and x3 in the existing Pareto front solution set and the selected reference point r. Figure 4B Medium Figure 4A The newly added grey area indicates the hypervolume improvement after adding x4 new sampling points to the existing solution set.
[0132] See also Figure 4A and Figure 4B It can be seen that a larger HV(·) value indicates a better Pareto front because it covers a larger target space and provides a wider range of feasible solutions. Therefore, by maximizing EHVI, the search process can be guided towards configurations that can improve the overall quality and diversity of the Pareto front, thereby optimizing multiple objectives simultaneously.
[0133] The above introduces the selection and solution of the acquisition function in the MOBO algorithm.
[0134] From the above, by maximizing the acquisition function value, one or more new configuration parameters to be evaluated, or several new sampling points to be evaluated, can be solved.
[0135] Step S323, configure the inference engine according to the new configuration parameters and replay the historical inference requests, so as to obtain the corresponding engine performance indicators according to the operation data.
[0136] It should be noted that the introduction to this step can also refer to the above introduction to step S313, which will not be repeated here.
[0137] Step S324, update the observation data using the observation samples constructed using the new configuration parameters and the corresponding engine performance indicators.
[0138] Specifically, each observation sample constructed by each set of new configuration parameters and corresponding engine performance indicators may be added to the current observation data, thereby realizing this round of update of the observation data.
[0139] As described above, by executing sub-steps S321-S324, any round of iterative update in the execution process of the BO algorithm can be completed. The total number of rounds of iterative update can be pre-set or depend on other termination conditions, such as the number of samples in the observed data reaching a given threshold.
[0140] Furthermore, after the last round of iterative updates, several groups of configuration parameters that make the engine performance index reach the optimal or close to the optimal can be obtained by executing the BO algorithm output, where "several" refers to one or more. In addition, the performance index values corresponding to each group of configuration parameters can also be output. It can be understood that these data can be provided to the staff to assist in the decision-making of the parameter configuration of the inference engine in subsequent practical applications.
[0141] In summary, by adopting the parameter tuning method of the large model inference engine provided in the embodiment of this specification, using the BO algorithm, for example, using the SOBO algorithm and the MOBO algorithm to work together to adapt to the optimization goals of different types of applications, the configuration parameters with optimal or near-optimal performance can be accurately and efficiently determined, thereby reducing the computing resources and time consumption of the inference engine to process requests and reducing the average cost.
[0142] The above mainly introduces the introduction of BO algorithm in the parameter tuning scheme to adapt to different optimization goals, thereby solving the above-mentioned problem 1 faced by the current inference engine parameter tuning, namely: the optimization goals are complex, and different upper-layer applications focus on different optimization goals. In addition, the applicant found that the important problems faced by the inference engine parameter tuning also include:
[0143] Problem 2: Complex parameter constraints, there are some constraints between the parameters of the inference engine. Some of these constraints are known, they are hard-coded in the code of the inference engine, and can be found in the official documentation of the inference engine. For example, enable prefix caching (enable_prefix_caching) and enable chunked prefill (enable_chunked_prefill) cannot be set to true at the same time, because these two parameters are incompatible.
[0144] However, some constraints are hidden constraints. Specifically, as large model inference engines continuously integrate new technologies and are frequently updated, their stability is often affected. For specific request loads and GPU types, certain parameter combinations may cause the inference engine to pop up errors, crash the program, and make it impossible to collect performance metrics. In this paper, these infeasible parameter combinations are called hidden constraints. Frequent violations of hidden constraints will lead to a large number of invalid observations, which not only impairs the tuning efficiency, but also prevents the surrogate model from fully learning the relationship between the objective function and the configuration points, thereby affecting the quality of performance tuning. In practical applications, hidden constraints are related to hardware and request loads and cannot be known in advance.
[0145] To address problem 2, we propose to build a machine learning model to predict the feasibility probability of parameter configurations in the parameter tuning scenario and model hidden constraints by restricting the feasibility probability to be greater than or equal to a given threshold. By restricting the solver to obey the hidden constraints during configuration recommendations, the number of invalid observations can be significantly reduced without affecting alternative theoretical models.
[0146] Figure 5 This is a diagram of the implementation architecture of the solution for performance tuning of the inference engine based on the BO algorithm and the feasibility prediction model disclosed in the embodiment of this specification, which illustrates the following 9 steps (the execution order is not unique, as long as it conforms to the data flow logic):
[0147] Step 1: Obtain the request load and defined performance optimization goals of the service provider application, and sample multiple groups of configuration parameters in the parameter space.
[0148] In the second step, the large model inference engine is started according to the sampled configuration parameters and the request load is replayed. The performance indicators corresponding to the performance optimization goals and the feasibility labels are collected from the running results of the inference engine, so as to initialize the observation data using the configuration parameter-performance indicator pairs and initialize the feasibility data using the configuration parameter-feasibility label pairs.
[0149] Step 3: Use the feasibility data to establish a feasibility prediction model and learn hidden constraints. Exemplarily, the feasibility prediction model can be implemented as a random forest regression model.
[0150] In step 4, a proxy model is constructed for each performance optimization objective.
[0151] In step 5, based on the predictions of the surrogate model, the acquisition function is used to evaluate the parameter configuration.
[0152] In steps 6 and 7, the solver is run based on the acquisition function and the feasibility prediction model to obtain the configuration parameters to be observed.
[0153] In step 8, the inference engine is started according to the configuration parameters to be observed and the request load is replayed.
[0154] In step 9, performance indicators and feasibility labels are collected based on the running results of the inference engine, and the newly added configuration parameter-performance indicator pairs are added to the observation data, and the newly added configuration parameter-feasibility label pairs are added to the feasibility data.
[0155] Steps 3 to 9 are run in a loop until the termination condition is met, for example, the number of samples in the observed data reaches a given threshold.
[0156] Next, the specific implementation steps of the above tuning solution are introduced in combination with more drawings and embodiments.
[0157] Figure 6 The second process step diagram of the parameter tuning method of the large model inference engine disclosed in the embodiment of this specification. The large model inference engine is used to run the target large model to process the inference request initiated by the target application. It should be noted that the introduction of the target large model and the target application can be referred to the relevant description in the aforementioned embodiment, and no further elaboration is given.
[0158] It should be understood that the execution subject of the tuning method can be any device, platform, server or equipment cluster with computing and processing capabilities. Figure 6 As shown, the tuning method includes the following steps:
[0159] Step S610, obtain multiple observation samples and multiple feasibility samples, where each sample includes a set of configuration parameters, and a sample label determined by replaying the historical reasoning requests of the target application after configuring the large model inference engine according to the set of configuration parameters; wherein the observation sample label is an engine performance indicator, and the feasibility sample label indicates whether an abnormality occurs in the engine operation.
[0160] In one implementation, the implementation of this step may include the following sub-steps S611-S614:
[0161] Step S611, obtaining the request load of the target application, including the model input and output information corresponding to the historical reasoning request. It should be noted that the description of this step can refer to the above introduction to step S311, and will not be repeated here.
[0162] Step S612, sampling multiple groups of configuration parameters from the parameter space. It should be noted that the description of this step can refer to the above introduction to step S312, and will not be repeated here.
[0163] Step S613, configure the large model reasoning engine according to each group of configuration parameters and replay the request load, so as to determine the corresponding feasibility label and engine performance index according to each operation data of the large model reasoning engine.
[0164] For any set of configuration parameters, whether an abnormality occurs in the operation of the inference engine based on its corresponding operating data analysis. If the analysis shows that an abnormality occurs, the feasibility label corresponding to the set of configuration parameters is determined to indicate that an abnormality occurs, for example, the label value is set to 0; if the analysis shows that no abnormality occurs, the corresponding feasibility label is determined to indicate that no abnormality occurs, for example, the label value is set to 1.
[0165] Furthermore, when the analysis shows that no abnormality has occurred, the engine performance indicators are collected based on the running data, or the indicator values corresponding to the engine performance indicators are determined; when the analysis shows that an abnormality has occurred, the engine performance indicators are generally unable to be collected, or the collected engine performance indicators are incorrect or unusable, and therefore, the use of this set of configuration parameters to construct the observation sample can be abandoned.
[0166] From the above, the feasibility label and engine performance index corresponding to the configuration parameter can be determined. It should be noted that the description of this step can also refer to the above introduction to step S313.
[0167] Step S614, constructing observation samples and feasibility samples based on the feasibility labels and engine performance indicators corresponding to each group of configuration parameters.
[0168] As described above, by executing sub-steps S611-S614, multiple observation samples and multiple feasibility samples can be constructed.
[0169] In another embodiment, the acquisition of observation samples and feasibility samples may also be performed independently of each other.
[0170] From the above, multiple observation samples and multiple feasibility samples can be obtained.
[0171] Next, in step S620, based on multiple observation samples and multiple feasibility samples, multiple rounds of iterative updates are performed according to the BO algorithm and the feasibility prediction model to obtain several groups of configuration parameters that optimize the engine performance indicators and whose feasibility is greater than a predetermined threshold.
[0172] Any round of iterative update may include the following sub-steps S621-S625:
[0173] Step S621, training a proxy model for fitting an objective function based on current observation data; the observation data initially includes the plurality of observation samples, and the objective function is defined as a mapping relationship between configuration parameters and engine performance indicators.
[0174] It should be noted that the introduction to this step can refer to the aforementioned introduction to step S321 and will not be repeated here.
[0175] Step S622, training a feasibility prediction model (or simply a prediction model) based on current feasibility data, where the feasibility data initially includes the multiple feasibility samples.
[0176] It should be noted that the feasibility prediction model can be implemented as a random forest regression model or a linear regression model, etc. In addition, the training of the feasibility prediction model can be implemented by using an existing supervised training method, which will not be described in detail.
[0177] Step S623, with the goal of maximizing the function value of the preset acquisition function and making the feasibility probability predicted by the prediction model greater than the probability threshold, determine the new configuration parameters to be evaluated.
[0178] In one embodiment, multiple groups of candidate configuration parameters can be determined with the goal of maximizing the acquisition function value, and then the prediction model can be used to process the multiple groups of candidate configuration parameters separately to obtain corresponding multiple feasibility probabilities, so that the candidate configuration parameters with feasibility probabilities greater than the probability threshold are classified as new configuration parameters to be evaluated.
[0179] It should be noted that for the determination of the multiple groups of candidate configuration parameters, reference may be made to the aforementioned description of step S322.
[0180] In another embodiment, for the candidate sampling points to be scored by the acquisition function, the feasibility may be predicted using the prediction model first, and then the acquisition function may be used to score them when the predicted probability is greater than the probability threshold.
[0181] In addition, given that the known and hidden constraints prune the solution space, the solver may not be able to come up with a sufficient number of solutions. In this case, random sampling of collocation points can be used to compensate for the lack of sampling points.
[0182] From the above, we can get the new configuration parameters to be evaluated.
[0183] Step S624, configure the large model reasoning engine according to the new configuration parameters and replay the historical reasoning requests, so as to construct new observation samples and feasibility samples according to the operating data.
[0184] It should be noted that the introduction to this step can refer to the introduction to steps S613 and S614 mentioned above.
[0185] Step S625, adding the constructed new observation sample to the current observation data, and adding the constructed new feasibility sample to the current feasibility data, thereby realizing the current round of updating of the observation data and feasibility data.
[0186] As described above, by executing sub-steps S621-S625, any round of iterative updates can be achieved, wherein the agent model, the prediction model, the observation data and the feasibility data are updated.
[0187] Furthermore, the applicant proposes that the probability threshold used in sub-step S623 can also be dynamically adjusted. Specifically, any round of iterative update can also include step S626 (executed after step S624, and Figure 6(not shown): When the feasibility samples newly constructed in this round include infeasible samples, the probability threshold is increased, for example, the current probability threshold is updated to a value after adding a predetermined positive number to reduce the possibility of recommending infeasible parameter configurations in the future; when the feasibility samples constructed in consecutive predetermined rounds including this round (for example, 5 consecutive rounds) are all feasible samples, the probability threshold is lowered, for example, the current probability threshold is updated to a value after subtracting a predetermined positive number to allow the discovery of potential excellent configurations. For another example, when the feasible samples constructed in several consecutive rounds including this round are all feasible samples, but the number of rounds does not reach the predetermined round, the probability threshold remains unchanged.
[0188] In summary, the parameter tuning method of the large model inference engine provided in the embodiment of this specification is adopted. On the basis of using the BO algorithm, a feasibility prediction model is further constructed to learn hidden constraints. During the tuning process, the feasibility prediction model will be continuously optimized using the latest samples to ensure that the hidden constraints are intelligently followed by improving the prediction accuracy, thereby improving the quality of performance tuning, and can more accurately and efficiently determine the configuration parameters with optimal or near-optimal performance, thereby reducing the computing resources and cost consumption of the inference engine processing requests.
[0189] Figure 6 Demonstrate the parameter tuning method flow that can solve Problem 1 and Problem 2. In addition, the applicant found that the important problems faced by the inference engine parameter tuning include:
[0190] Problem 3: Huge tuning overhead. Each time you adjust the parameters, you need to replay the request load to know the performance results. Even if you adjust only a few or dozens of times, it usually takes several or dozens of hours to tune. As large models are gradually applied to various fields and there are more and more large model apps, tuning them will result in a lot of time overhead.
[0191] In order to solve problem 3, the applicant proposes that a method of parallel evaluation of multiple groups of configuration parameters can be further introduced into the parameter tuning scheme to accelerate the tuning process. It should be understood that parallel evaluation can be based on Figure 3 The method shown in the figure is introduced, which can solve problem 1 + problem 3, or it can be based on Figure 6 By introducing the method shown in, problem 1+problem 2+problem 3 can be solved.
[0192] Specifically, the above step S323 or step S624 of configuring the inference engine according to the new configuration parameters and replaying the historical inference request can be implemented as follows: according to the given parallelism k (≥2), k groups of configuration parameters are selected from the new configuration parameters for parallel evaluation. It should be noted that although parallel evaluation requires additional computing resources, many computing resources in the production cluster are often idle during non-peak hours such as nighttime and weekends, and can be conveniently used to adjust the inference engine.
[0193] Further, in one embodiment, when the SOBO algorithm is used, k points can be randomly selected from the Pareto frontier solved by the aforementioned multi-objective solver for evaluation. In another embodiment, when the MOBO algorithm is used, the first k points with the highest EHVI solved by the solver can be selected for evaluation. In this way, since the predefined total number of observation points is fixed, parallel tuning can be used to speed up k times.
[0194] From the above, by introducing parallel evaluation, the process of inference engine tuning can be accelerated, thus solving problem 3.
[0195] In summary, the inference engine tuning method disclosed in the embodiments of this specification can be used to adapt to a variety of different optimization goals by using the SOBO algorithm and the MOBO algorithm to work together. Furthermore, known constraints can be hard-coded as solver constraints, and hidden constraints can be learned using a feasibility prediction model to intelligently protect hidden constraints during the tuning process; or, parallel evaluation can be used to use additional resources to accelerate the tuning process.
[0196] Corresponding to the above-mentioned optimization scheme, the embodiments of this specification also disclose an optimization device. Figure 7 This is a functional structural diagram of a parameter tuning device for a large model inference engine disclosed in an embodiment of this specification, wherein the large model inference engine is used to run a target large model to process an inference request initiated by a target application. Figure 7 As shown, the device 700 includes:
[0197] The sample acquisition module 710 is configured to acquire multiple observation samples, wherein each observation sample includes a set of configuration parameters, and after configuring the large model inference engine according to the set of parameters, the engine performance index determined by replaying the historical inference request of the target application. The algorithm execution module 720 is configured to initialize the observation data in the BO algorithm using the multiple observation samples, and define the objective function in the BO algorithm as a mapping relationship between the configuration parameters and the engine performance index, thereby executing the BO algorithm to obtain several sets of configuration parameters that optimize the engine performance index.
[0198] In some embodiments, the sample acquisition module 710 specifically includes: a request load acquisition unit 711, configured to obtain the request load of the target application, including model input and output information corresponding to historical reasoning requests; a parameter sampling unit 712, configured to sample multiple groups of configuration parameters from the parameter space, configure the large model reasoning engine according to each group of configuration parameters and replay the request load, so as to determine the corresponding engine performance indicators according to each operation data of the large model reasoning engine; an observation sample construction unit 713, configured to construct the multiple observation samples based on the each group of configuration parameters and their corresponding engine performance indicators.
[0199] Furthermore, in some specific embodiments, the parameter sampling unit 712 is specifically configured to: adopt a quasi-Monte Carlo algorithm based on a Sobol sequence to sample the multiple groups of parameter configurations from the parameter space.
[0200] In some embodiments, the type of engine performance indicator is provided by the target application.
[0201] In some embodiments, the target application is an offline application, and the engine performance indicator includes a throughput indicator; or, the target application is an online non-interactive application, and the engine performance indicator includes a tail delay; or, the target application is an online interactive application, and the engine performance indicator includes a first word time TTFT and a per-word time TPOT.
[0202] In some embodiments, the algorithm execution module 720 executes any round of iterative updates in multiple rounds of iterative updates through the following units included therein:
[0203] The proxy model training unit 721 is configured to train a proxy model for fitting the objective function based on the current observation data. The new parameter determination unit 722 is configured to determine the new configuration parameters to be evaluated with the goal of maximizing the function value of the preset acquisition function; wherein the input of the acquisition function is the output of the proxy model. The indicator collection unit 723 is configured to configure the large model inference engine according to the new configuration parameters and replay the historical inference requests, so as to obtain the corresponding engine performance indicators according to the operating data; the observation data update unit 724 is configured to update the observation data using the observation samples constructed by the new configuration parameters and the corresponding engine performance indicators.
[0204] Further, in some specific embodiments, the indicator type of the engine performance indicator is 1, and the number of the acquisition functions is multiple; the new parameter determination unit 722 is specifically configured as: taking multiple function values corresponding to the multiple acquisition functions as multiple objectives, and adopting a multi-objective optimization algorithm to obtain a Pareto frontier solution set; based on the Pareto frontier solution set, determining the new configuration function to be evaluated.
[0205] In some other specific embodiments, there are multiple indicator types of the engine performance indicator, and the acquisition function is expected hypervolume improvement EHVI.
[0206] In some further specific embodiments, the sample acquisition module 710 is further configured to: acquire multiple feasibility samples, wherein each sample includes a set of configuration parameters, and after configuring the large model inference engine according to the set of configuration parameters, a feasibility label determined by replaying the historical inference request, the label indicating whether an abnormality occurs in the engine operation. The algorithm execution module 720 also includes: a prediction model training unit 725, configured to train a prediction model based on current feasibility data, wherein the feasibility data initially includes the multiple feasibility samples; wherein the prediction model is used to limit the feasibility probability of the new configuration parameter to be greater than or equal to a probability threshold; a feasibility data updating unit 726, configured to determine the feasibility label corresponding to the new configuration parameter according to the operating data, and update the feasibility data using the feasibility sample constructed by the new configuration parameter and the corresponding feasibility label.
[0207] Furthermore, in some examples, the algorithm execution module 720 also includes: a probability threshold updating unit 727, configured to increase the current probability threshold when the feasibility samples determined in this round include infeasible samples; or, when the feasibility samples determined in consecutive predetermined rounds including this round are all feasible samples, lower the current probability threshold.
[0208] In some further specific embodiments, the indicator collection unit 723 is specifically configured to: select k groups of configuration parameters from the new configuration parameters for parallel evaluation according to a given parallelism k.
[0209] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute Figure 3 or Figure 6 The method described.
[0210] According to another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, Figure 3 or Figure 6 The method described.
[0211] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.
[0212] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.
Claims
1. A parameter tuning method for a large model inference engine, wherein the large model inference engine is used to run a target large model to process an inference request initiated by a target application; the method comprises: Acquire multiple observation samples, wherein each observation sample includes a set of configuration parameters, and an engine performance indicator determined by replaying historical reasoning requests of the target application after configuring the large model reasoning engine according to the set of parameters; The observation data in the Bayesian optimization BO algorithm is initialized using the multiple observation samples, and the objective function in the BO algorithm is defined as a mapping relationship between configuration parameters and engine performance indicators, so as to execute the BO algorithm and obtain several groups of configuration parameters that optimize the engine performance indicators.
2. The method according to claim 1, wherein: Get multiple observation samples, including: Obtaining the request payload of the target application, including model input and output information corresponding to the historical reasoning request; Sampling multiple groups of configuration parameters from the parameter space, configuring the large model inference engine according to each group of configuration parameters and replaying the request load, thereby determining corresponding engine performance indicators according to each operation data of the large model inference engine; The plurality of observation samples are constructed based on the groups of configuration parameters and their corresponding engine performance indicators.
3. The method according to claim 2, wherein: Sample multiple sets of configuration parameters from the parameter space, including: The plurality of groups of parameter configurations are sampled from the parameter space using a quasi-Monte Carlo algorithm based on a Sobol sequence. The method according to claim 1 , wherein the type of engine performance indicator is provided by the target application.
5. The method according to claim 1 or 4, wherein: The target application is an offline application, and the engine performance indicator includes a throughput indicator; Alternatively, the target application is an online non-interactive application, and the engine performance indicator includes tail latency; Alternatively, the target application is an online interactive application, and the engine performance indicators include first word time TTFT and per word time TPOT.
6. The method according to claim 1, wherein: Executing the BO algorithm involves multiple rounds of iterative updates, wherein any round of iterative updates includes: Training a proxy model for fitting the objective function based on current observation data; Determining new configuration parameters to be evaluated with the goal of maximizing the function value of a preset acquisition function; wherein the input of the acquisition function is the output of the proxy model; Configuring the large model reasoning engine according to the new configuration parameters and replaying the historical reasoning requests, thereby obtaining corresponding engine performance indicators according to the operating data; The observation data is updated using an observation sample constructed by the new configuration parameters and corresponding engine performance indicators.
7. The method according to claim 6, wherein the indicator type of the engine performance indicator is one, and the number of the acquisition functions is multiple; wherein, With the goal of maximizing the function value of the preset acquisition function, determine the new configuration parameters to be evaluated, including: Taking maximizing multiple function values corresponding to the multiple acquisition functions as multiple objectives, a multi-objective optimization algorithm is used to obtain a Pareto frontier solution set; Based on the Pareto frontier solution set, the new configuration function to be evaluated is determined.
8. The method according to claim 6, wherein there are multiple indicator types of engine performance indicators, and the acquisition function is expected hypervolume improvement (EHVI).
9. The method according to claim 6, wherein: Before executing the BO algorithm, the method further includes: Acquire multiple feasibility samples, wherein each sample includes a set of configuration parameters, and a feasibility label determined by replaying the historical reasoning request after configuring the large model reasoning engine according to the set of configuration parameters, and the label indicates whether an abnormality occurs in the engine operation; Any round of iterative updating also includes: Training a prediction model based on current feasibility data, wherein the feasibility data initially includes the plurality of feasibility samples; wherein the prediction model is used to limit the feasibility probability of the new configuration parameter to be greater than or equal to a probability threshold; The feasibility tag corresponding to the new configuration parameter is determined according to the operation data, and the feasibility data is updated using a feasibility sample constructed by the new configuration parameter and the corresponding feasibility tag.
10. The method according to claim 9, wherein: Any round of iterative updates also includes: If the feasibility samples determined in this round contain infeasible samples, increase the current probability threshold; or, When all feasibility samples determined in consecutive scheduled rounds including the current round are feasible samples, the current probability threshold is lowered.
11. The method according to claim 6, wherein: Configuring the large model reasoning engine according to the new configuration parameters and replaying the historical reasoning request includes: According to a given parallelism k, k groups of configuration parameters are selected from the new configuration parameters for parallel evaluation.
12. A parameter tuning device for a large model inference engine, wherein the large model inference engine is used to run a target large model to process an inference request initiated by a target application; the device comprises: a sample acquisition module configured to acquire a plurality of observation samples, wherein each observation sample includes a set of configuration parameters, and an engine performance indicator determined by replaying historical reasoning requests of the target application after configuring the large model reasoning engine according to the set of parameters; The algorithm execution module is configured to use the multiple observation samples to initialize the observation data in the Bayesian optimization BO algorithm, and define the objective function in the BO algorithm as a mapping relationship between configuration parameters and engine performance indicators, so as to execute the BO algorithm and obtain several groups of configuration parameters that optimize the engine performance indicators.
13. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed in a computer, the computer is caused to execute the method according to any one of claims 1 to 11.
14. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 11 is implemented.
Citation Information
Cited By
Method and device for adjusting engine parameters of inference engine
CN119378693A
Method, device and equipment for acquiring reasoning performance data of large model reasoning cluster
CN119718878A
Parameter optimization system, method and device of reasoning service engine, storage medium and program product
CN122470833A
