Bayesian optimization-based big language model reasoning service performance optimization method
By constructing a parameter space and objective function, and combining the 'sampling-modeling-exploration-iteration' process of the Bayesian optimization algorithm, the problems of complex parameter dependencies and high testing costs in large language model inference services are solved, achieving efficient and accurate performance optimization and improving throughput and latency performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-31
- Publication Date
- 2026-04-14
AI Technical Summary
Existing Bayesian optimization techniques have adaptability defects in large language model inference service scenarios. They are difficult to capture complex dependencies between parameters, and the testing cost is high, resulting in slow convergence speed and easy getting trapped in local optima.
We construct a parameter space, design an objective function, use a Bayesian optimization algorithm for intelligent search, and combine the performance test results of the large language model inference service to achieve efficient optimization through an optimization process of 'sampling-modeling-exploration-iteration'.
Find the optimal parameter configuration within a limited number of experiments, significantly improve performance and resource utilization, optimize efficiency, adaptability, and improve throughput by 20%~50% and reduce latency by 10%~30%.
Smart Images
Figure CN121859952A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence and high-performance computing technology, and in particular to a method, system, computer-readable storage medium, and electronic device for performance optimization of large language model inference services based on Bayesian optimization. Background Technology
[0002] In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities in fields such as natural language processing and intelligent question answering. Optimizing the performance of their inference services has become a key requirement for supporting high-concurrency, low-latency scenarios. Currently, deployment schemes based on efficient inference frameworks such as vLLM have become the mainstream choice due to their significant improvement in computational efficiency. However, the performance metrics of inference services (such as throughput and end-to-end latency) are affected by multiple coupled factors, including parameter configurations during service startup (such as tensor parallelism, batch size, and memory allocation strategies) as well as the dynamic characteristics of client requests (such as the number of concurrent requests and the length of the input / output token sequence).
[0003] Traditional performance optimization methods mainly include two categories: grid search and random search. Grid search determines the optimal solution by exhaustively searching all possible combinations within a preset parameter space, but its computational complexity increases exponentially with the parameter dimension, making it extremely inefficient in high-dimensional scenarios. Although random search reduces computation through random sampling, it lacks effective use of historical experimental data, is highly unpredictable, and cannot guarantee convergence to the global optimum.
[0004] Bayesian optimization, as an intelligent optimization paradigm based on probabilistic surrogate models, fits the objective function (i.e., performance index) by iteratively updating a Gaussian process model with equal probability, and balances exploration and utilization using a sampling function. It can efficiently approximate the optimal solution within a limited number of experiments, significantly reducing evaluation costs. However, existing Bayesian optimization techniques still have adaptability limitations in LLM inference service scenarios: on the one hand, the parameter space of inference services is wide-ranging (e.g., the number of parallel processes can reach tens, and the token length spans greatly), and parameters have strong correlations (e.g., the synergistic effect between batch size and concurrency), making it difficult for general Bayesian optimization models to capture such complex dependencies; on the other hand, inference performance testing requires significant GPU resources and each stress test is time-consuming (usually on the order of minutes or even hours), placing extremely high demands on the sample efficiency of the optimization algorithm. Existing solutions are not customized for these characteristics, resulting in slow convergence speed and susceptibility to local optima when directly applied.
[0005] Therefore, there is an urgent need for a Bayesian optimization method that deeply integrates the characteristics of LLM inference services (such as parameter correlation and high testing cost) to achieve fast and accurate positioning of high-performance parameter configuration. Summary of the Invention
[0006] To address the aforementioned issues, this invention proposes a novel performance optimization method and system for large language model inference services based on Bayesian optimization. This method achieves efficient optimization of inference performance by constructing a parameter space, designing an objective function, employing a Bayesian optimization algorithm for intelligent search, and combining this with performance stress testing results from large language model inference services (such as VLLM). Specifically, it includes: determining the key parameters affecting inference performance and their value ranges; constructing an objective function that integrates throughput and latency; executing an optimization process of "sampling-modeling-exploration-iteration" based on a Bayesian optimization framework; and ensuring the reliability of the optimal parameters through intelligent parameter tuning and result verification. In summary, through five key designs—parameter co-design, multi-index objective function, Bayesian optimization process for parameter tuning, engineering verification, and adaptation to various models and inference frameworks—this invention solves the problems of low efficiency, difficulty in balancing, strong blindness, and poor scalability associated with traditional grid search or random search. It can find the optimal parameter configuration within a limited number of experiments, saving computational costs during the tuning process while significantly improving the performance and resource utilization of large language model inference services.
[0007] Specifically, this application provides the following technical solutions: The first aspect of this application provides a performance optimization method for large language model inference services based on Bayesian optimization, the method comprising: S1. Parameter Space Construction: Determine the key parameters that affect the performance of the large language model inference service and their value ranges. The key parameters include inference service startup parameters and client request parameters. S2. Design the objective function: Construct an objective function for comprehensively evaluating inference performance. The objective function is based on a weighted fusion of throughput and latency metrics. S3, Bayesian optimization iteration: Based on the Bayesian optimization framework, sequential search is performed, and the optimal parameter combination is gradually approached through initial sampling, construction of a probability model, selection of a collection function and iterative update; S4. Optimal Parameter Verification: Perform multi-dimensional verification on the optimal parameter combination obtained from the Bayesian optimization iteration steps to ensure the reliability of parameter configuration and business adaptability.
[0008] Furthermore, in the method of this application, the inference service startup parameters in step S1 include: tensor parallelism, GPU memory utilization, maximum batch processing token count, and maximum concurrent sequence count; The client request parameters include: number of concurrent requests, number of input tokens, and number of output tokens.
[0009] Furthermore, in the method of this application, the objective function in step S2 is f(x), which represents the optimal performance value and is obtained by calculating the comprehensive performance score. The calculation formula is as follows: ; In the formula, Throughput(x) For average request throughput, Latency(x) To calculate the average token latency, a weighted sum is calculated by combining the average request throughput with the inverse of the average token latency using preset weights. α The weighting coefficient is 0 < α <1.
[0010] Furthermore, in the method of this application, the weight coefficient α is dynamically adjusted according to the business scenario type. If the business is a "high-concurrency API service", the weight of the average request throughput item is increased; if the business is a "real-time dialogue", the weight of the average token latency item is increased.
[0011] Furthermore, in the method of this application, step S3 specifically includes: S31. Initial random sampling: Randomly select an initial parameter combination in the parameter space, perform performance testing, and obtain initial observation data; S32. Probabilistic Model Construction: A probabilistic proxy model is trained based on the initial observation data. The probabilistic proxy model is used to predict the performance score and uncertainty of untested parameter combinations. S33. Acquisition function selection: Based on the prediction results of the probabilistic proxy model, the next parameter combination to be evaluated is selected through the acquisition function to balance exploration and utilization; S34. Iterative Update: Perform performance testing on the selected parameter combination, add the newly obtained observation data to the dataset, and update the probabilistic proxy model; S35. Stopping condition judgment: The iteration is terminated when the preset number of iterations or the performance convergence condition is met.
[0012] Furthermore, in the method of this application, the probabilistic surrogate model is a Gaussian process model or a tree-structured Parzen estimator.
[0013] Furthermore, in the method of this application, the acquisition function is any one of the following: The upper confidence bound function is used to select a parameter combination that has a high prediction score and high uncertainty. The expected improvement function is used to select a combination of parameters where the expected improvement value is greater than the current optimal value.
[0014] Furthermore, in the method of this application, step S4 includes: S41. Stability verification: Perform multiple performance tests continuously using the optimal parameter combination, and evaluate the parameter stability based on the fluctuation range of performance indicators. S42. Business adaptability verification: Perform performance tests in real business scenarios to verify the effectiveness of the optimal parameter combination in practical applications; S43. Comparison and verification: Compare the performance of the optimal parameter combination with the empirical parameter configuration and the grid search results to verify the optimization effect.
[0015] A second aspect of this application provides a performance optimization system for large language model inference services based on Bayesian optimization. The system, when running, implements the steps of the aforementioned performance optimization method for large language model inference services based on Bayesian optimization. The system includes: The parameter space construction module is used to determine the key parameters that affect the performance of the large language model inference service and their value ranges. The key parameters include inference service startup parameters and client request parameters. The objective function design module is used to construct an objective function that comprehensively evaluates inference performance. The objective function is based on a weighted fusion of throughput and latency metrics. The Bayesian optimization iteration module is used to perform sequential search based on the Bayesian optimization framework. It gradually approaches the optimal parameter combination through initial sampling, building a probability model, selecting a sampling function, and iterative updates. The optimal parameter verification module is used to perform multi-dimensional verification of the optimal parameter combination obtained by Bayesian optimization iteration to ensure the reliability of parameter configuration and business adaptability.
[0016] A third aspect of this application provides an electronic device, including: a memory and a processor; Memory: Used to store computer programs; Processor: Used to execute the computer program to implement the steps of the aforementioned Bayesian optimization-based large language model inference service performance optimization method.
[0017] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the aforementioned Bayesian optimization-based large language model inference service performance optimization method.
[0018] In summary, compared with the prior art, the present invention has the following significant advantages: (1) High optimization efficiency: Through the Bayesian optimization intelligent search mechanism, the number of experiments is reduced (only 1 / 5 to 1 / 3 of the grid search), reducing GPU resource consumption and time cost.
[0019] (2) Strong adaptability: By adjusting the parameter space, it can adapt to different scales (e.g., 7B, 32B, 671B, etc.) and different types of models (e.g., DeepSeek, Qwen, etc.).
[0020] (3) Significant performance improvement: It can find the optimal parameter configuration that balances throughput and latency. Compared with empirical parameter configuration, throughput can be increased by 20%~50% and latency can be reduced by 10%~30%.
[0021] (4) Significantly enhanced scalability: In addition to being applicable to the vLLM inference framework, performance optimization can also be achieved by adjusting the parameter space under inference frameworks such as MindSpore, PaddlePaddle, and TensorFlow.
[0022] Other features and advantages of this application will be set forth in detail in the following description, or will become apparent through the implementation of the relevant technical solutions of this application. The objectives and other advantages of this application can be achieved through the technical features and means explicitly pointed out in the description, claims, and drawings, and will be obtained through the implementation of these technical contents. Attached Figure Description
[0023] To more clearly illustrate the technical solution of this application, the accompanying drawings involved in the description of this application will be briefly introduced below. It should be noted that the drawings only show some embodiments of this application. For those skilled in the art, other related drawings can be derived from these drawings without creative effort.
[0024] Figure 1 This is a framework diagram of the implementation process of the embodiments of this application.
[0025] Figure 2 This is a flowchart illustrating the overall implementation of the Bayesian optimization-based performance optimization method for large language model inference services in this application.
[0026] Figure 3 This is a structural diagram of the performance optimization system for large language model inference services based on Bayesian optimization, as described in this application.
[0027] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.
[0029] In this document, the term "comprising" and any variations thereof (such as "including," "including," etc.) are open-ended expressions and should be understood as "including but not limited to," meaning that the listed content is not exhaustive and may include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on," meaning that the basis or condition referred to may not be the only factor and may involve other relevant factors. The term "one embodiment" should be understood as "at least one embodiment," meaning that the described embodiment is not the only possible implementation, and other similar embodiments may exist.
[0030] In this application, the terms "a" and "a plurality of" are used to modify related elements or features, and their expression is illustrative rather than restrictive. Unless otherwise expressly stated in the context, "a" should be understood as "at least one," and "a plurality of" should be understood as "at least two." Those skilled in the art should reasonably interpret these terms based on the semantic and logical relationships of the context to ensure that they cover the possibility of "one or more."
[0031] Example: A performance optimization method for large language model inference services based on Bayesian optimization Core steps: 1. Parameter space construction, including configuring inference service startup parameters and client request parameters.
[0032] 2. Objective function design.
[0033] 3. Bayesian optimization iteration.
[0034] 4. Optimal parameter verification.
[0035] Figure 1 The implementation process framework diagram of this solution includes the following processes: I. Parameter Space Construction Inference service startup parameters include tensor parallelism (based on the number of GPUs, such as 1~8), GPU memory utilization (such as 0.6~0.9, to avoid memory overflow), maximum batch processing tokens (such as 512~12800, not exceeding the model context length), and maximum concurrent sequence count (such as 1~32, based on the single GPU's capacity).
[0036] Client request parameters include the number of concurrent requests (e.g., 1~64, covering peak business demand), the number of input tokens (e.g., 32~128000), and the number of output tokens (e.g., 32~128000).
[0037] II. Objective Function Design Define an objective function f(x) to represent the optimal performance value, and calculate the overall performance score (the higher the score, the better the performance). ; In the formula, Throughput(x) For average request throughput, Latency(x) To calculate the average token latency, a weighted sum is calculated by combining the average request throughput with the inverse of the average token latency using preset weights. α Weighting coefficients (0 < α <1). If the business is a "high-concurrency API service", the throughput weight can be increased; if the business is a "real-time dialogue", the low latency weight can be increased.
[0038] III. Bayesian Optimization Iteration The core process of Bayesian optimization is a closed loop of "sampling → modeling → exploration → iteration". The essence of Bayesian optimization is to replace "exhaustive search" with "probabilistic models," predicting "which parameter combinations might be better" through a small number of experimental results, then prioritizing the testing of these combinations to gradually approach the optimal solution. For the VLLM scenario, the specific process is as follows: 1. Initial random sampling: Obtaining the "first wave of experimental data" Objective: To provide initial "prior information" for probabilistic models and avoid the model having "no data to learn" at the beginning.
[0039] Operation: Randomly select 10 to 20 sets of parameter combinations from the defined parameter space (the number is adjusted according to the parameter dimensions; the more dimensions there are, the more initial samples can be added appropriately). Each set of parameters corresponds to a complete VLLM inference performance test (start the service → send requests → record throughput / latency → calculate the objective function score).
[0040] Example: Randomly sample combinations such as "Tensor parallelism = 2, concurrency = 16, input token = 128" and "Tensor parallelism = 1, concurrency = 32, input token = 256" to obtain 15 sets of corresponding data for "parameter → performance score".
[0041] 2. Constructing a probabilistic model: Predicting the relationship between parameters and performance. Core logic: Use the initial sampled "parameter-score" data to train a probabilistic model such as a Gaussian process (GP) or a tree-based Parzen estimator (TPE).
[0042] Model output: For any set of untested parameters, it not only predicts the "possible performance score", but also the "uncertainty of the score" (i.e. "how reliable the prediction is").
[0043] For example, the model predicts that the score for "parameter A" might be 80 (low uncertainty, indicating that it is close to the known optimal value), and the score for "parameter B" might be 75 (high uncertainty, indicating that there is a lack of data in this area).
[0044] 3. Select the acquisition function: Decide "which parameter to measure next". Objective: To balance "exploration" (testing parameters with high uncertainty to avoid missing the optimal solution) and "utilization" (testing parameters with high prediction scores to quickly approach the optimal solution).
[0045] Commonly used data acquisition functions (adapted to VLLM scenarios): UCB (Upper Confidence Bound): Prioritizes parameters with "high prediction score + high uncertainty". Suitable for scenarios with a large VLLM parameter space, it can balance exploring new regions and utilizing known advantageous regions.
[0046] EI (Expected Improvement): Prioritizes parameters that have a high probability of being better than the current best score. Suitable for later optimization (when there are already better parameters, it focuses on "deepening" the optimal region).
[0047] For example, through UCB calculation, it was found that the "prediction score + uncertainty compensation" of "parameter C (tensor parallelism = 3, concurrency = 24)" was the highest. Therefore, the next test target was set as parameter C.
[0048] 4. Iterative Update: Loop through "Test → Update Model → Select Next" Operation: Perform a VLLM performance test using the parameter combination selected by the acquisition function to obtain new "parameter-score" data; add the new data to the dataset and retrain the probabilistic model; repeat the process of "selecting parameters → testing → updating the model".
[0049] 5. Stopping conditions: The number of iterations reaches a preset value (e.g., 30-50 times, to balance the optimization effect and testing cost, as a single test of vllm takes a long time, so too many iterations are unnecessary); After N consecutive iterations, the performance score no longer improves (e.g., in 5 consecutive tests, the score of the new parameters does not exceed the current optimal value, indicating that it is close to the optimal solution).
[0050] IV. Optimal Parameter Verification For the optimal parameter combination obtained by Bayesian optimization, conduct stability verification, business adaptability verification, or comparative verification.
[0051] 1. Stability verification: Execute 5 to 10 performance tests continuously with optimal parameters and observe the fluctuation range of throughput and latency (if the fluctuation is ≤10%, it indicates that the parameter configuration is stable and there is no obvious abnormality).
[0052] 2. Business adaptability verification: Conduct tests in actual business scenarios (such as using prompt input from real users, rather than simulated input from benchmark tests) to confirm whether the performance of the optimal parameters in real scenarios meets expectations (avoiding the situation of "good performance in benchmark tests but poor performance in business scenarios").
[0053] 3. Comparative verification: Compare the optimal parameters with "empirical parameters" (such as the official recommended configuration of vllm) and "optimal solution of grid search" to verify whether the parameters found by Bayesian optimization are really better (e.g., finding a better result with 30 tests than with 50 grid searches).
[0054] In summary, the above methods can achieve intelligent and efficient performance improvement in large language model inference services, specifically in the following aspects: Regarding optimization efficiency, compared to traditional grid search which requires traversing the entire parameter space (often exceeding a thousand experiments when the parameter dimension exceeds five) and the blind sampling of random search, this invention's "data-driven exploration" mechanism based on Bayesian optimization can compress the effective number of experiments to 1 / 5 to 1 / 3 of traditional methods. For example, for a VLLM inference service scenario containing seven key parameters, only 30 to 50 iterations are needed to approximate the global optimum, significantly reducing the time spent on GPU / NPU and other computing resources. A single optimization process can save more than 60% of computing power costs, making it particularly suitable for resource-constrained scenarios during the debugging phase of large model inference services.
[0055] Figure 2 The diagram shows the overall implementation flow of the Bayesian optimization-based large language model inference service performance optimization method of this application, including the following steps: S1. Parameter Space Construction: Determine the key parameters that affect the performance of the large language model inference service and their value ranges. The key parameters include inference service startup parameters and client request parameters. S2. Design the objective function: Construct an objective function for comprehensively evaluating inference performance. The objective function is based on a weighted fusion of throughput and latency metrics. S3, Bayesian optimization iteration: Based on the Bayesian optimization framework, sequential search is performed, and the optimal parameter combination is gradually approached through initial sampling, construction of a probability model, selection of a collection function and iterative update; S4. Optimal Parameter Verification: Perform multi-dimensional verification on the optimal parameter combination obtained from the Bayesian optimization iteration steps to ensure the reliability of parameter configuration and business adaptability.
[0056] Figure 3 The figure shows a performance optimization system for a large language model inference service based on Bayesian optimization proposed in this application. The system implements the aforementioned performance optimization method for a large language model inference service based on Bayesian optimization during operation. The system includes: The parameter space construction module is used to determine the key parameters that affect the performance of the large language model inference service and their value ranges. The key parameters include inference service startup parameters and client request parameters. The objective function design module is used to construct an objective function that comprehensively evaluates inference performance. The objective function is based on a weighted fusion of throughput and latency metrics. The Bayesian optimization iteration module is used to perform sequential search based on the Bayesian optimization framework. It gradually approaches the optimal parameter combination through initial sampling, building a probability model, selecting a sampling function, and iterative updates. The optimal parameter verification module is used to perform multi-dimensional verification of the optimal parameter combination obtained by Bayesian optimization iteration to ensure the reliability of parameter configuration and business adaptability.
[0057] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of systems, methods, and computer program products according to various embodiments of this application, including architecture, functionality, and operation. In these figures, each block may represent a module, program segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should be noted that each block in the block diagrams and / or flowcharts, and combinations thereof, can be implemented using either a dedicated hardware-based system or a combination of dedicated hardware and computer instructions to achieve the specified function or operation.
[0058] like Figure 4 As shown in the illustration, an embodiment of this application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing a processor-executable computer program, and a communication bus 340. The processor 310, communication interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 executes the executable computer program to implement the steps of the aforementioned Bayesian optimization-based large language model inference service performance optimization method.
[0059] It is understood that, in addition to memory and a processor, this electronic device may also include input devices (such as a keyboard), output devices (such as a display), and other communication modules. These input devices, output devices, and other communication modules all communicate with the processor through I / O interfaces (i.e., input / output interfaces).
[0060] The operations described in this application can be implemented by writing computer program code using one or more programming languages or a combination thereof. The programming languages include, but are not limited to, the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc. Conventional procedural programming languages, such as "C" or similar programming languages.
[0061] The execution methods of program code include, but are not limited to: It runs entirely on the user's computer; Part of it executes on the user's computer, and part of it executes on a remote computer; Execute as a standalone software package; It is executed entirely on a remote computer or server.
[0062] In scenarios involving remote computers, the remote computer can connect to the user's computer via any type of network, including but not limited to local area networks (LANs) or wide area networks (WANs). Furthermore, the remote computer can also connect to external computers through an internet service provider, for example, by utilizing the internet for connection.
[0063] Furthermore, this application also discloses a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of an electronic device, enables the electronic device to perform the various steps of the Bayesian optimization-based large language model inference service performance optimization method disclosed in this application.
[0064] In the context of this application, a computer-readable storage medium refers to a tangible medium capable of storing computer program code and related data. Specific examples include, but are not limited to, the following: (1) Portable computer disk: such as floppy disks and other removable magnetic storage media.
[0065] (2) Hard disk: including mechanical hard disks and solid-state hard disks and other fixed storage devices.
[0066] (3) Random Access Memory (RAM): A volatile storage medium used for temporary storage of data and program code.
[0067] (4) Read-only memory (ROM): a non-volatile storage medium used to store fixed programs and data.
[0068] (5) Erasable programmable read-only memory (EPROM) or flash memory: non-volatile storage media that supports multiple erasures and reprogrammings.
[0069] (6) Fiber optic storage devices: storage media based on fiber optic technology.
[0070] (7) Portable compact disc read-only memory (CD-ROM): a read-only medium that stores data in the form of an optical disc.
[0071] (8) Optical storage devices: such as DVDs, Blu-ray discs and other storage media based on optical principles.
[0072] (9) Magnetic storage devices: such as magnetic tapes, disks and other storage media based on magnetic principles.
[0073] (10) Any suitable combination of the above: for example, combining multiple storage media to meet different storage needs.
[0074] These computer-readable storage media can be used to store the program code and related data described in this application to support program execution and persistent data storage.
[0075] Specifically, according to embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, embodiments of this application relate to a computer program product comprising a computer program carried on a non-transitory computer-readable medium. This computer program contains program code for executing the Bayesian optimization-based large language model inference service performance optimization method disclosed in this application. When this computer program is executed by a processing system, it can achieve the functions defined in the embodiments of this application.
[0076] While the foregoing discussion contains several specific implementation details, these details should not be construed as limiting the scope of this application. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features. Furthermore, this application should also cover other technical solutions formed by any combination of the above-described technical features or their equivalents without departing from the foregoing disclosed concept.
[0077] Those skilled in the art should also understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features, without departing from the spirit and scope of the technical solutions of the embodiments of this application. These modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for performance optimization of large language model inference services based on Bayesian optimization, characterized in that, The method includes: S1. Parameter Space Construction: Determine the key parameters that affect the performance of the large language model inference service and their value ranges. The key parameters include inference service startup parameters and client request parameters. S2. Design the objective function: Construct an objective function for comprehensively evaluating inference performance. The objective function is based on a weighted fusion of throughput and latency metrics. S3, Bayesian optimization iteration: Based on the Bayesian optimization framework, sequential search is performed, and the optimal parameter combination is gradually approached through initial sampling, construction of a probability model, selection of a collection function and iterative update; S4. Optimal Parameter Verification: Perform multi-dimensional verification on the optimal parameter combination obtained from the Bayesian optimization iteration steps to ensure the reliability of parameter configuration and business adaptability.
2. The method according to claim 1, characterized in that, The inference service startup parameters mentioned in step S1 include: tensor parallelism, GPU memory utilization, maximum batch processing token count, and maximum concurrent sequence count; The client request parameters include: number of concurrent requests, number of input tokens, and number of output tokens.
3. The method according to claim 1, characterized in that, The objective function mentioned in step S2 is f(x), which represents the optimal performance value. It is obtained by calculating the comprehensive performance score, and its calculation formula is as follows: ； In the formula, Throughput(x) For average request throughput, Latency(x) To calculate the average token latency, a weighted sum is calculated by combining the average request throughput with the inverse of the average token latency using preset weights. α The weighting coefficient is 0 < α <1.
4. The method according to claim 3, characterized in that, The weighting coefficient α is dynamically adjusted according to the business scenario type. If the business is a "high-concurrency API service", the weight of the average request throughput item is increased; if the business is a "real-time dialogue", the weight of the average token latency item is increased.
5. The method according to claim 1, characterized in that, Step S3 specifically includes: S31. Initial random sampling: Randomly select an initial parameter combination in the parameter space, perform performance testing, and obtain initial observation data; S32. Probabilistic Model Construction: A probabilistic proxy model is trained based on the initial observation data. The probabilistic proxy model is used to predict the performance score and uncertainty of untested parameter combinations. S33. Acquisition function selection: Based on the prediction results of the probabilistic proxy model, the next parameter combination to be evaluated is selected through the acquisition function to balance exploration and utilization; S34. Iterative Update: Perform performance testing on the selected parameter combination, add the newly obtained observation data to the dataset, and update the probabilistic proxy model; S35. Stopping condition judgment: The iteration is terminated when the preset number of iterations or the performance convergence condition is met.
6. The method according to claim 5, characterized in that, The probabilistic surrogate model is a Gaussian process model or a tree-structured Parzen estimator.
7. The method according to claim 5, characterized in that, The acquisition function is any one of the following: The upper confidence bound function is used to select a parameter combination that has a high prediction score and high uncertainty. The expected improvement function is used to select a combination of parameters where the expected improvement value is greater than the current optimal value.
8. The method according to claim 1, characterized in that, Step S4 includes: S41. Stability verification: Perform multiple performance tests continuously using the optimal parameter combination, and evaluate the parameter stability based on the fluctuation range of performance indicators. S42. Business adaptability verification: Perform performance tests in real business scenarios to verify the effectiveness of the optimal parameter combination in practical applications; S43. Comparison and verification: Compare the performance of the optimal parameter combination with the empirical parameter configuration and the grid search results to verify the optimization effect.
9. A performance optimization system for large language model inference services based on Bayesian optimization, characterized in that, The system implements the steps of the Bayesian optimization-based large language model inference service performance optimization method as described in any one of claims 1-8 during system operation, and the system includes: The parameter space construction module is used to determine the key parameters that affect the performance of the large language model inference service and their value ranges. The key parameters include inference service startup parameters and client request parameters. The objective function design module is used to construct an objective function that comprehensively evaluates inference performance. The objective function is based on a weighted fusion of throughput and latency metrics. The Bayesian optimization iteration module is used to perform sequential search based on the Bayesian optimization framework. It gradually approaches the optimal parameter combination through initial sampling, building a probability model, selecting a sampling function, and iterative updates. The optimal parameter verification module is used to perform multi-dimensional verification of the optimal parameter combination obtained by Bayesian optimization iteration to ensure the reliability of parameter configuration and business adaptability.
10. An electronic device, characterized in that, include: Memory and processor; Memory: Used to store computer programs; Processor: for executing the computer program to implement the steps of the Bayesian optimization-based large language model inference service performance optimization method as described in any one of claims 1-8.