MaaS platform large model reasoning parameter automatic optimization acceleration method

Abstract

The invention provides a MaaS platform large model reasoning parameter automatic optimization acceleration method, which comprises the following steps: constructing a standardized container environment and preparing pressure measurement data by detecting hardware configuration; traversing a multi-dimensional parameter space by adopting an intelligent search algorithm, and automatically discovering an optimal reasoning parameter of a specific hardware and model combination through a pressure measurement and performance scoring formula; the optimal parameter combinations corresponding to various hardware models and model versions are persistently stored in a structured database, and a reusable parameter knowledge base is formed; and when the model is deployed, the current hardware environment is automatically identified, and corresponding optimal parameter configuration is intelligently matched and loaded from the parameter library, so that zero-configuration optimal deployment is realized. Through a parameter combination feasibility verification mechanism, parameter combinations which can cause system errors or memory overflow can be found and eliminated in time, it is ensured that the finally selected parameters can improve the performance, and stable operation of the system can also be ensured.

Description

Technical Field

[0001] This invention relates to the field of computer technology, specifically to a method for automatically optimizing and accelerating large model inference parameters on a MaaS platform. Background Technology

[0002] With the rapid development of artificial intelligence technology, large models have demonstrated powerful capabilities in many fields such as natural language processing and image recognition. However, the inference process of large models often consumes a lot of computing resources and time, which has become a serious bottleneck in practical applications, especially in scenarios requiring real-time response. The emergence of MaaS (Model as a Service) platforms has provided users with a convenient way to use large models. Users no longer need to build complex model training and deployment environments themselves; they can quickly use various large models simply through the MaaS platform. However, different hardware models and model versions have different parameter requirements during inference. Finding the optimal acceleration parameters for specific hardware and models to improve inference efficiency is a significant challenge currently facing MaaS platforms. Currently, traditional inference methods cannot automatically and efficiently optimize parameters when faced with different hardware and model combinations, resulting in the throughput performance of large models not being fully utilized and failing to meet the ever-increasing business demands. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, this invention proposes an automatic optimization and acceleration method for large model inference parameters on the MaaS platform. Through intelligent parameter search strategies, especially the optimization of memory utilization parameters, it can maximize the utilization of hardware resources while ensuring system stability. This avoids resource waste or performance bottlenecks caused by improper parameter settings.

[0004] To achieve the above objectives, this invention proposes an automatic optimization and acceleration method for large model inference parameters on a MaaS platform. Step S1: By detecting hardware configuration, building a standardized container environment, and preparing load test data, a basic test environment is established for subsequent parameter optimization.

[0005] Step S2: Employ an intelligent search algorithm to traverse the multi-dimensional parameter space and automatically discover the optimal inference parameters for a specific hardware and model combination through stress testing and performance scoring formulas.

[0006] Step S3: Persistently store the optimal parameter combinations corresponding to various hardware models and model versions in a structured database to form a reusable parameter knowledge base.

[0007] Step S4: When deploying the model, automatically identify the current hardware environment, intelligently match and load the corresponding optimal parameter configuration from the parameter library, and achieve zero-configuration optimized deployment.

[0008] Furthermore, step S1 specifically includes: S11: Detect and verify the hardware environment, and obtain the graphics processor model, unified computing device architecture version and multi-graphics processor topology information; S12: Build a standardized operating environment, use container technology to encapsulate the unified computing device architecture driver and the large language model inference framework, and ensure that the version of the unified computing device architecture in the container is no higher than the host version. S13: Prepare the load testing dataset and load testing scripts for subsequent performance testing; When steps S11-S13 are all completed without errors, step S2 is triggered; Step S2 specifically includes: S21: Define the search space for performance parameters of the inference framework, including five key parameters: tensor parallel size, enable block pre-filling, memory utilization, maximum number of sequences, and scheduler steps. S22: Generate a parameter combination matrix according to the preset parameter search rules; S23: Perform load testing for each parameter combination, send concurrent requests to the inference service through the load testing script, record the request start time, the first byte return time, and the request end time, and calculate the number of output tokens per second and the first byte latency metrics; S24: Calculate the score for each parameter combination using a performance scoring formula based on queuing theory, and determine the optimal parameter combination; S25: Implement search process optimization, including early stopping strategy and breakpoint continuation testing mechanism.

[0009] Furthermore, step S3 specifically includes: After the optimal parameter combination is determined in step S24, the combination is marked as a state to be precipitated, and the process proceeds to step S3. S31: Collect the optimal parameters for different hardware models; S32: Collects the optimal parameters for different open-source large model versions; S33: Store the hardware model, model version, and corresponding optimal parameters in a structured database and create a composite index; Once the parameters are successfully stored, the system enters a standby state, waiting for step S4 to be called. Step S4 specifically includes: S41: Detects current hardware information via an automatic identification engine; S42: Based on the model version and hardware information, use a hash lookup algorithm to retrieve the optimal matching parameters from the parameter library; S43: When an exact match fails, perform a similarity match or parameter inheritance; S44: Start the inference service using the retrieved optimal parameters; The above method achieves zero-configuration optimization of model deployment, shortening the original 1700-hour manual tuning process to automatic completion, saving more than 90% of labor costs.

[0010] Furthermore, the hardware environment detection and verification in step S11 specifically includes: Execute system management interface commands and parse the output results to obtain graphics processor model information; Execute compiler version commands and parse the output to obtain unified computing device architecture version information; Execute the system management interface topology command and parse the output to obtain the graphics processor interconnect topology; If any detection command fails, log an error and terminate subsequent steps.

[0011] Furthermore, the intelligent search algorithm employs a branch and bound strategy, dynamically pruning the search space by monitoring resource utilization and performance indicator trends to reduce invalid parameter combination tests. The parameter search rules in step S22 include: Rule 1: The tensor parallel size parameter is tried in ascending order of 2, 4, 8, and the parameter value is fixed when the model starts successfully; Rule 2: Enable block pre-filling parameters to iterate through the two values, true and false; Rule 3: The video memory utilization parameter is searched from largest to smallest in the order of 0.95, 0.9, 0.8, 0.7, 0.6, and the attempt to try smaller values ​​stops when the startup fails; Rule 4: The maximum sequence number parameter is searched in ascending order of 64, 128, 256, 512, and 1024; Rule 5: The scheduler step number parameter is searched in ascending order of 1, 2, 4, 8, 16, 32, 64; when the maximum sequence number or the scheduler step number parameter causes the startup to fail, attempts to reach larger values ​​are stopped.

[0012] Furthermore, the stress test execution in step S23 specifically includes: Start the inference service and wait for it to be ready; Use a load testing script to send requests concurrently, with the request data coming from a pre-prepared dataset; Real-time collection of performance metrics, including requests processed per second, number of tokens output per second, first byte latency, and average response time; The load test will terminate when the test duration reaches 300 seconds or the request completion rate reaches 95%. The collected performance data is stored in a structured format for use in step S24.

[0013] Furthermore, the performance scoring formula in step S24 is as follows: Score = (Number of lexicons in Group A × Maximum value (First delay of Group B, 1000)) / (Number of lexicons in Group B × Maximum value (First delay of Group A, 1000)) in: The word count of group A represents the number of word bytes output per second for parameter combination A; The first delay of group A represents the delay of the first byte of parameter combination A, in milliseconds; The word count of group B represents the number of word bytes output per second for parameter combination B; The first delay of group B represents the delay of the first byte of parameter combination B, in milliseconds; The maximum value function ensures that the latency value is not less than 1000 milliseconds, based on Miller's law and the user experience threshold. A score greater than 1 indicates that parameter combination A performs better than parameter combination B.

[0014] Furthermore, the optimization of the search process in step S25 includes: Early stop strategy: Monitor the inference service startup logs and immediately terminate the current test when an error or memory overflow error is detected; if the same parameter is increased three times consecutively but the performance metric does not improve, stop subsequent tests for that parameter; Breakpoint continuation mechanism: Records the test status of each parameter combination, and supports continuing the execution of unfinished parameter combination tests from the point of interruption; Resource monitoring: Real-time monitoring of video memory usage; automatically reduces the number of concurrent requests when usage exceeds 95%.

[0015] Furthermore, the structured database in step S33 includes the following fields: load test encoding, output tokens per second, total tokens per second, first letter latency, throughput requests per second, unified computing device architecture version, inference framework version, image name, parameter combination, number of graphics processor cards, graphics processor card model, video memory size, creation time, update time, and status identifier; wherein, a composite index based on graphics processor model and model version is established, supporting millisecond-level queries.

[0016] Furthermore, the parameter retrieval in step S42 specifically includes: Construct query key-value pairs: combine the hardware model and model version to form a unique key; Perform a hash lookup: Search for an exact match in the hash table of the parameter library; Handling multiple results: When multiple matching results exist, select the record with the most recent creation time and the best status; Return parameter combination: Parses the retrieved parameter combination into an executable startup command; The similarity matching and parameter inheritance in step S43 include: Graphics processor hardware similarity matching: When the target graphics processor model does not exist, the closest graphics processor model parameters are selected based on the principle that the difference in video memory size does not exceed 20% and the difference in computing power does not exceed 30%. Model family parameter inheritance: When the target model version does not exist, parameters are inherited from the model family according to the principle of priority for the same parameter size and secondary for the same series; Computing power calculation formula: Computing power value = Floating-point operations per second × Number of tensor cores × Frequency coefficient; Inheritance example: The Tongyi Qianwen 2.5 version 7B instruction model can inherit parameters from the Tongyi Qianwen 2 version 7B instruction model or the Tongyi Qianwen 1.5 version 7B instruction model.

[0017] Furthermore, the method also includes parameter validity verification and anomaly handling steps: After step S44, monitor the running status of the model within 5 minutes after it starts; Record actual throughput and response latency, and calculate the deviation from expected performance; When the actual performance deviates from the expected performance by more than 15%, the parameter re-search process is triggered. If the startup fails three times in a row, it will revert to the default parameter configuration and send an alarm.

[0018] Furthermore, the hardware models include, but are not limited to: NVIDIA A100, V100, A10, H100, RTX 4090, RTX 3090, A40, A30, and T4 series graphics processors; supporting single-card, dual-card, quad-card, and eight-card configurations.

[0019] Furthermore, the open-source large model version includes, but is not limited to: The Tongyi Qianwen series supports all parameter scales from Tongyi Qianwen 7B to 72B, covering Tongyi Qianwen, Tongyi Qianwen 1.5, Tongyi Qianwen 2, Tongyi Qianwen 2.5 and Tongyi Qianwen 3 versions; DeepExploration Series: Supports DeepExploration R1, DeepExploration V3 and their distilled versions; Alpaca Series: Supports all parameter scales from Alpaca 7B to 405B, covering Alpaca, Alpaca 2, Alpaca 3, Alpaca 3.1, and Alpaca 3.2 versions.

[0020] Acceleration systems based on large language models and automatic parameterization engines include: Environmental detection module: used to execute step S1, including hardware detection submodule, environmental verification submodule and data preparation submodule; Parameter search module: used to execute step S2, including parameter generation submodule, load test execution submodule and performance evaluation submodule; Parameter management module: used to execute step S3, including data storage submodule and index management submodule; Automatic loading module: Used to execute step S4, including parameter retrieval submodule and service startup submodule.

[0021] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention provides an automatic optimization and acceleration method for large model inference parameters on a MaaS platform. By automatically searching and optimizing key parameters of the vLLM framework, it achieves a significant improvement in the performance of large model inference. According to test results, the optimal parameters can improve inference performance by 10% to 20% compared to the default parameters. This means that under the same hardware conditions, more concurrent requests can be handled, significantly improving the system's throughput and response speed.

[0022] 2. This invention provides an automatic optimization and acceleration method for large model inference parameters on MaaS platforms, fully automating the parameter tuning process that originally required manual intervention, saving more than 90% of labor costs. According to the documentation, a manual load test takes 0.5 hours; for 350 parameter combinations, this would have required 175 hours of manual debugging time. Now, this can be completed through an automated system, greatly liberating the productivity of technical personnel.

[0023] 3. This invention provides an automatic optimization and acceleration method for large model inference parameters on the MaaS platform, solving the problem of high deployment barriers for large models on the MaaS platform. Users do not need to deeply understand the complex parameter system of the vLLM framework. The system will automatically match the optimal parameters according to the hardware environment and model type, realizing "zero configuration" deployment, allowing non-professionals to quickly deploy high-performance large model services. Attached Figure Description

[0024] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the process of this invention; Figure 2 This is a diagram illustrating GPU model identification; Figure 3 A diagram illustrating CUDA version identification; Figure 4This is a schematic diagram for identifying multi-GPU topology structures. Figure 5 This is a schematic diagram of performance A; Figure 6 This is a schematic diagram of performance B. Detailed Implementation

[0026] The technical solution of the present invention will be more clearly and completely explained below with reference to the accompanying drawings and through the description of preferred embodiments of the present invention.

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0028] like Figure 1 As shown, the present invention is: Step S1: Establish a basic test environment for subsequent parameter optimization by detecting hardware configuration, building a standardized container environment and preparing load test data; Step S2: Employ an intelligent search algorithm to traverse the multi-dimensional parameter space and automatically discover the optimal inference parameters for a specific hardware and model combination through stress testing and performance scoring formulas; Step S3: Persistently store the optimal parameter combinations corresponding to various hardware models and model versions in a structured database to form a reusable parameter knowledge base; Step S4: When deploying the model, automatically identify the current hardware environment, intelligently match and load the corresponding optimal parameter configuration from the parameter library, and achieve zero-configuration optimized deployment.

[0029] As a specific implementation method, to achieve the above objectives, this invention provides a method for accelerating inference applications of large models deployed on a MaaS platform based on large language models and automatic parameterization, including... GPU model identification: Detected and identified using the `nvidia-smi -L` command, such as... Figure 2 As shown; CUDA version: Detected and identified using the `nvcc --version` command, such as... Figure 3 As shown; Multi-GPU topology: Detected and identified using the `nvidia-smi topo -m` command, such as... Figure 4 As shown; Standardized operating environment.

[0030] Standardization is achieved by packaging the runtime environment using Docker technology. The runtime environment includes: CUDA driver and vLLM package. Regarding CUDA version, the requirement is: container CUDA version ≤ host CUDA version. The `nvidia-smi` command can be used within the container to check if CUDA is running correctly. Regarding vLLM version, it is recommended to use the latest version of vLLM to ensure compatibility with newer models and backward compatibility with older models.

[0031] Preparation of load test data and load test scripts.

[0032] The load testing dataset used in this example is the ShareGPT_V3_unfiltered_cleaned_split dataset, but other open-source datasets can be used.

[0033] Load testing script: Refer to the sglang load testing script.

[0034] Optimal parameters are automatically searched.

[0035] A summary of vLLM performance parameters. The main performance parameters are shown in Table 1: Table 1

[0036] Table 1 shows the main parameters. Other vLLM parameters are also applicable to this scheme for testing and output of optimal parameters.

[0037] Automatically search for the optimal parameter combination that offers high performance and saves resources.

[0038] The parameter rules are described as follows: Rule 1: Try increasing the tensor-parallel-size parameter from small to large. If the model starts normally, fix the value. For example, setting it to 2 failed to start the Qwen2.5-32B model, but setting it to 4 succeeded, so fix it at 4.

[0039] Rule 2: The enable-chunked-prefill parameter can be either true or false.

[0040] Rule 3: The `gpu-memory-utilization` parameter is searched from largest to smallest. A smaller setting results in less VRAM usage, but a very small setting may cause startup failure. Setting it to 0.5 is equivalent to halving the number of GPUs, and this has already been ruled out in the previous step. If startup fails and other parameters remain unchanged, smaller parameters will not be tried again. For example, if a startup failure occurs after setting it to 0.8, settings of 0.7 and 0.6 will not be tried for stress testing.

[0041] Rule 4: The parameters max-num-seqs (number of parallel sequences) and num-scheduler-steps (number of schedules per run) are set from smallest to largest according to their value range. Larger parameter settings consume more GPU memory. If startup fails and other parameters remain unchanged, larger subsequent parameters will not be attempted.

[0042] Based on the above rules, a maximum of 350 (2*5*5*7) parameter combinations can be obtained. Each parameter combination is stress-tested, and the output token throughput (tok / s) and first-byte latency (Mean TTFT (ms)) are parsed and recorded.

[0043] After recording, the optimal parameters are obtained by comparing them according to the following rules.

[0044] Suppose there are two records A and B. A.tokens represents the number of tokens output per second by record A, and A.TTFT represents the first byte delay of record A. The calculation formula is as follows: Score = (A.tokens*Max(B.TTFT,1000)) / (B.tokens*Max(A.TTFT,1000)) A score greater than 1 means that A's score is higher than B's. Conversely, a score less than 1 means that A's score is lower than B's.

[0045] Formula explanation: 1. Theoretical basis of the scoring formula: based on the delay-throughput trade-off model of queuing theory and the psychological basis of the 1000ms threshold (Miller's Law).

[0046] 2. Starting from the user satisfaction function U(latency, throughput), the optimal solution is obtained through comparison, and a practical formula is simplified.

[0047] 3. Sensitivity analysis of the formula: The impact of parameter changes on the score. When TTFT is less than 1000ms, it is not included in the score calculation. The higher the number of tokens, the higher the score.

[0048] Example of score calculation: A performance screenshot as follows Figure 5 As shown: According to the screenshot, A: tokens are 2042, and TTFT is 829.

[0049] B performance screenshots are as follows Figure 6 As shown: According to the screenshot, B: tokens is 2183, and TTFT is 463.

[0050] Score = (2042 * Max(463,1000)) / (2183 * Max(829,1000)) = 2042 / 2183 =0.93 Since the score is less than 1, B performs better than A.

[0051] Step 2.3: Feasibility verification of parameter combinations.

[0052] The vllm serve output log is analyzed in real time. When an ERROR or out of memory error is detected, the process is terminated in a timely manner and the next combination parameter verification is performed.

[0053] Search process optimization and breakpoint resume testing.

[0054] With a large number of parameter combinations and a lengthy load testing process, an early stop strategy is used to improve load testing efficiency, and breakpoint resumption of testing improves the success rate. The early stop strategy, due to resource limitations, is described in the parameter rule description; if other parameters remain unchanged, and the same parameter is increased three times consecutively without improving performance, the test will terminate early. Breakpoint resumption of testing, by recording and analyzing logs, allows testing to resume from an incomplete and forcibly interrupted parameter combination, continuing the load testing with subsequent parameters, thus improving the success rate.

[0055] This database collects and summarizes the optimal acceleration parameters for different hardware models, open-source large model versions, and their corresponding vLLMs. Hardware models include, but are not limited to, the following Nvidia cards: A100, V100, A10, H100, RTX 4090, etc. Open-source large model versions include, but are not limited to, the following vendors: Qwen series: Qwen, Qwen1.5, Qwen2, Qwen2.5, Qwen3 versions with various parameters; DeepSeek series: DeepSeek-R1, DeepSeek-V3, DeepSeek-R1-Distill-Qwen-32B, DeepSeek-R1-Distill-Llama-70B, etc.; Llama series: Llama, Llama-2, Llama-3, Llama-3.1, Llama-3.2 versions with various parameters. The performance of the large models is recorded in a database table. The database table structure is designed as follows: CREATE TABLE `air_speedup_record` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `code` varchar(100) DEFAULT NULL COMMENT 'Load Testing Encoding', `output_tokens` bigint(10) DEFAULT NULL COMMENT 'Output tokens / s', `tokens` bigint(10) DEFAULT NULL COMMENT 'Total tokens / s', `head_delay` bigint(10) DEFAULT NULL COMMENT 'First letter delay (ms)', `req` double(10,1) DEFAULT NULL COMMENT 'Throughput req / s', `cuda` varchar(10) DEFAULT NULL COMMENT 'cuda version', `version` varchar(10) DEFAULT NULL COMMENT 'vLLM version', `mirror` varchar(225) DEFAULT NULL COMMENT 'mirror name', `params` varchar(225) DEFAULT NULL COMMENT 'parameter', `gpu_count` bigint(5) DEFAULT NULL COMMENT 'Number of GPU cards', `gpu_type` varchar(50) DEFAULT NULL COMMENT 'gpu card model', `video_memory` bigint(5) DEFAULT NULL COMMENT 'Video memory (G)', `create_time` datetime DEFAULT NULL COMMENT 'Creation Time', `update_time` datetime DEFAULT NULL COMMENT 'Modified time', `status` varchar(1) DEFAULT NULL COMMENT 'Status 1 Normal 2 Optimal 9 Delete', PRIMARY KEY (`id`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Inference Acceleration Stress Test Record'; The automatic identification engine automatically loads the optimal startup parameters. The hardware information acquisition module within the engine automatically identifies information such as the GPU hardware model. Based on the model version and the identified hardware model, the engine retrieves the corresponding optimal parameters from the parameter library and starts the model.

[0056] Query and retrieval rules: If a GPU hardware model match fails, matching can be performed based on similar memory parameters and similar GPU computing power parameters. If a model match fails, parameter inheritance can be performed based on model family. For example, if a match for Qwen2.5-7B-Instruct fails, parameter inheritance can be performed from the Qwen2-7B-Instruct or Qwen1.5-7B-Instruct models.

[0057] The above steps address the high deployment barriers and low inference efficiency of large models on MaaS platforms, improving customer satisfaction. Manual load testing takes 0.5 hours per test, with 3500 parameter combinations, totaling 1700 hours. Automated load testing saves 90% of manpower. Testing results show that optimal parameters improve performance by 10% to 20% compared to default parameters.

[0058] This embodiment provides a method for accelerating the inference application of large models deployed on a model-as-a-service platform based on a large language model inference framework and an automatic parameterization engine. The example is the deployment of the Tongyi Qianwen 2.5 version 32B model on a server equipped with eight NVIDIA A100 graphics processors.

[0059] The system architecture of this embodiment includes four core modules: an environment detection module, a parameter search module, a parameter management module, and an auto-loading module. These modules work together to achieve automatic optimization for inference acceleration.

[0060] In step S1, environment preparation and verification, the system was first equipped with eight A100-SXM4-80GB graphics processors using the `nvidia-smi -L` command. The CUDA version was confirmed to be 12.1 using `nvcc --version`. The GPU interconnect topology was obtained using `nvidia-smitopo -m`, showing that the eight cards were interconnected via NVLink. A standardized Docker runtime environment was then built, using the base image `nvidia / cuda:12.1.0-runtime-ubuntu22.04`, and installing vllm version 0.6.1 and related dependencies. The load testing data used was the ShareGPT_V3_unfiltered_cleaned_split dataset, containing approximately 100,000 real-world dialogue data points, which can fully simulate real-world usage scenarios.

[0061] In step S2, the automatic search for optimal parameters defines a search space for five key parameters: tensor parallel size (3 values: 2, 4, 8), enabling block pre-filling (2 values: true, false), memory utilization (5 values: 0.95 to 0.6), maximum sequence number (5 values: 64 to 1024), and scheduler steps (7 values: 1 to 64). Theoretically, there are 2 × 2 × 5 × 5 × 7 = 350 possible parameter combinations.

[0062] The parameter search employed an intelligent strategy. Initially, it attempted to start the model with a tensor parallel size of 2, but insufficient GPU memory prevented startup. An attempt with 4 also failed, ultimately determining that 8 GPUs were needed for successful deployment of the 32B model. After determining the tensor parallel size, it began iterating through other parameter combinations. Each combination was started using the `vllm serve` command. After waiting 30 seconds to ensure the service was fully ready, a load testing script sent 1000 requests at a rate of 10 requests per second for approximately 300 seconds.

[0063] Key performance metrics, including the number of words per second and the first byte latency, are recorded in real time during load testing. For example, the parameter combination (8, True, 0.9, 256, 4) measures 2834.5 words per second with a first byte latency of 125.3 milliseconds; while the default parameter combination (8, False, 0.9, 256, 1) only achieves 2156.7 words per second with a first byte latency as high as 287.6 milliseconds.

[0064] The performance score is calculated using a formula based on queuing theory. Taking the two sets of parameters mentioned above as examples, the score of the optimized parameter combination relative to the default parameters is: (2834.5×1000) / (2156.7×1000)=1.314, indicating a performance improvement of 31.4%. This formula cleverly balances the two key indicators of throughput and latency, and the 1000-millisecond threshold is based on user experience psychology research.

[0065] The search process employs several optimization strategies. When an "out of memory" error is detected, the current test is immediately terminated, and subsequent attempts at larger parameter values ​​are skipped. If a parameter is increased three times consecutively without performance improvement, the search for that parameter is terminated early. The system supports breakpoint continuation; by recording the index of tested parameter combinations, it can resume from the last position after an unexpected interruption.

[0066] In step S3, parameter sedimentation, the optimal parameters obtained from the search are stored in the `air_speedup_record` table of the MySQL database. This table contains fields such as hardware model, number of GPUs, model name, model parameter scale, optimal parameter combination JSON, and performance metrics. A composite index based on GPU model and model name is established to ensure query efficiency. In this example, the stored records show that under the A100 8-card configuration, the optimal parameters for the Tongyi Qianwen 2.5-32B model are: Tensor Parallelism 8, Block Prefilling Enabled, Memory Utilization 0.9, Maximum Sequence Count 256, and Scheduler Steps 4.

[0067] In step S4, which automatically loads optimal parameters, when a new deployment request arrives, the system automatically detects the current hardware environment and identifies eight A100 graphics cards. Then, based on the user-specified model "Qwen2.5-32B-Instruct", it constructs the query key "A100-SXM4-80GB_8_Qwen2.5-Instruct_32B" and performs a hash lookup in the parameter database. Upon successful matching, it parses the parameter combination in JSON format, constructs the complete startup command, and executes it.

[0068] If an exact match fails, the system will perform a similarity match. For example, when deployed on eight A100-PCIE-40GB graphics cards, the system calculates a 50% difference in video memory, exceeding the 20% threshold, but the computing power is the same. Therefore, it will try to use the parameters of the A100-SXM4-40GB with the same computing power. For cases of model version mismatch, such as deploying the Qwen2.5-32B-Chat model, the system will inherit parameters from Qwen2.5-32B-Instruct because they belong to the same model family and have the same parameter size.

[0069] The implementation yielded significant results. Through automated parameter optimization, model inference performance improved by 31.4%, and first-byte latency decreased by 56.4%. The entire optimization process was fully automated, reducing the tuning work that previously required professional engineers to spend weeks to within 48 hours. The optimized parameters can be directly reused under the same hardware configuration, and the optimal configuration can be loaded in just seconds during new deployments.

[0070] This method has been successfully applied to various hardware and model combinations, including achieving an 18% performance improvement by deploying the Alpaca 2-13B model on four V100 GPUs, a 22% performance improvement by deploying the Deep Exploration V3-7B model on two RTX 4090 GPUs, and a 15% performance improvement by deploying the Tongyi Qianwen 3-14B model on a single H100 GPU. Through continuous accumulation, the parameter library now covers optimal parameters for over 50 mainstream hardware configurations and more than 100 model versions, forming a complete inference acceleration knowledge base.

[0071] A parameter validity verification mechanism ensures system stability. After model startup, the system continuously monitors its running status for 5 minutes. If the actual performance deviates from the expected performance by more than 15%, the system automatically triggers minor parameter fine-tuning. Typically, adjusting the memory utilization or maximum sequence number based on the original parameters is sufficient to restore the expected performance. This adaptive mechanism enables the system to cope with load changes and environmental differences.

[0072] The above-described specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Various modifications, substitutions, and improvements made by those skilled in the art to the technical solutions of the present invention based on the provided textual description and drawings, without departing from the design concept and spirit of the present invention, should all fall within the scope of protection of the present invention. The scope of protection of the present invention is determined by the claims.

Claims

1. A method for automatically optimizing and accelerating inference parameters of large models on a MaaS platform, characterized in that, include: Step S1: Establish a basic test environment for subsequent parameter optimization by detecting hardware configuration, building a standardized container environment and preparing load test data; Step S2: Employ an intelligent search algorithm to traverse the multi-dimensional parameter space and automatically discover the optimal inference parameters for a specific hardware and model combination through stress testing and performance scoring formulas; Step S3: Persistently store the optimal parameter combinations corresponding to various hardware models and model versions in a structured database to form a reusable parameter knowledge base; Step S4: When deploying the model, automatically identify the current hardware environment, intelligently match and load the corresponding optimal parameter configuration from the parameter library, and achieve zero-configuration optimized deployment.

2. The automatic optimization and acceleration method for large model inference parameters on the MaaS platform according to claim 1, characterized in that, Step S1 specifically includes: S11: Detect and verify the hardware environment, and obtain the graphics processor model, unified computing device architecture version and multi-graphics processor topology information; S12: Build a standardized operating environment, use container technology to encapsulate the unified computing device architecture driver and the large language model inference framework, and ensure that the version of the unified computing device architecture in the container is no higher than the host version. S13: Prepare the load testing dataset and load testing scripts for subsequent performance testing; When steps S11-S13 are all completed without errors, step S2 is triggered; Step S2 specifically includes: S21: Define the search space for performance parameters of the inference framework, including five key parameters: tensor parallel size, enable block pre-filling, memory utilization, maximum number of sequences, and scheduler steps. S22: Generate a parameter combination matrix according to the preset parameter search rules; S23: Perform load testing for each parameter combination, send concurrent requests to the inference service through the load testing script, record the request start time, the first byte return time, and the request end time, and calculate the number of output tokens per second and the first byte latency metrics; S24: Calculate the score for each parameter combination using a performance scoring formula based on queuing theory, and determine the optimal parameter combination; S25: Implement search process optimization, including early stopping strategy and breakpoint continuation testing mechanism.

3. The automatic optimization and acceleration method for large model inference parameters on the MaaS platform according to claim 1, characterized in that, Step S3 specifically includes: After the optimal parameter combination is determined in step S24, the combination is marked as a state to be precipitated, and the process proceeds to step S3. S31: Collect the optimal parameters for different hardware models; S32: Collects the optimal parameters for different open-source large model versions; S33: Store the hardware model, model version, and corresponding optimal parameters in a structured database and create a composite index; Once the parameters are successfully stored, the system enters a standby state, waiting for step S4 to be called. Step S4 specifically includes: S41: Detects current hardware information via an automatic identification engine; S42: Based on the model version and hardware information, use a hash lookup algorithm to retrieve the optimal matching parameters from the parameter library; S43: When an exact match fails, perform a similarity match or parameter inheritance; S44: Start the inference service using the retrieved optimal parameters; The above method achieves zero-configuration optimization of model deployment, shortening the original 1700-hour manual tuning process to automatic completion, saving more than 90% of labor costs.

4. The automatic optimization and acceleration method for large model inference parameters on the MaaS platform according to claim 2, characterized in that, The hardware environment detection and verification in step S11 specifically includes: Execute system management interface commands and parse the output results to obtain graphics processor model information; Execute compiler version commands and parse the output to obtain unified computing device architecture version information; Execute the system management interface topology command and parse the output to obtain the graphics processor interconnect topology; If any detection command fails, log an error and terminate subsequent steps.

5. The automatic optimization and acceleration method for large model inference parameters on the MaaS platform according to claim 2, characterized in that, The intelligent search algorithm employs a branch and bound strategy, dynamically pruning the search space by monitoring resource utilization and performance indicator trends to reduce invalid parameter combination tests. The parameter search rules in step S22 include: Rule 1: Tensor parallelization attempts the size parameter in the order of 2, 4, 8, from smallest to largest, and fixes the parameter value when the model starts successfully; Rule 2: Enable block pre-filling parameters to iterate through the two values, true and false; Rule 3: The video memory utilization parameter is searched from largest to smallest in the order of 0.95, 0.9, 0.8, 0.7, 0.6, and the attempt to try smaller values ​​stops when the startup fails; Rule 4: The maximum sequence number parameter is searched in ascending order of 64, 128, 256, 512, and 1024; Rule 5: The scheduler step number parameter is searched in ascending order of 1, 2, 4, 8, 16, 32, 64; when the maximum sequence number or the scheduler step number parameter causes the startup to fail, attempts to reach larger values ​​are stopped.

6. The automatic optimization and acceleration method for large model inference parameters on the MaaS platform according to claim 2, characterized in that, The stress test execution in step S23 specifically includes: Start the inference service and wait for it to be ready; Use a load testing script to send requests concurrently, with the request data coming from a pre-prepared dataset; Real-time collection of performance metrics, including requests processed per second, number of tokens output per second, first byte latency, and average response time; The load test will terminate when the test duration reaches 300 seconds or the request completion rate reaches 95%. The collected performance data is stored in a structured format for use in step S24.

7. The automatic optimization and acceleration method for large model inference parameters on the MaaS platform according to claim 2, characterized in that, The performance scoring formula in step S24 is as follows: Score = (Number of lexicons in Group A × Maximum value (First delay of Group B, 1000)) / (Number of lexicons in Group B × Maximum value (First delay of Group A, 1000)) in: The word count of group A represents the number of word bytes output per second for parameter combination A; The first delay of group A represents the delay of the first byte of parameter combination A, in milliseconds; The word count of group B represents the number of word bytes output per second for parameter combination B; The first delay of group B represents the delay of the first byte of parameter combination B, in milliseconds; The maximum value function ensures that the latency value is not less than 1000 milliseconds, based on Miller's law and the user experience threshold. A score greater than 1 indicates that parameter combination A performs better than parameter combination B.

8. The automatic optimization and acceleration method for large model inference parameters on the MaaS platform according to claim 2, characterized in that, The search process optimization in step S25 includes: Early stop strategy: Monitor the inference service startup logs and immediately terminate the current test when an error or memory overflow error is detected; if the same parameter is increased three times in a row but the performance metric does not improve, stop subsequent tests of the parameter. Breakpoint continuation mechanism: Records the test status of each parameter combination, and supports continuing the execution of unfinished parameter combination tests from the point of interruption; Resource monitoring: Real-time monitoring of video memory usage; automatically reduces the number of concurrent requests when usage exceeds 95%.

9. The automatic optimization and acceleration method for large model inference parameters on the MaaS platform according to claim 3, characterized in that, The structured database in step S33 includes the following fields: load test encoding, output tokens per second, total tokens per second, first letter latency, throughput requests per second, unified computing device architecture version, inference framework version, image name, parameter combination, number of graphics processor cards, graphics processor card model, video memory size, creation time, update time, and status identifier; among them, a composite index based on graphics processor model and model version is established, supporting millisecond-level queries.

10. The automatic optimization and acceleration method for large model inference parameters on the MaaS platform according to claim 3, characterized in that, The parameter retrieval in step S42 specifically includes: Construct query key-value pairs: combine the hardware model and model version to form a unique key; Perform a hash lookup: Search for an exact match in the hash table of the parameter library; Handling multiple results: When multiple matching results exist, select the record with the most recent creation time and the best status; Return parameter combination: Parses the retrieved parameter combination into an executable startup command; The similarity matching and parameter inheritance in step S43 include: Graphics processor hardware similarity matching: When the target graphics processor model does not exist, the closest graphics processor model parameters are selected based on the principle that the difference in video memory size does not exceed 20% and the difference in computing power does not exceed 30%. Model family parameter inheritance: When the target model version does not exist, parameters are inherited from the model family according to the principle of priority for the same parameter size and secondary for the same series; Computing power calculation formula: Computing power value = Floating-point operations per second × Number of tensor cores × Frequency coefficient; Inheritance example: The Tongyi Qianwen 2.5 version 7B instruction model can inherit parameters from the Tongyi Qianwen 2 version 7B instruction model or the Tongyi Qianwen 1.5 version 7B instruction model.

Images