MaaS platform large model inference parameter automatic optimization and acceleration method

Abstract

The application provides a MaaS platform large model inference parameter automatic optimization acceleration method, comprising the following steps: detecting hardware configuration, constructing a standardized container environment, and preparing pressure test data; using an intelligent search algorithm to traverse a multi-dimensional parameter space, automatically discovering optimal inference parameters of a specific hardware and model combination through pressure testing and a performance scoring formula; persistently storing optimal parameter combinations corresponding to various hardware models and model versions to a structured database to form a reusable parameter knowledge base; automatically identifying a current hardware environment during model deployment, intelligently matching and loading corresponding optimal parameter configurations from the parameter base, and realizing zero-configuration optimization deployment. Through a parameter combination feasibility verification mechanism, the application can timely discover and exclude parameter combinations that may cause system errors or memory overflow, ensuring that the finally selected parameters can both improve performance and guarantee stable operation of the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer technology, in particular to a MaaS platform large model inference parameter automatic optimization acceleration method. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, large models have shown great capabilities in natural language processing, image recognition and many other fields. However, the inference process of large models often consumes a large amount of computing resources and time, which has become a serious bottleneck in actual application, especially in scenarios that require real-time response.

[0003] The emergence of MaaS (Model as a Service) platform provides users with a convenient way to use large models. Users do not need to build complex model training and deployment environments, but only need to use various large models through the MaaS platform. However, different hardware models and model versions have different requirements for parameters during inference. How to find the optimal acceleration parameters for specific hardware and models to improve inference efficiency is an important challenge currently faced by MaaS platforms.

[0004] Currently, traditional inference methods cannot automatically and efficiently optimize parameters when facing different hardware and model combinations, resulting in the throughput performance of large models being difficult to fully exploit, and being unable to meet the growing business demands. SUMMARY

[0005] To overcome the shortcomings of the prior art, the present application proposes a MaaS platform large model inference parameter automatic optimization acceleration method, which can maximize the use of hardware resources while ensuring system stability through intelligent parameter search strategies, especially the optimization of memory utilization rate parameters. Avoids resource waste or performance bottlenecks caused by improper parameter settings.

[0006] To achieve the above purpose, the present application proposes a MaaS platform large model inference parameter automatic optimization acceleration method, step S1: detecting hardware configuration, building a standardized container environment and preparing stress test data to establish a basic test environment for subsequent parameter optimization.

[0007] Step S2: Use intelligent search algorithms to traverse multi-dimensional parameter space, automatically find the optimal inference parameters for specific hardware and model combinations through stress testing and performance scoring formulas.

[0008] Step S3: persistently store the optimal parameters corresponding to various hardware models and model versions into a structured database to form a reusable parameter knowledge base.

[0009] Step S4: Automatically identify the current hardware environment during model deployment, intelligently match and load the corresponding optimal parameter configuration from the parameter library, and realize zero-configuration optimization deployment.

[0010] Further, step S1 specifically includes:

[0011] S11: Detect and verify the hardware environment, obtain the graphics processor model, unified computing device architecture version, and multi-graphics processor topology information;

[0012] S12: Build a standardized running environment, encapsulate the unified computing device architecture driver and large language model inference framework using container technology, and ensure that the unified computing device architecture version within the container is not higher than the host version;

[0013] S13: Prepare stress testing data sets and stress testing scripts for subsequent performance testing;

[0014] When steps S11-S13 are all completed and no errors occur, step S2 is triggered;

[0015] Step S2 specifically includes:

[0016] S21: Define the inference framework performance parameter search space, including five key parameters: tensor parallel size, enable block pre-filling, memory utilization, maximum sequence number, and scheduler step number;

[0017] S22: Generate a parameter combination matrix according to the preset parameter search rule;

[0018] S23: Perform stress testing on each parameter combination, send concurrent requests to the inference service through the stress testing script, record the request start time, first byte return time, and request end time, and calculate the output token per second and first byte latency indicators;

[0019] S24: Calculate the score of each parameter combination using the performance scoring formula based on queuing theory, and determine the optimal parameter combination;

[0020] S25: Implement search process optimization, including early stopping strategy and breakpoint resuming mechanism.

[0021] Further, step S3 specifically includes:

[0022] After step S24 determines the optimal parameter combination, mark the combination as a pending state and enter step S3;

[0023] S31: Collect optimal parameters for different hardware models;

[0024] S32: Collect optimal parameters for different open-source large model versions;

[0025] S33: Store the hardware model, model version and corresponding optimal parameters into a structured database and establish a composite index;

[0026] When the parameters are successfully stored, the system enters a standby state and waits for the call of step S4;

[0027] Step S4 specifically includes:

[0028] S41: Detect the current hardware information through an automatic recognition engine;

[0029] S42: According to the model version and hardware information, use a hash lookup algorithm to retrieve the matching optimal parameters from the parameter library;

[0030] S43: When the exact match fails, perform similarity matching or parameter inheritance;

[0031] S44: Start the inference service using the retrieved optimal parameters;

[0032] Through the above method, the zero-configuration optimization of model deployment is realized, and the manual tuning process originally requiring 1700 hours is shortened to automatic completion, saving more than 90% of labor costs.

[0033] Further, the hardware environment detection and verification in step S11 specifically includes:

[0034] Executing a system management interface command, parsing the output result to obtain the graphics processor model information;

[0035] Executing a compiler version command, parsing the output result to obtain the unified computing device architecture version information;

[0036] Executing a system management interface topology command, parsing the output result to obtain the graphics processor interconnection topology structure;

[0037] When any detection command fails, record an error log and terminate the subsequent steps.

[0038] Further, the intelligent search algorithm adopts a branch and bound strategy, dynamically prunes the search space by monitoring the resource usage rate and performance index change trend, and reduces invalid parameter combination testing, and the parameter search rule in step S22 includes:

[0039] Rule one: The tensor parallel size parameter is tried from small to large in the order of 2, 4, 8, and when the model is successfully started, the parameter value is fixed;

[0040] Rule two: Enable the block pre-padding parameter to traverse the true and false values;

[0041] Rule three: search the GPU utilization rate parameter in the order of 0.95, 0.9, 0.8, 0.7, 0.6 from large to small, and stop trying smaller values when the startup fails;

[0042] Rule four: search the maximum sequence number parameter in the order of 64, 128, 256, 512, 1024 from small to large;

[0043] Rule five: search the scheduler step number parameter in the order of 1, 2, 4, 8, 16, 32, 64 from small to large; wherein, when the maximum sequence number or the scheduler step number parameter causes the startup to fail, stop trying larger values.

[0044] Further, the stress test execution in the step S23 specifically includes:

[0045] Start the inference service and wait for the service to be ready;

[0046] Send requests concurrently using the stress test script, and the request data comes from the pre-prepared data set;

[0047] Collect performance indicators in real time, including the number of requests processed per second, the number of word units output per second, the first byte latency, and the average response time;

[0048] Terminate the stress test when the stress test duration reaches 300 seconds or the request completion rate reaches 95%;

[0049] Structure the collected performance data for storage for use in step S24.

[0050] Further, the performance score formula in the step S24 is:

[0051] Score = (Group A Word Unit Number × Max(Group B First Delay, 1000)) / (Group B Word Unit Number × Max(Group A First Delay, 1000))

[0052] Wherein:

[0053] Group A Word Unit Number represents the number of word units output per second of parameter combination A;

[0054] Group A First Delay represents the first byte latency of parameter combination A, in milliseconds;

[0055] Group B Word Unit Number represents the number of word units output per second of parameter combination B;

[0056] Group B First Delay represents the first byte latency of parameter combination B, in milliseconds;

[0057] The Max function ensures that the latency value is not less than 1000 milliseconds, based on the user experience threshold of Miller's Law;

[0058] Scores greater than 1 indicate that the performance of parameter combination A is better than parameter combination B.

[0059] Further, the search process optimization in step S25 includes:

[0060] Early stopping strategy: monitor inference service startup logs, terminate the current test immediately when an error or memory overflow error is detected; stop subsequent testing of the same parameter when it is increased for 3 consecutive times but the performance indicator is not improved;

[0061] Breakpoint resume mechanism: record the test state of each parameter combination to support the continuation of incomplete parameter combination testing from the breakpoint;

[0062] Resource monitoring: real-time monitoring of GPU usage, automatically reducing the number of concurrent requests when the usage exceeds 95%.

[0063] Further, the structured database in step S33 includes the following fields: load test code, output tokens per second, total tokens per second, first letter delay, throughput per second request number, unified computing device architecture version, inference framework version, image name, parameter combination, number of graphics processing unit cards, graphics processing unit card model, GPU size, creation time, update time and state identifier; Wherein, a composite index based on graphics processing unit model and model version is established to support millisecond-level queries.

[0064] Further, the parameter retrieval in step S42 specifically includes:

[0065] Build query key value: concatenate hardware model and model version to form a unique key;

[0066] Perform hash lookup: find an exact match in the hash table of the parameter library;

[0067] Handle multiple results: when there are multiple matching results, select the record with the latest creation time and the optimal state;

[0068] Return parameter combination: parse the retrieved parameter combination into an executable startup command;

[0069] The similarity matching and parameter inheritance in step S43 include:

[0070] Graphics processing unit hardware similarity matching: when the target graphics processing unit model does not exist, select the closest graphics processing unit model parameter according to the principle that the GPU size difference does not exceed 20% and the computing power difference does not exceed 30%;

[0071] Model family parameter inheritance: when the target model version does not exist, inherit parameters from the model family according to the principle of priority of the same parameter size and second priority of the same series;

[0072] Computing formula: computing power value = floating point operation frequency per second × tensor core number × frequency coefficient;

[0073] Inheritance example: the 2.5 version 7B instruction model of TongYiQianWan can inherit parameters from the 2 version 7B instruction model of TongYiQianWan or the 1.5 version 7B instruction model of TongYiQianWan.

[0074] Further, the method further comprises parameter validity verification and exception handling steps:

[0075] After step S44, the running state within 5 minutes after the model is started is monitored;

[0076] The actual throughput and response delay are recorded, and the deviation from the expected performance is calculated;

[0077] When the actual performance deviates from the expected performance by more than 15%, the parameter re-search process is triggered;

[0078] When the continuous startup fails for 3 times, the default parameter configuration is rolled back and an alarm is sent.

[0079] Further, the hardware model includes but is not limited to: Nvidia A100, V100, A10, H100, RTX 4090, RTX 3090, A40, A30, T4 series graphics processors; support single card, double card, four card and eight card configuration.

[0080] Further, the open source large model version includes but is not limited to:

[0081] TongYiQianWan series: support all parameter scales of TongYiQianWan 7B to 72B, covering TongYiQianWan, TongYiQianWan 1.5, TongYiQianWan 2, TongYiQianWan 2.5, TongYiQianWan 3 version;

[0082] Deep exploration series: support deep exploration R1, deep exploration V3 and its distillation version;

[0083] Alpaca series: support all parameter scales of Alpaca 7B to 405B, covering Alpaca, Alpaca 2, Alpaca 3, Alpaca 3.1, Alpaca 3.2 version.

[0084] The acceleration system based on large language model and automatic parameterization engine includes:

[0085] Environment detection module: used to execute step S1, containing hardware detection submodule, environment verification submodule and data preparation submodule;

[0086] Parameter search module: used to execute step S2, containing parameter generation submodule, stress test execution submodule and performance evaluation submodule;

[0087] Parameter management module: used for executing step S3, containing a data storage submodule and an index management submodule;

[0088] Automatic loading module: used for executing step S4, containing a parameter retrieval submodule and a service starting submodule.

[0089] Compared with the prior art, the beneficial effects of the present application are:

[0090] 1. The present application provides a MaaS platform large model inference parameter automatic optimization acceleration method, which realizes a substantial improvement in the performance of large model inference by automatically searching and optimizing the key parameters of the vLLM framework. According to the test results, the optimal parameters can improve the inference performance by 10% to 20% compared with the default parameters, which means that under the same hardware conditions, more concurrent requests can be processed, significantly improving the throughput and response speed of the system.

[0091] 2. The present application provides a MaaS platform large model inference parameter automatic optimization acceleration method, which completely automates the parameter tuning process that originally required manual intervention, saving more than 90% of labor costs. According to the document description, manual stress testing takes 0.5 hours, and for 350 parameter combinations, the original 175 hours of manual debugging time can now be completed through the automated system, greatly liberating the productivity of technical personnel.

[0092] 3. The present application provides a MaaS platform large model inference parameter automatic optimization acceleration method, which solves the problem of high deployment threshold for large models on the MaaS platform. Users do not need to deeply understand the complex parameter system of the vLLM framework, and the system will automatically match the optimal parameters according to the hardware environment and model type, realizing "zero configuration" deployment, so that non-professionals can also quickly deploy high-performance large model services. BRIEF DESCRIPTION OF DRAWINGS

[0093] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0094] Figure 1 is a flowchart of the present application;

[0095] Figure 2 is a GPU model identification schematic diagram;

[0096] Figure 3 is a CUDA version identification schematic diagram;

[0097] Figure 4A performance diagram for;

[0098] Figure 5 A performance diagram for;

[0099] Figure 6 A performance diagram for. DETAILED DESCRIPTION

[0100] The technical solutions of the present application will be described more clearly and completely below by combining the drawings and the description of the preferred embodiments of the present application.

[0101] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below by combining specific examples.

[0102] As Figure 1 shown, the present application is:

[0103] Step S1: Establish a basic test environment for subsequent parameter optimization by detecting hardware configuration, building a standardized container environment and preparing stress test data;

[0104] Step S2: Automatically discover the optimal inference parameters of a specific hardware and model combination by traversing the multi-dimensional parameter space using an intelligent search algorithm, stress testing and a performance scoring formula;

[0105] Step S3: Persistently store the optimal parameter combinations corresponding to various hardware models and model versions to a structured database to form a reusable parameter knowledge base;

[0106] Step S4: Automatically identify the current hardware environment when deploying a model, intelligently match and load the corresponding optimal parameter configuration from the parameter library, and realize zero-configuration optimized deployment.

[0107] As a specific embodiment, in order to achieve the above purpose, the present application provides an application method for accelerating inference of a large model deployed on a MaaS platform based on a large language model and automatic parameterization, which comprises

[0108] GPU model identification: detected and identified by the nvidia-smi -L command, as shown in Figure 2 ;

[0109] CUDA version: detected and identified by the nvcc --version command, as shown in Figure 3 ;

[0110] Multi-GPU topology: detected and identified by the nvidia-smi topo -m command, as shown in Figure 4 ;

[0111] Standardization of running environment.

[0112] The running environment is packaged using docker technology to achieve standardization. The running environment includes: cuda driver, vLLM package. Regarding the cuda version, the requirements are: container CUDA version <= host CUDA version, and the nvidia-smi command can be used to detect whether the cuda is running normally in the container. Regarding the vLLM version, the latest version of vLLM is recommended, which can be compatible with new version models and backward compatible with low version models.

[0113] Prepare the pressure test data and pressure test script.

[0114] The example pressure test dataset is ShareGPT_V3_unfiltered_cleaned_split, and other open source datasets can be used.

[0115] Pressure test script: refer to the sglang pressure test script.

[0116] Optimal parameter automatic search.

[0117] vLLM performance parameter arrangement. The main performance parameters are shown in Table 1:

[0118] Table 1

[0119]

[0120] The main parameters in Table 1 are applicable to this scheme for testing and outputting the optimal parameters.

[0121] Automatic search for the best parameter combination with high performance and resource saving.

[0122] The parameter rules are described as follows:

[0123] Rule 1: The tensor-parallel-size (vector parallel) parameter is tried from small to large, and the value is fixed if the model can be normally started. For example, setting it to 2 fails to start the Qwen2.5-32B model, trying to set it to 4 succeeds, and fixing it to 4.

[0124] Rule 2: The enable-chunked-prefill parameter takes the value range true, false.

[0125] Rule 3: The gpu-memory-utilization (GPU memory pre-occupation ratio) parameter is searched from large to small, and the smaller the setting, the less GPU memory is occupied. Setting a small value may cause startup failure. After setting 0.5, the meaning is the same as reducing the number of cards by half, which has been excluded in the previous step. After startup failure, if other parameters remain unchanged, do not try smaller parameters. For example, after setting 0.8 fails to start, do not try to set 0.7 and 0.6 for pressure testing.

[0126] Rule 4: max-num-seqs (number of parallel sequences), num-scheduler-steps (number of single scheduling) parameters are arranged from small to large according to the value range. The larger the parameter setting, the more GPU memory is consumed. If other parameters remain unchanged after startup fails, subsequent larger parameters will not be attempted.

[0127] According to the above rules, a total of 350 (2*5*5*7) parameter combinations can be obtained. The parameters are tested, and the output tokens (Output token throughput (tok / s)), first byte latency (Mean TTFT (ms)) performance are analyzed and recorded.

[0128] After recording, the best parameters are obtained by comparing the following rules.

[0129] Suppose there are two records A and B. A.tokens is the output tokens per second of A, and A.TTFT is the first byte latency of A. The calculation formula is as follows:

[0130] Score = (A.tokens*Max(B.TTFT,1000)) / (B.tokens*Max(A.TTFT,1000))

[0131] Score>1, which means that the score of A is higher than that of B. Conversely, it means that the score of A is lower than that of B.

[0132] Explanation of the formula:

[0133] 1. Theoretical basis of the scoring formula: Delay-throughput trade-off model based on queuing theory, psychological basis of 1000ms threshold (Miller's law).

[0134] 2. From the user satisfaction function U(latency, throughput), the optimal solution is obtained by comparison method, and the practical formula is simplified.

[0135] 3. Sensitivity analysis of the formula: The parameter change has an impact on the score. When TTFT is less than 1000ms, it is not involved in the score calculation. The higher the tokens, the higher the score.

[0136] Example of score calculation:

[0137] The performance screenshot of A is as shown in Figure 5 .

[0138] According to the screenshot, A: tokens is 2042, and TTFT is 829.

[0139] The performance screenshot of B is as shown in Figure 6 .

[0140] According to the screenshot, B:tokens is 2183 and TTFT is 463.

[0141] Score = (2042 * Max(463,1000)) / (2183 * Max(829,1000)) = 2042 / 2183 =0.93

[0142] Score is less than 1, so B performance is better than A.

[0143] Step 2.3: Parameter combination feasibility verification.

[0144] Real-time analysis of vllm serve output logs, terminate the process when ERROR or out of memory is detected, and proceed to the next combination parameter verification.

[0145] Search process optimization and breakpoint resume.

[0146] Parameter combination is large, and the stress test process is long. Early stop strategy is used to improve stress test efficiency and breakpoint resume to improve operation success rate. Early stop strategy resource limit causes early stop reference parameter rule description; when other parameters remain unchanged, the same parameter is continuously improved for three times but does not improve performance, and the process is terminated in advance. Breakpoint resume can start from the parameter combination that has not been completed and has been forced to interrupt by recording and analyzing logs, continue subsequent parameter stress test, and improve the success rate of operation.

[0147] Different hardware models, open source large model versions, and corresponding vLLM optimal acceleration parameters are precipitated. Hardware models include but are not limited to the following Nvidia cards: A100, V100, A10, H100, RTX 4090, etc. Open source large model versions include but are not limited to the following manufacturers: Qwen series: Qwen, Qwen1.5, Qwen2, Qwen2.5, Qwen3 version parameter model; DeepSeek series: DeepSeek-R1, DeepSeek-V3, DeepSeek-R1-Distill-Qwen-32B, DeepSeek-R1-Distill-Llama-70B, etc.; Llama series: Llama, Llama-2, Llama-3, Llama-3.1, Llama-3.2 version parameter model. Large model performance is recorded to a database table. The database table structure is designed as follows:

[0148] CREATE TABLE `air_speedup_record` (

[0149] `id` bigint(20) NOT NULL AUTO_INCREMENT,

[0150] `code` varchar(100) DEFAULT NULL COMMENT 'Load Testing Encoding',

[0151] `output_tokens` bigint(10) DEFAULT NULL COMMENT 'Output tokens / s',

[0152] `tokens` bigint(10) DEFAULT NULL COMMENT 'Total tokens / s',

[0153] `head_delay` bigint(10) DEFAULT NULL COMMENT 'First letter delay (ms)',

[0154] `req` double(10,1) DEFAULT NULL COMMENT 'Throughput req / s',

[0155] `cuda` varchar(10) DEFAULT NULL COMMENT 'cuda version',

[0156] `version` varchar(10) DEFAULT NULL COMMENT 'vLLM version',

[0157] `mirror` varchar(225) DEFAULT NULL COMMENT 'mirror name',

[0158] `params` varchar(225) DEFAULT NULL COMMENT 'parameter',

[0159] `gpu_count` bigint(5) DEFAULT NULL COMMENT 'Number of GPU cards',

[0160] `gpu_type` varchar(50) DEFAULT NULL COMMENT 'gpu card model',

[0161] `video_memory` bigint(5) DEFAULT NULL COMMENT 'Video memory (G)',

[0162] `create_time` datetime DEFAULT NULL COMMENT 'Creation Time',

[0163] `update_time` datetime DEFAULT NULL COMMENT 'Modified time',

[0164] `status` varchar(1) DEFAULT NULL COMMENT 'Status 1 Normal 2 Optimal 9 Delete',

[0165] PRIMARY KEY (`id`)

[0166] ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Inference Acceleration Stress Test Record';

[0167] The automatic identification engine automatically loads the optimal startup parameters. The hardware information acquisition module within the engine automatically identifies information such as the GPU hardware model. Based on the model version and the identified hardware model, the engine retrieves the corresponding optimal parameters from the parameter library and starts the model.

[0168] Query and retrieval rules: If a GPU hardware model match fails, matching can be performed based on similar memory parameters and similar GPU computing power parameters. If a model match fails, parameter inheritance can be performed based on model family. For example, if a match for Qwen2.5-7B-Instruct fails, parameter inheritance can be performed from the Qwen2-7B-Instruct or Qwen1.5-7B-Instruct models.

[0169] The above steps address the high deployment barriers and low inference efficiency of large models on MaaS platforms, improving customer satisfaction. Manual load testing takes 0.5 hours per test, with 3500 parameter combinations, totaling 1700 hours. Automated load testing saves 90% of manpower. Testing results show that optimal parameters improve performance by 10% to 20% compared to default parameters.

[0170] This embodiment provides a method for accelerating the inference application of large models deployed on a model-as-a-service platform based on a large language model inference framework and an automatic parameterization engine. The example is the deployment of the Tongyi Qianwen 2.5 version 32B model on a server equipped with eight NVIDIA A100 graphics processors.

[0171] The system architecture of this embodiment includes four core modules: an environment detection module, a parameter search module, a parameter management module, and an auto-loading module. These modules work together to achieve automatic optimization for inference acceleration.

[0172] In the environment preparation and verification step S1, first, it is detected by the nvidia-smi -L command that the system is equipped with 8 A100-SXM4-80GB graphic processors, the CUDA version is confirmed to be 12.1 by nvcc --version, and the GPU interconnection topology is obtained by nvidia-smitopo -m, which shows that the 8 cards are interconnected by NVLink high-speed interconnection. Then the standard Docker running environment is built, the base image is selected as nvidia / cuda:12.1.0-runtime-ubuntu22.04, and vllm 0.6.1 version and related dependencies are installed. The stress test data uses the ShareGPT_V3_unfiltered_cleaned_split data set, which contains about 100,000 real conversation data and can fully simulate the actual use scenario.

[0173] In the step S2 of automatic search of optimal parameters, the search space of five key parameters is defined: the tensor parallel size contains three values of 2, 4 and 8, the enable block pre-filling contains two values of true and false, the memory utilization contains five values of 0.95 to 0.6, the maximum sequence number contains five values of 64 to 1024, and the scheduler step number contains seven values of 1 to 64. In theory, there are 2x2x5x5x7=350 parameter combinations.

[0174] The parameter search adopts an intelligent strategy. First, try to start the model with a tensor parallel size of 2, and find that the memory is insufficient to start, continue to try 4, and still fail, and finally determine that 8 cards in parallel are needed to successfully deploy the 32B model. After determining the tensor parallel size, start traversing other parameter combinations. Each combination starts the service through the vllm serve command, waits for 30 seconds to ensure that the service is fully ready, and then sends 1000 requests using the stress test script, with a request rate of 10 per second and a stress test duration of about 300 seconds.

[0175] During the stress test, the key performance indicators are recorded in real time, including the number of output tokens per second and the first byte latency. For example, the parameter combination (8, True, 0.9, 256, 4) measures the number of output tokens per second as 2834.5 and the first byte latency as 125.3 milliseconds; while the default parameter combination (8, False, 0.9, 256, 1) can only reach 2156.7 tokens per second, with a first byte latency of 287.6 milliseconds.

[0176] The performance score is calculated using a formula based on queuing theory. Using the two sets of parameters above as an example, the score of the optimized parameter combination relative to the default parameters is: (2834.5 x 1000) / (2156.7 x 1000) = 1.314, indicating a 31.4% performance improvement. This formula cleverly balances the two key indicators of throughput and latency, and the 1000 millisecond threshold is based on user experience psychology research.

[0177] The search process implements several optimization strategies. When an "out of memory" error is detected, the current test is immediately terminated and the subsequent attempt with larger parameter values is skipped. When a parameter is increased for 3 consecutive times but the performance does not improve, the search for that parameter is ended early. The system supports breakpoint resuming, by recording the index of the parameter combination that has been tested, it can continue from the last position after an unexpected interruption.

[0178] In step S3 parameter sedimentation, the optimal parameters obtained by the search are stored in the air_speedup_record table of the MySQL database. This table contains fields such as hardware model, number of GPUs, model name, model parameter size, optimal parameter combination JSON, performance indicators, and establishes a composite index based on GPU model and model name to ensure query efficiency. For this example, the stored record shows that the optimal parameters for the Qwen2.5-32B model under the A100 8-card configuration are tensor parallelism 8, enable block prefetching, memory utilization 0.9, maximum sequence number 256, and scheduler step number 4.

[0179] In step S4 automatic loading of optimal parameters, when a new deployment request arrives, the system automatically detects the current hardware environment and identifies 8 A100 graphics cards. Then, according to the user-specified model "Qwen2.5-32B-Instruct", it constructs the query key "A100-SXM4-80GB_8_Qwen2.5-Instruct_32B" and performs a hash lookup in the parameter library. After successful matching, the JSON-formatted parameter combination is parsed, a complete startup command is constructed, and execution is performed.

[0180] If the exact match fails, the system will perform a similarity match. For example, when deployed on 8 A100-PCIE-40GB graphics cards, the system calculates the memory difference to be 50%, which exceeds the 20% threshold, but the computing power is the same, so it will try to use the parameters of A100-SXM4-40GB with the same computing power. For cases where the model version does not match, such as deploying the Qwen2.5-32B-Chat model, the system will inherit the parameters from Qwen2.5-32B-Instruct because they belong to the same model family and have the same parameter size.

[0181] The implementation effect is remarkable. Through automatic parameter optimization, the model inference performance is improved by 31.4%, and the first byte delay is reduced by 56.4%. The entire optimization process is fully automated, and the tuning work that originally takes weeks of professional engineers is compressed to 48 hours. The optimized parameters can be directly reused under the same hardware configuration, and the optimal configuration can be loaded in seconds when deployed.

[0182] The method has been successfully applied to various hardware and model combinations, including deploying alpaca 2-13B model on 4 V100s to achieve 18% performance improvement, deploying deep exploration V3-7B model on 2 RTX 4090s to achieve 22% performance improvement, and deploying tongyi qianwen 3-14B model on a single H100 to achieve 15% performance improvement. Through continuous accumulation, the parameter library has covered the optimal parameters of more than 50 mainstream hardware configurations and more than 100 model versions, forming a complete inference acceleration knowledge base.

[0183] The parameter effectiveness verification mechanism ensures the stability of the system. After the model is started, the running state is continuously monitored for 5 minutes. If the actual performance deviates from the expected value by more than 15%, the system will automatically trigger a small range of parameter fine-tuning. Usually, adjusting the memory utilization or maximum sequence number based on the original parameters can restore the expected performance. This adaptive mechanism enables the system to respond to load changes and environmental differences.

[0184] The above specific embodiments only describe the preferred embodiments of the present application, and do not limit the protection scope of the present application. Without departing from the design concept and spirit of the present application, various modifications, substitutions and improvements of the technical solutions of the present application made by those skilled in the art according to the description and drawings provided by the present application should belong to the protection scope of the present application. The protection scope of the present application is determined by the claims.

Claims

1. A method for automatically optimizing and accelerating MaaS platform large model inference parameters, characterized in that, Comprise: Step S1: Establish a basic test environment for subsequent parameter optimization by detecting hardware configuration, building a standardized container environment, and preparing stress testing data; specifically including: S11: Detect and verify the hardware environment, obtain the graphics processor model, unified computing device architecture version, and multi-graphics processor topology information; S12: Build a standardized running environment, encapsulate the unified computing device architecture driver and large language model inference framework using container technology, and ensure that the unified computing device architecture version in the container is not higher than the host version; S13: Prepare stress testing data sets and stress testing scripts for subsequent performance testing; When steps S11-S13 are all completed and no errors occur, trigger step S2; Step S2: Use an intelligent search algorithm to traverse the multi-dimensional parameter space, automatically discover the optimal inference parameters for a specific hardware and model combination through stress testing and a performance scoring formula; step S2 specifically includes: S21: Define the performance parameter search space of the inference framework, including five key parameters: tensor parallelism size, enable block pre-filling, memory utilization, maximum sequence number, and scheduler step number; S22: Generate a parameter combination matrix according to the pre-set parameter search rules; the intelligent search algorithm uses a branch and bound strategy to dynamically prune the search space by monitoring resource usage and performance indicator trends, reducing invalid parameter combination testing; the parameter search rules include: Rule one: The tensor parallelism size parameter is tried from small to large in the order of 2, 4, 8, and when the model successfully starts, the parameter value is fixed; Rule two: The enable block pre-filling parameter is traversed with true and false values; Rule three: The memory utilization parameter is searched from large to small in the order of 0.95, 0.9, 0.8, 0.7, and 0.6, and when the startup fails, stop trying smaller values; Rule four: The maximum sequence number parameter is searched from small to large in the order of 64, 128, 256, 512, and 1024; Rule five: The scheduler step number parameter is searched from small to large in the order of 1, 2, 4, 8, 16, 32, and 64; Wherein, when the maximum sequence number or scheduler step number parameter causes startup failure, stop trying larger values; S23: Perform stress testing on each parameter combination, send concurrent requests to the inference service through the stress testing script, record the request start time, first byte return time, and request end time, and calculate the output token per second and first byte latency indicators; the stress testing specifically includes: Start the inference service and wait for the service to be ready; Use the stress testing script to send requests concurrently, and the request data comes from the pre-prepared data set; Real-time collection of performance indicators, including the number of requests processed per second, output tokens per second, first byte latency, and average response time; Structurize the collected performance data for use in step S24; S24: Calculate the score of each parameter combination using a performance scoring formula based on queuing theory to determine the optimal parameter combination; the performance scoring formula is: Score = (Group A Token Count x Max(Group B First Delay, 1000)) / (Group B Token Count x Max(Group A First Delay, 1000)) Wherein: The word number of the first group represents the number of words per second output by the parameter combination A; The first delay of the first group represents the first byte delay of the parameter combination A, in milliseconds; The word number of the second group represents the number of words per second output by the parameter combination B; The first delay of the second group represents the first byte delay of the parameter combination B, in milliseconds; The Max function ensures that the delay value is not less than 1000 milliseconds, which is a threshold based on psychological research on user experience; A score greater than 1 indicates that the performance of the parameter combination A is better than that of the parameter combination B; S25: Implement search process optimization, including early stop strategy and breakpoint resume mechanism; Step S3: The optimal parameter combination corresponding to various hardware models and model versions is stored in a structured database to form a reusable parameter knowledge base; Step S4: Automatically identify the current hardware environment when deploying the model, intelligently match and load the corresponding optimal parameter configuration from the parameter library, and realize zero-configuration optimization deployment.

2. The MaaS platform large model inference parameter automatic optimization and acceleration method according to claim 1, characterized in that, Step S3 specifically includes: After determining the optimal parameter combination in step S24, mark the combination as a pending state and enter step S3; S31: Collect optimal parameters under different hardware models; S32: Collect optimal parameters of different open source large model versions; S33: Store the hardware model, model version and corresponding optimal parameters in a structured database and establish a composite index; When the parameters are successfully stored, the system enters a standby state and waits for the call of step S4; Step S4 specifically includes: S41: Detect the current hardware information through an automatic recognition engine; S42: Retrieve the matching optimal parameters from the parameter library using a hash lookup algorithm based on the model version and hardware information; S43: When exact matching fails, perform similarity matching or parameter inheritance; S44: Start the inference service using the retrieved optimal parameters.

3. The MaaS platform large model inference parameter automatic optimization and acceleration method according to claim 1, characterized in that, The hardware environment detection and verification in step S11 specifically includes: Execute system management interface commands to parse the output results and obtain graphics processor model information; Execute compiler version commands to parse the output results and obtain unified computing device architecture version information; Execute system management interface topology commands to parse the output results and obtain graphics processor interconnection topology structure; When any detection command fails, record an error log and terminate the subsequent steps.

4. The MaaS platform large model inference parameter automatic optimization and acceleration method according to claim 1, characterized in that, The search process optimization in step S25 includes: Early stop strategy: Monitor the inference service startup log and terminate the current test immediately when an error or memory overflow error is detected; stop subsequent testing of the parameter when the same parameter increases for 3 consecutive times but the performance indicator does not improve; Breakpoint resume mechanism: Record the test status of each parameter combination to support continuing the testing of incomplete parameter combinations from the breakpoint; Resource monitoring: Real-time monitoring of GPU usage, automatically reducing the number of concurrent requests when the usage exceeds 95%.

5. The MaaS platform large model inference parameter automatic optimization acceleration method according to claim 2, characterized in that, The structured database in the step S33 includes the following fields: stress code, output word units per second, total word units per second, first letter delay, throughput per second request number, unified computing device architecture version, inference framework version, image name, parameter combination, number of graphics processor cards, graphics processor card model, memory size, creation time, update time and state identifier; wherein, a composite index based on graphics processor model and version is established to support millisecond-level query.

6. The MaaS platform large model inference parameter automatic optimization and acceleration method according to claim 2, characterized in that, The parameter retrieval in the step S42 specifically includes: Building a query key value: splicing the hardware model and version to form a unique key; Performing hash lookup: searching for an exact match in the hash table of the parameter library; Processing multiple results: when there are multiple matching results, selecting the record with the latest creation time and the optimal state; Returning the parameter combination: parsing the retrieved parameter combination into an executable start command; The similarity matching and parameter inheritance in the step S43 include: Graphics processor hardware similarity matching: when the target graphics processor model does not exist, selecting the closest graphics processor model parameter according to the principle that the memory size difference does not exceed 20% and the computing power difference does not exceed 30%; Model family parameter inheritance: when the target model version does not exist, inheriting parameters from the model family according to the principle of priority of the same parameter size and second priority of the same series; Computing power calculation formula: computing power value = number of floating point operations per second × number of tensor cores × frequency coefficient.

Images