Training method of reasoning deployment configuration performance prediction model of large language model and reasoning deployment configuration recommendation method and device

Abstract

The invention discloses a training method of an inference deployment configuration performance prediction model of a large language model and an inference deployment configuration recommendation method and device.The method comprises the steps that a first structural feature, a first interaction feature and throughput performance data under different structural parameters and configuration parameters of a sample model are obtained; comprising structure parameters, configuration parameters and operation resource quantization parameters of the large language model; taking the throughput performance data as a regression target, and training a regression model by using the first structural feature and the first interaction feature to obtain a reasoning deployment configuration regression model; the reasoning deployment configuration regression model is used for predicting throughput performance data of the large language model under given configuration. According to the method, systematic modeling is carried out on a complex mapping relation among a model structure, input and output characteristics, a parallel strategy, concurrent configuration and throughput performance, and the performance is accurately predicted in a data driving mode, so that optimal configuration is recommended, and efficient and reusable language model reasoning deployment optimization is realized.

Description

Technical Field

[0001] This invention relates to the field of high-performance reasoning and system optimization technology for large language models, and in particular to training methods, inference deployment configuration recommendation methods, and apparatus for predicting the performance of large language model inference deployment configuration. Background Technology

[0002] With the explosive growth in the parameter size of large language models and the deepening of application scenarios, deploying efficient large language model (LM) inference services has become a critical and challenging task. To fully utilize the computing power of modern GPU clusters, various parallel strategies have been proposed, including tensor parallelism (TP), pipeline parallelism (PP), data parallelism (DP), and various hybrid parallel schemes. Different parallel strategies have their own advantages and disadvantages in terms of latency, throughput, communication overhead, and resource utilization. How to select the optimal parallel strategy for a specific application scenario remains an open question.

[0003] The configuration selection of current mainstream inference systems (such as vLLM) heavily relies on engineers' experience and repeated trial and error. Faced with rapidly iterating model architectures (such as different families like LLaMA, Qwen, and Gemma) and complex structural metadata (number of layers, number of heads, hidden dimensions, etc.), finding the optimal parallel strategy (TP / PP combination) and determining concurrency and batch processing parameters for each new model or input / output scenario to optimize throughput and response time becomes a time-consuming, labor-intensive process that makes it difficult to reuse experience. This fragmented manual tuning is not only inefficient and computationally expensive, but also severely hinders the agility of model deployment and its portability across different hardware environments. For example, CN119150994A discloses a method for optimizing inference in large language models. This method uses the configuration parameter data of the large language model and the configuration analysis results to automatically tune the configuration parameters based on reinforcement learning, obtaining the optimal parameter configuration. The configuration parameters include GPU configuration information, model architecture, and model type, such as the number of model layers, hidden layer dimension, feedforward neural network intermediate layer dimension, maximum sequence length, number of attention heads and vocabulary length, GPU memory, bandwidth, message latency, etc. It also includes the number of parameters, computational cost, memory usage, and latency obtained through further static analysis. This tuning method requires obtaining a large number of parameters and analyzing the detailed structure of the model, thus it is inefficient and computationally expensive. Summary of the Invention

[0004] Objectives of this invention: The first objective is to provide a training method for a large language model's inference deployment configuration performance prediction model, thereby improving the accuracy and generalization ability of the prediction model; the second objective is to provide an inference deployment configuration recommendation method, thereby improving the efficiency and accuracy of configuration recommendation; the third objective is to provide a training device for a large language model's inference deployment configuration performance prediction model; and the fourth objective is to provide an inference deployment configuration recommendation device.

[0005] Technical solution: The training method of the inference deployment configuration performance prediction model of a large language model according to the present invention includes:

[0006] Obtain the first structural feature and the first interaction feature of the sample model; the sample model includes several different large language models, the first structural feature includes the structural parameters and configuration parameters of the large language model, and the first interaction feature includes the quantification parameters of the running resources of the large language model under different structural parameters and configuration parameters;

[0007] Obtain the throughput performance data of the sample model under different structural parameters and configuration parameters;

[0008] Using the throughput performance data as the regression target, a regression model is trained using the first structural feature and the first interaction feature to obtain the inference deployment configuration regression model; the inference deployment configuration regression model is used to predict the throughput performance data of a large language model under a given configuration.

[0009] Furthermore, the structural parameters of the sample model include numerical features of the model structure, input length, and output length; the configuration parameters of the sample model include parallel configuration parameters and concurrent configuration parameters.

[0010] Furthermore, the throughput performance data includes throughput, first token latency, and generation latency, and a multi-objective regression model is trained using the throughput performance data as the target.

[0011] Furthermore, the first interaction features include: single GPU memory usage estimation features, computation-to-communication ratio features, and pipeline bubble rate features.

[0012] Furthermore, when the structural parameters of the obtained sample model are missing or insufficient, feature completion is performed using preset derivation rules.

[0013] Furthermore, the feature completion using preset derivation rules includes:

[0014] Based on the name and / or first structural features of the obtained sample model, the sample model is classified into a preset model type, and the structural parameters of the sample model are completed using the architecture design parameters of the preset model type as prior knowledge.

[0015] Furthermore, the completion of the structural parameters of the sample model includes:

[0016] Calculate and complete the attention head dimension of the sample model based on the known hidden layer dimension and the known number of attention heads;

[0017] And / or, calculate and complete the attention grouping query ratio of the sample model based on the known number of attention heads;

[0018] And / or, determine the known feedforward network amplification ratio according to the preset model type corresponding to the sample model, and calculate and complete the feedforward network intermediate layer dimension of the sample model according to the known feedforward network amplification ratio;

[0019] And / or, calculate and complete the total number of parameters of the sample model based on the known hidden layer dimension and the known total number of layers.

[0020] Furthermore, before training the regression model using the throughput performance data as the regression target and the first structural feature and the first interaction feature, the method further includes normalizing the first structural feature and the first interaction feature of the sample model; the normalization method includes:

[0021] The numerical features of the model structure are logarithmically transformed and then normalized to standard scores.

[0022] The parallel configuration parameters are normalized according to their proportion relative to the total number of GPUs.

[0023] The concurrency configuration parameters are normalized according to their ratio to the video memory capacity.

[0024] The present invention provides a method for recommending inference deployment configurations for large language models, comprising:

[0025] The second structural feature and the second interaction feature of the target model are obtained; the target model is a large language model to be deployed; the structural feature includes the structural parameters of the target model and several sets of configuration parameters, and the second interaction feature includes the quantitative parameters of the running resources of the target model under different configuration parameters;

[0026] The second structural feature and the second interaction feature are input into the trained inference deployment configuration regression model to obtain the predicted throughput performance data of the target model under different combinations of configuration parameters; the inference deployment configuration regression model is trained according to the training method of the inference deployment configuration performance prediction model of the large language model;

[0027] The configuration parameters corresponding to the highest predicted throughput performance data are recommended for the target model.

[0028] Furthermore, based on the structural parameters of the target model, a hardware-constrained configuration space pruning strategy is used to enumerate all feasible parallel and concurrent configurations, resulting in several sets of configuration parameters.

[0029] Furthermore, the hardware-constrained configuration space pruning strategy includes memory constraint pruning, parallelism constraint pruning, and communication constraint pruning.

[0030] The memory constraint pruning excludes configuration parameter combinations that exceed the GPU memory capacity.

[0031] The parallelism constraint pruning is to exclude configuration parameter combinations that do not satisfy tensor parallelism constraints and / or pipeline parallelism constraints.

[0032] The parallelism constraint is: the tensor parallelism can be divided equally by the total number of attention heads and the number of key value heads in the model; the pipeline parallelism constraint is: the pipeline parallelism can be divided equally by the total number of layers in the model.

[0033] The communication constraint pruning prioritizes reducing configuration parameter combinations with high communication overhead; wherein, the communication overhead under different configuration parameters is evaluated based on the deployed GPU cluster hardware topology.

[0034] Furthermore, the configuration parameters recommended to the target model corresponding to the highest throughput performance data include:

[0035] Configuration parameters corresponding to prediction throughput performance data with prediction uncertainty not lower than their corresponding thresholds are selected as candidate configuration parameters. From the candidate configuration parameters, the configuration parameter corresponding to the highest prediction throughput performance data is selected and recommended to the target model.

[0036] Furthermore, a threshold for prediction uncertainty is determined based on the standard deviation of the training error of the regression model configured by the inference deployment.

[0037] Furthermore, the structural parameters of the target model include numerical features of the model structure, input length, and output length; the configuration parameters of the target model include parallel configuration parameters and concurrent configuration parameters.

[0038] Furthermore, the second interaction feature includes: single GPU memory usage estimation feature, computation-to-communication ratio feature, and pipeline bubble rate feature.

[0039] Furthermore, when the structural parameters of the acquired target model are missing, feature completion is performed using preset derivation rules.

[0040] Furthermore, the feature completion using preset derivation rules includes:

[0041] Based on the name and / or first structural features of the target model, the target model is classified into a preset model type, and the structural parameters of the target model are completed using the architectural design parameters of the model of that type as prior knowledge.

[0042] Furthermore, the completion of the structural parameters of the target model includes:

[0043] Calculate and complete the attention head dimension of the target model based on the known hidden layer dimension and the known number of attention heads;

[0044] And / or, calculate and complete the attention grouping query ratio of the target model based on the known number of attention heads of the target model;

[0045] And / or, determine the known feedforward network amplification ratio according to the preset model type corresponding to the target model, and calculate and complete the feedforward network intermediate layer dimension of the target model according to the known feedforward network amplification ratio;

[0046] And / or, calculate and complete the total number of parameters of the target model based on the known hidden layer dimension and the known total number of layers.

[0047] Furthermore, before inputting the second structural feature and the second interaction feature into the trained inference deployment configuration regression model, the method further includes: normalizing the second structural feature and the second interaction feature of the target model; the normalization method includes:

[0048] The numerical features of the model structure are logarithmically transformed and then normalized to standard scores.

[0049] The parallel configuration parameters are normalized according to their proportion relative to the total number of GPUs.

[0050] The concurrency configuration parameters are normalized according to their ratio to the video memory capacity.

[0051] The present invention provides a method for deploying a large language model for inference, comprising: deploying the large language model to be deployed according to the configuration parameters recommended in the method for recommending the configuration of the large language model for inference deployment.

[0052] Based on the same inventive concept, the training device for a large language model inference deployment configuration performance prediction model according to the present invention includes:

[0053] The acquisition module is used to acquire the first structural feature and the first interaction feature of the sample model; the sample model includes several different large language models, the first structural feature includes the structural parameters and configuration parameters of the large language model, and the first interaction feature includes the quantification parameters of the running resources of the large language model under different structural parameters and configuration parameters.

[0054] Simultaneously, it is used to obtain the throughput performance data of the sample model under different structural parameters and configuration parameters;

[0055] The training module is used to train a regression model using the throughput performance data as the regression target and the first structural feature and the first interaction feature to obtain an inference deployment configuration regression model; the inference deployment configuration regression model is used to predict the throughput performance data of a large language model under a given configuration.

[0056] Based on the same inventive concept, the present invention provides a large language model reasoning deployment configuration recommendation device, comprising:

[0057] The acquisition module is used to acquire the second structural features and second interaction features of the target model; the target model is a large language model to be deployed; the structural features include the structural parameters of the target model and several sets of configuration parameters, and the second interaction features include the runtime resource quantification parameters of the target model under different configuration parameters;

[0058] The prediction module is used to input the second structural feature and the second interaction feature into the trained inference deployment configuration regression model to obtain the predicted throughput performance data of the target model under different combinations of configuration parameters; the inference deployment configuration regression model is trained according to the training method of the inference deployment configuration performance prediction model of the large language model.

[0059] The recommendation module is used to select the configuration parameters corresponding to the highest predicted throughput performance data and recommend them to the target model.

[0060] Based on the same inventive concept, the present invention provides a reasoning deployment device for a large language model, comprising:

[0061] The deployment module is used to deploy the large language model to be deployed according to the configuration parameters recommended in the inference deployment configuration recommendation method of the large language model.

[0062] Based on the same inventive concept, the electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the training method of the inference deployment configuration performance prediction model of the large language model, or the inference deployment configuration recommendation method of the large language model, or the inference deployment method of the large language model.

[0063] Based on the same inventive concept, the computer-readable storage medium of the present invention stores a computer program, which, when executed by a processor, implements a training method for a performance prediction model of inference deployment configuration of a large language model, or a recommendation method for inference deployment configuration of a large language model, or a method for inference deployment of a large language model, when loaded onto a processor.

[0064] Based on the same inventive concept, the computer program product of the present invention includes a computer program that, when executed by a processor, implements a training method for a performance prediction model of inference deployment configuration of a large language model, or a recommendation method for inference deployment configuration of a large language model, or a method for inference deployment of a large language model, based on the computer program being loaded into the processor.

[0065] Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows: (1) The present invention systematically regresses and quantitatively predicts the complex relationship between model structure, input and output features, parallel strategy, concurrent configuration and throughput performance through a data-driven approach. Based on the regression model constructed from real benchmark test data, the present invention can accurately capture the complex nonlinear relationship between configuration parameters and performance, and the prediction results are reliable. (2) Based on the structural parameters of the target model and the characteristics of the deployment scenario, the present invention automatically predicts the performance under different configurations in a very short time and recommends the optimal parallel strategy, thereby completely avoiding the time-consuming, labor-intensive and difficult-to-reuse experimental tuning process in the traditional method. This automation capability significantly reduces the technical threshold and computing cost of large-scale language model deployment. (3) Through rigorous feature engineering and confidence evaluation, the present invention ensures the high accuracy and strong generalization ability of the prediction model, and can flexibly adapt to different model families, scales and diverse input and output scenarios. (4) The multi-level recommendation results output by the present invention, including the optimal parallel strategy and concurrent configuration parameters, provide sufficient basis for users to make trade-off decisions between performance and various constraints, and enhance the practicality and flexibility of the solution. Attached Figure Description

[0066] Figure 1 This is a flowchart of the training method for the inference deployment configuration performance prediction model in Embodiment 1 of the present invention.

[0067] Figure 2 This is a flowchart of the inference deployment configuration recommendation method in Embodiment 2 of the present invention.

[0068] Figure 3 The following is a detailed flowchart of the recommended method for inference deployment configuration in Embodiment 2 of the present invention.

[0069] Figure 4 This is a structural diagram of the training device for the inference deployment configuration performance prediction model in Embodiment 4 of the present invention.

[0070] Figure 5 This is a structural diagram of the recommended device for inference deployment configuration in Embodiment 5 of the present invention. Detailed Implementation

[0071] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0072] Example 1

[0073] like Figure 1 As shown, the training method for the inference deployment configuration performance prediction model of the large language model of the present invention includes the following steps.

[0074] Step 101: Obtain the first structural feature and the first interaction feature of the sample model; the sample model includes several different large language models, the first structural feature includes the structural parameters and configuration parameters of the large language model, and the first interaction feature includes the quantification parameters of the running resources of the large language model under different structural parameters and configuration parameters.

[0075] Step 102: Obtain the throughput performance data of the sample model under different structural parameters and configuration parameters;

[0076] Step 103: Using the throughput performance data as the regression target, train a regression model using the first structural feature and the first interaction feature to obtain the inference deployment configuration regression model; the inference deployment configuration regression model is used to predict the throughput performance data of a large language model under a given configuration.

[0077] Specifically, in step 101 of this embodiment, a feature vector for predicting parallel computing performance is extracted and constructed. The feature vector includes a first structural feature of the sample model and a first interaction feature constructed based on domain knowledge. The feature vector is designed as a model-independent unified numerical representation, enabling the regression model to predict the performance of new model structures that have not appeared in the training data, thereby achieving cross-model generalization recommendation capability.

[0078] The first structural feature includes the structural parameters and configuration parameters of the large language model. The structural parameters of the sample model include the numerical features of the model structure, the input length, and the output length; the configuration parameters of the sample model include parallel configuration parameters and concurrent configuration parameters. As an example, the first structural feature includes the numerical features of the model structure, the input and output lengths, the parallel configuration parameters, and the concurrent configuration parameters. The concurrent configuration parameters are used to model resource contention and batch processing efficiency in multi-request concurrent scheduling scenarios, enabling the method to adapt to different inference scheduling strategies. The maximum number of concurrent requests reflects the request queue depth, and the maximum number of batch processing tokens reflects the upper limit of the computational load for a single inference.

[0079] Furthermore, when the structural parameters of the acquired sample model are missing or insufficient, feature completion is performed using preset derivation rules. The feature completion using preset derivation rules includes: classifying the sample model into a preset model type based on the acquired sample model's name and / or first structural feature, and using the architectural design parameters of that model type as prior knowledge to complete the structural parameters of the sample model.

[0080] In this embodiment, when constructing feature vectors, if the number of model structural features extracted from the model configuration file or related materials is insufficient to meet the feature dimensions required for training, then the missing structural features are filled in using pre-defined derivation rules based on prior knowledge of the large language model architecture to ensure the consistency of the feature vectors. Feature filling using pre-defined derivation rules includes the following steps:

[0081] Step (1), Model Identification Step: Based on the name of the large language model or the numerical features of the known model structure, classify the target model into the preset model type. The model types include LLaMA series, Qwen series, Gemma series, Mistral series, GLM series, Baichuan series, InternLM series, Phi series, Yi series, Facon series, and MPT series.

[0082] Step (2), Model-Specific Derivation: Based on the identified model type, apply the architectural design priors of that model type, including: feedforward network amplification factor, default value of attention head dimension, number of attention heads, and hidden layer dimension;

[0083] Step (3), Parameter Completion: Based on the Transformer architecture and known model structure numerical features, calculate the missing structural features and perform parameter completion, including: the dimensions of the intermediate layers of the feedforward network. Equal to the hidden layer dimension Multiply by the feedforward amplification ratio Round up to the preset granularity; Attention grouping query ratio Equal to the number of attention heads Divide by the number of key values Attention head dimension Equal to the hidden layer dimension Divide by the number of attention heads Alternatively, the default value (128) of the Transformer structure can be used.

[0084] As an example, the parameter completion derivation rules based on the standard Transformer architecture prior are as follows:

[0085] a) Attention head dimension completion: If the attention head dimension is missing... However, the dimensions of the hidden layers are known. and attention head count Then calculate using the following formula: ;

[0086] b) Derivation of the attention ratio for grouped queries: If the attention ratio for grouped queries is missing... However, the number of attention heads is known. and key value head number Then calculate using the following formula: This ratio is used to determine whether the model uses optimized attention mechanisms such as GQA / MQA, and directly affects... The memory usage and computation pattern of the cache;

[0087] c) Feedforward network dimension estimation: If the intermediate layer dimensions of the feedforward network are missing... The system first determines the model name (e.g., ...). , Find typical feedforward network amplification ratios in the preset architecture pattern library. If found, calculate using the following formula: ;

[0088] d) Total parameter estimation: If the total parameters are missing However, given other key dimensions, they can be estimated using approximate formulas based on Transformer block parameters: , This represents the number of model layers. This formula is derived from the observation that the attention layer and feedforward layer in the standard Transformer are dominated by parameters, and the coefficients can be fine-tuned depending on whether a bias term is used, etc.

[0089] Based on this derived rule, structural features that have a key impact on throughput performance (such as grouped query attention ratio) can be automatically derived from a limited or partially known set of model parameters. These derived features are directly related to the model's key-value cache usage, computational mode, and communication overhead, enabling the regression model to more accurately characterize the impact of different model structures (such as whether GQA / MQA is used) on performance, thereby improving the accuracy and generalization ability of predictions.

[0090] The first interactive feature includes quantification parameters of the runtime resources of the large language model under different structural and configuration parameters. As an example, the first interactive feature includes single GPU memory usage estimation features (calculated based on the number of model parameters, KV cache size, and input / output length), computation-to-communication ratio features (calculated based on tensor parallelism and model hidden layer dimension to calculate the ratio of AllReduce communication overhead to computational load), and pipeline bubble rate features (calculated based on pipeline parallelism and micro-batch number to calculate pipeline idle ratio).

[0091] Specifically, in step 102 of this embodiment, on the hardware platform, for various large language models, actual throughput performance data of the large language models are collected to construct an initial structured dataset. The structured dataset is presented in the form of a CSV file. The throughput performance data covers different input and output lengths, different parallel configurations, and different concurrency settings. The throughput performance data includes throughput TPS, first token delay TTFT, and generation delay TPOT.

[0092] Specifically, the execution order of steps 101 and 102 can be interchanged.

[0093] Specifically, in step 103 of this embodiment, before training the inference deployment configuration regression model using the throughput performance data as the regression target and utilizing the first structural feature and the first interaction feature, the method further includes normalizing the first structural feature and the first interaction feature of the sample model.

[0094] As an example, this embodiment employs a grouping normalization strategy to normalize the first structural features with different dimensions and the first interaction features constructed based on domain knowledge. The numerical features of the model structure are normalized after logarithmic transformation to eliminate the influence of parameter magnitude differences. Parallel configuration parameters are normalized proportionally to the total number of GPUs; concurrency configuration parameters are normalized proportionally to the video memory capacity. Specifically, it includes the following:

[0095] e) Normalization of numerical features of the model structure: This set of features mainly includes the number of model parameters, number of layers, number of attention heads, etc., and their numerical range is usually large and skewed. To eliminate the influence of differences in parameter magnitudes and make them approximately conform to a normal distribution, a logarithmic transformation followed by standard scores is used. The normalization strategy. The specific formula is:

[0096] in, These are the original eigenvalues. For a very small positive number, such as To prevent taking the logarithm of zero, and The feature in the training dataset is processed by... The transformed mean and standard deviation.

[0097] f) Normalization of Parallel Configuration Parameters: This set of features includes pipeline parallelism and tensor parallelism, and their reasonable values ​​are closely related to the total number of available GPUs. Normalization is performed using a ratio relative to the total number of GPUs, ensuring that their values ​​fall within a certain range. Within the interval. The specific formula is:

[0098]

[0099] in, The original parallelism of a certain parallel strategy (e.g., tensor parallelism is 4). This represents the total number of currently available GPUs.

[0100] g) Normalization of Concurrency Configuration Parameters: This group of features mainly involves the maximum number of batch processing tokens and the maximum number of concurrent requests for the inference service. These parameters are limited by the GPU's memory capacity. Normalization is performed using a ratio relative to the memory capacity to reflect the GPU's memory usage. The specific formula is:

[0101]

[0102] in, The original concurrency configuration parameter values, The approximation is the sum of the model weights and the cache required to process a token (in GB). The cache required to process a token can be estimated by multiplying the total number of model layers by the number of key headers, the dimension of the attention header, the number of bytes, and then multiplying by two. This refers to the GPU's video memory capacity (in GB).

[0103] Specifically, in step 103 of this embodiment, the throughput performance data includes throughput, first token delay, and generation delay, and a multi-objective regression model is trained using the throughput performance data as the target.

[0104] This embodiment uses throughput performance data as the regression target. Based on the normalized feature vector, an inference deployment configuration regression model is trained to predict throughput performance data under a given configuration. In this step, the inference deployment configuration regression model adopts a multi-task learning architecture, simultaneously predicting multiple throughput performance data. By sharing the semantic feature representations corresponding to the numerical features of different model structures, it learns the correlation between different throughput performance data, thereby improving prediction accuracy.

[0105] In practical applications, the multi-objective regression model uses numerical features of the model structure to predict the performance of unseen models. When the target model is not in the training set, a feature vector is constructed based on its structural parameters, and knowledge transfer prediction is performed using the performance patterns learned by the trained model on similar structural models.

[0106] Example 2

[0107] like Figure 2 and Figure 3 As shown in this embodiment, a method for recommending the inference deployment configuration of a large language model includes the following steps:

[0108] Step 201: Obtain the second structural feature and the second interaction feature of the target model; the target model is a large language model to be deployed; the structural feature includes the structural parameters of the target model and several sets of configuration parameters, and the second interaction feature includes the quantitative parameters of the running resources of the target model under different configuration parameters;

[0109] Step 202: Input the second structural feature and the second interaction feature into the trained inference deployment configuration regression model to obtain the predicted throughput performance data of the target model under different combinations of configuration parameters; the inference deployment configuration regression model is trained according to the method described in Example 1;

[0110] Step 203: Select the configuration parameters corresponding to the highest predicted throughput performance data and recommend them to the target model.

[0111] Specifically, in step 201 of this embodiment, the structural parameters of the target model in the second structural feature include numerical features of the model structure, input length, and output length. The configuration parameters of the target model in the second structural feature include parallel configuration parameters and concurrent configuration parameters. The second interaction feature includes runtime resource quantification parameters under different configuration parameters. When the obtained structural parameters of the target model are missing, feature completion is performed using preset derivation rules. The derivation rules are the same as in Embodiment 1, and will not be repeated in this embodiment.

[0112] Specifically, in this embodiment, based on the structural parameters of the target model, a hardware-constrained configuration space pruning strategy is used to enumerate all feasible parallel and concurrent configurations to obtain several sets of configuration parameters.

[0113] Furthermore, the hardware-constrained configuration space pruning strategy includes memory-constrained pruning, parallelism-constrained pruning, and communication-constrained pruning. Each pruning strategy is described below.

[0114] The memory constraint pruning involves estimating the number of parameters in the target large language model and the KV cache, and excluding configuration combinations that exceed the GPU memory capacity. Specifically, based on the number of parameters in the target large language model, the preset inference sequence length, and the candidate batch size, the maximum total memory required by a single GPU for each configuration combination is calculated and compared with the physical GPU memory capacity. Configurations with a total requirement exceeding a preset threshold of physical capacity are discarded. The calculation formula is:

[0115]

[0116]

[0117]

[0118] in, For model parameter display memory, For KV caching, The number of parameters in the target model. The number of bytes for the target model weight data type. The candidate tensor parallelism. The preset maximum number of tokens for batch processing. The number of layers in the target model. For each attention head, the key value is the number of heads (for multi-head attention, Equals the total number of attention heads; for grouped query attention, (less than the total number of attention heads) Dimensions for each attention head.

[0119] The parallelism constraint pruning involves filtering the feasibility of parallel configurations based on the inherent structural parameters of the target large language model. This includes tensor parallelism constraints and pipeline parallelism constraints, specifically:

[0120] Tensor parallelism constraint: candidate tensor parallelism It must be able to divide the total number of attention heads of the model at the same time. With key value head number That is, it needs to meet and ;

[0121] Pipeline parallelism constraints: candidate pipeline parallelism It must be divisible by the total number of layers in the model. That is, it needs to meet .

[0122] The communication constraint pruning is based on the hardware topology of the GPU cluster being deployed, evaluating and ranking the communication overhead of different configurations. Specifically, it identifies GPU groups directly connected via high-speed interconnects (such as NVLink) within the cluster, i.e., an NVLink domain. For candidate configurations, if the candidate's tensor parallelism... If the allocated GPU spans multiple NVLink domains, then its... If the cross-link latency of the group communication is higher than that of the intra-NVLink domain communication, the priority of this configuration should be reduced, and the configuration in which the tensor parallel group is completely located in the same NVLink domain should be given priority.

[0123] This pruning strategy, based on hardware constraints (GPU memory, parallelism, and communication topology), can quickly eliminate invalid or unreasonable configuration combinations in the early stages of the search, significantly reducing the search space. This not only improves recommendation efficiency but also ensures the feasibility of the recommendation results on the target hardware platform, avoiding deployment failures caused by resource configuration conflicts.

[0124] Specifically, in step 202 of this embodiment, before constructing a feature vector by combining the second structural features and the second interaction features of the target model and inputting it into the trained inference deployment configuration regression model, the feature vector is also normalized. The normalization method is the same as in embodiment 1, and will not be described again in this embodiment.

[0125] Specifically, in step 203 of this embodiment, the confidence level of the predicted throughput performance data from step 202 is evaluated. The prediction error distribution on the validation set during the training phase is used as a reference, and the standard deviation of the prediction error is used as a measure of uncertainty. Specifically, after training is completed, the standard deviation of the error between the predicted value and the true value on the validation set is calculated, and the value at the 90th percentile of the standard deviation values ​​arranged in descending order is set as the confidence threshold. If the uncertainty of the predicted throughput performance data exceeds this threshold, the prediction result is marked as low confidence, and prediction results with high confidence are selected.

[0126] Specifically, for the filtered predicted throughput performance data, the combination of parallel and concurrent configurations with the highest predicted throughput performance is selected as the optimal configuration for output. This includes the optimal parallel strategy and concurrent configuration parameters. When outputting the optimal configuration, multi-objective optimization is performed on three performance metrics: throughput, first token latency, and generation latency. The optimal configuration set is recommended to users, allowing them to choose between throughput and latency based on their actual business scenarios.

[0127] The large language model inference deployment configuration recommendation method in this embodiment abstracts the concurrent configuration parameters into a numerical representation of the maximum number of concurrent requests and the maximum number of batch processing tokens, thereby achieving generalized adaptation to different inference scheduling strategies. The inference scheduling strategies include continuous batch processing scheduling, iterative scheduling, or static batch processing scheduling. This method can provide configuration recommendations without retraining for a specific scheduling strategy.

[0128] Example 3

[0129] The inference deployment method for a large language model described in this embodiment deploys the large language model to be deployed according to the configuration parameters recommended in the method described in Embodiment 2.

[0130] Example 4

[0131] like Figure 4 As shown in this embodiment, a training device for a performance prediction model of inference deployment configuration for a large language model includes:

[0132] The acquisition module is used to acquire the first structural feature and the first interaction feature of the sample model; the sample model includes several different large language models, the first structural feature includes the structural parameters and configuration parameters of the large language model, and the first interaction feature includes the quantification parameters of the running resources of the large language model under different structural parameters and configuration parameters.

[0133] Simultaneously, it is used to obtain the throughput performance data of the sample model under different structural parameters and configuration parameters;

[0134] The training module is used to train a regression model using the throughput performance data as the regression target and the first structural feature and the first interaction feature to obtain an inference deployment configuration regression model; the inference deployment configuration regression model is used to predict the throughput performance data of a large language model under a given configuration.

[0135] Example 5

[0136] like Figure 5 As shown in this embodiment, a large language model inference deployment configuration recommendation device includes:

[0137] The acquisition module is used to acquire the second structural features and second interaction features of the target model; the target model is a large language model to be deployed; the structural features include the structural parameters of the target model and several sets of configuration parameters, and the second interaction features include the runtime resource quantification parameters of the target model under different configuration parameters;

[0138] The prediction module is used to input the second structural feature and the second interaction feature into the trained inference deployment configuration regression model to obtain the predicted throughput performance data of the target model under different combinations of configuration parameters; the inference deployment configuration regression model is trained according to the training method of the inference deployment configuration performance prediction model of the large language model.

[0139] The recommendation module is used to select the configuration parameters corresponding to the highest predicted throughput performance data and recommend them to the target model.

[0140] Example 6

[0141] The electronic device described in this embodiment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the training method for the performance prediction model of the inference deployment configuration of the large language model described in Embodiment 1, or the inference deployment configuration recommendation method of the large language model described in Embodiment 2, or the inference deployment method of the large language model described in Embodiment 3.

[0142] Example 7

[0143] The computer-readable storage medium described in this embodiment stores a computer program. When the computer program is executed by a processor, it implements the training method for the performance prediction model of the inference deployment configuration of the large language model described in Embodiment 1, or the recommendation method for the inference deployment configuration of the large language model described in Embodiment 2, or the inference deployment method of the large language model described in Embodiment 3.

[0144] The computer-readable storage medium may include RAM, ROM, EEPROM, CDROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory or any other media that can be used to store program code in the form of instructions or data structures and is accessible by a computer.

[0145] The processor is used to execute a computer program stored in memory to implement the various steps in the methods described in the above embodiments.

[0146] Example 8

[0147] Based on the same inventive concept, the computer program product of the present invention includes a computer program that, when executed by a processor, implements, according to the computer program being loaded into the processor, the training method for the performance prediction model of the inference deployment configuration of the large language model described in Embodiment 1, or the recommendation method for the inference deployment configuration of the large language model described in Embodiment 2, or the inference deployment method of the large language model described in Embodiment 3.

Claims

1. A method for training an inference deployment configuration performance prediction model of a large language model, characterized in that, The method includes: Obtain the first structural feature and the first interaction feature of the sample model; the sample model includes several different large language models, the first structural feature includes the structural parameters and configuration parameters of the large language model, and the first interaction feature includes the quantification parameters of the running resources of the large language model under different structural parameters and configuration parameters. Obtain the throughput performance data of the sample model under different structural parameters and configuration parameters; Using the throughput performance data as the regression target, a regression model is trained using the first structural feature and the first interaction feature to obtain the inference deployment configuration regression model; the inference deployment configuration regression model is used to predict the throughput performance data of a large language model under a given configuration. 2.The training method of a large language model inference deployment configuration performance prediction model according to claim 1, wherein, The structural parameters of the sample model include numerical features of the model structure, input length, and output length; the configuration parameters of the sample model include parallel configuration parameters and concurrent configuration parameters. 3.The training method of a large language model inference deployment configuration performance prediction model according to claim 1, wherein, The throughput performance data includes throughput, first token latency, and generation latency. A multi-objective regression model is trained using the throughput performance data as the target.

4. The training method for the inference deployment configuration performance prediction model of the large language model according to claim 1, characterized in that, The first interactive features include: single GPU memory usage estimation features, computation-to-communication ratio features, and pipeline bubble rate features.

5. The training method for the inference deployment configuration performance prediction model of the large language model according to claim 1, characterized in that, When the structural parameters of the obtained sample model are missing or insufficient, feature completion is performed using preset derivation rules.

6. The training method for the inference deployment configuration performance prediction model of the large language model according to claim 5, characterized in that, The feature completion using preset derivation rules includes: Based on the name and / or first structural features of the obtained sample model, the sample model is classified into a preset model type, and the structural parameters of the sample model are completed using the architecture design parameters of the preset model type as prior knowledge.

7. The training method for the inference deployment configuration performance prediction model of a large language model according to claim 6, characterized in that, Completing the structural parameters of the sample model includes: Calculate and complete the attention head dimension of the sample model based on the known hidden layer dimension and the known number of attention heads; And / or, calculate and complete the attention grouping query ratio of the sample model based on the known number of attention heads; And / or, determine the known feedforward network amplification ratio according to the preset model type corresponding to the sample model, and calculate and complete the feedforward network intermediate layer dimension of the sample model according to the known feedforward network amplification ratio; And / or, calculate and complete the total number of parameters of the sample model based on the known hidden layer dimension and the known total number of layers.

8. The training method for the inference deployment configuration performance prediction model of the large language model according to claim 2, characterized in that, Before training the regression model using the throughput performance data as the regression target and the first structural feature and the first interaction feature, the method further includes normalizing the first structural feature and the first interaction feature of the sample model. Normalization methods include: The numerical features of the model structure are logarithmically transformed and then normalized to standard scores. The parallel configuration parameters are normalized according to their proportion relative to the total number of GPUs. The concurrency configuration parameters are normalized according to their ratio to the video memory capacity.

9. A method for recommending inference deployment configurations for large language models, characterized in that, The method includes: The second structural feature and the second interaction feature of the target model are obtained; the target model is a large language model to be deployed; the structural feature includes the structural parameters of the target model and several sets of configuration parameters, and the second interaction feature includes the runtime resource quantification parameters of the target model under different configuration parameters; The second structural feature and the second interaction feature are input into the trained inference deployment configuration regression model to obtain the predicted throughput performance data of the target model under different combinations of configuration parameters; the inference deployment configuration regression model is trained according to any one of claims 1-8. The configuration parameters corresponding to the highest predicted throughput performance data are recommended for the target model.

10. The method for recommending the inference deployment configuration of a large language model according to claim 9, characterized in that, Based on the structural parameters of the target model, a hardware-constrained configuration space pruning strategy is used to enumerate all feasible parallel and concurrent configurations, resulting in several sets of configuration parameters.

11. The method for recommending the inference deployment configuration of a large language model according to claim 10, characterized in that, The hardware-constrained configuration space pruning strategy includes memory constraint pruning, parallelism constraint pruning, and communication constraint pruning. The memory constraint pruning excludes configuration parameter combinations that exceed the GPU memory capacity. The parallelism constraint pruning is to exclude configuration parameter combinations that do not satisfy tensor parallelism constraints and / or pipeline parallelism constraints. The parallelism constraint is: the tensor parallelism can be divided equally by the total number of attention heads and the number of key value heads in the model; the pipeline parallelism constraint is: the pipeline parallelism can be divided equally by the total number of layers in the model. The communication constraint pruning prioritizes reducing configuration parameter combinations with high communication overhead; wherein, the communication overhead under different configuration parameters is evaluated based on the deployed GPU cluster hardware topology.

12. The method for recommending the inference deployment configuration of a large language model according to claim 9, characterized in that, The configuration parameters recommended to the target model for selecting the data with the highest predicted throughput performance include: Configuration parameters corresponding to prediction throughput performance data with prediction uncertainty not lower than their corresponding thresholds are selected as candidate configuration parameters. From the candidate configuration parameters, the configuration parameter corresponding to the highest prediction throughput performance data is selected and recommended to the target model.

13. The method for recommending the inference deployment configuration of a large language model according to claim 12, characterized in that, The threshold for prediction uncertainty is determined based on the standard deviation of the training error of the regression model configured for the inference deployment.

14. The method for recommending the inference deployment configuration of a large language model according to claim 10, characterized in that, The structural parameters of the target model include numerical features of the model structure, input length, and output length; the configuration parameters of the target model include parallel configuration parameters and concurrent configuration parameters.

15. The method for recommending the inference deployment configuration of a large language model according to claim 9, characterized in that, The second interactive features include: single GPU memory usage estimation feature, computation-to-communication ratio feature, and pipeline bubble rate feature.

16. The method for recommending the inference deployment configuration of a large language model according to claim 10, characterized in that, When the structural parameters of the target model are missing, feature completion is performed using preset derivation rules.

17. The method for recommending the inference deployment configuration of a large language model according to claim 16, characterized in that, The feature completion using preset derivation rules includes: Based on the name and / or first structural features of the target model, the target model is classified into a preset model type, and the structural parameters of the target model are completed using the architectural design parameters of the model of that type as prior knowledge.

18. The method for recommending the inference deployment configuration of a large language model according to claim 17, characterized in that, Completing the structural parameters of the target model includes: Calculate and complete the attention head dimension of the target model based on the known hidden layer dimension and the known number of attention heads; And / or, calculate and complete the attention grouping query ratio of the target model based on the known number of attention heads of the target model; And / or, determine the known feedforward network amplification ratio according to the preset model type corresponding to the target model, and calculate and complete the feedforward network intermediate layer dimension of the target model according to the known feedforward network amplification ratio; And / or, calculate and complete the total number of parameters of the target model based on the known hidden layer dimension and the known total number of layers.

19. The method for recommending the inference deployment configuration of a large language model according to claim 14, characterized in that, Before inputting the second structural feature and the second interaction feature into the trained inference deployment configuration regression model, the method further includes: normalizing the second structural feature and the second interaction feature of the target model; the normalization method includes: The numerical features of the model structure are logarithmically transformed and then normalized to standard scores. The parallel configuration parameters are normalized according to their proportion relative to the total number of GPUs. The concurrency configuration parameters are normalized according to their ratio to the video memory capacity.

20. A reasoning deployment method for a large language model, characterized in that, The method includes: deploying the large language model to be deployed according to the configuration parameters recommended in the method of any one of claims 9-19.

21. A training device for a performance prediction model of inference deployment configuration for a large language model, characterized in that, The device includes: The acquisition module is used to acquire the first structural feature and the first interaction feature of the sample model; the sample model includes several different large language models, the first structural feature includes the structural parameters and configuration parameters of the large language model, and the first interaction feature includes the quantification parameters of the running resources of the large language model under different structural parameters and configuration parameters. Simultaneously, it is used to obtain the throughput performance data of the sample model under different structural parameters and configuration parameters; The training module is used to train a regression model using the throughput performance data as the regression target and the first structural feature and the first interaction feature to obtain an inference deployment configuration regression model; the inference deployment configuration regression model is used to predict the throughput performance data of a large language model under a given configuration.

22. A reasoning deployment configuration recommendation device for a large language model, characterized in that, The device includes: The acquisition module is used to acquire the second structural features and second interaction features of the target model; the target model is a large language model to be deployed; the structural features include the structural parameters of the target model and several sets of configuration parameters, and the second interaction features include the runtime resource quantification parameters of the target model under different configuration parameters; The prediction module is used to input the second structural feature and the second interaction feature into the trained inference deployment configuration regression model to obtain the predicted throughput performance data of the target model under different combinations of configuration parameters; the inference deployment configuration regression model is trained according to the method of any one of claims 1-8; The recommendation module is used to select the configuration parameters corresponding to the highest predicted throughput performance data and recommend them to the target model.

23. A reasoning deployment device for a large language model, characterized in that, The device includes: The deployment module is used to deploy the large language model to be deployed according to the configuration parameters recommended in the method of any one of claims 9-19.

24. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded onto the processor, it implements the training method for the performance prediction model of the inference deployment configuration of the large language model as described in any one of claims 1-8, or the recommendation method for the inference deployment configuration of the large language model as described in any one of claims 9-19, or the inference deployment method of the large language model as described in claim 20.

25. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the inference deployment configuration performance prediction model of the large language model according to any one of claims 1-8, or the inference deployment configuration recommendation method of the large language model according to any one of claims 9-19, or the inference deployment method of the large language model according to claim 20.

26. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the inference deployment configuration performance prediction model of the large language model according to any one of claims 1-8, or the inference deployment configuration recommendation method of the large language model according to any one of claims 9-19, or the inference deployment method of the large language model according to claim 20.

Images