A Machine Learning-Based Parallelized Allocation Decision-Making Method and System for Dense LLM Inference

Abstract

This invention discloses a machine learning-based method and system for configuring parallelization of dense LLM inference. The method receives a query request containing the input and output lengths of a dense large language model to be configured. It enumerates all feasible parallelization strategies for the target model, inputs these strategies into a trained hybrid prediction model, and predicts the throughput of each parallelization strategy under the constraints of the query request. The parallelization strategies include tensor parallelism, pipeline parallelism, and the number of GPUs. The optimal parallelization strategy is selected as the parallelization strategy for the target model's inference. This invention learns performance patterns from real system operation data, avoids complex and imprecise modeling, and supports intelligent decision-making under changing resource constraints, outputting the optimal parallelization strategy for dense large language model inference.

Description

Technical Field

[0001] This invention relates to the fields of distributed computing and machine learning, and in particular to a dense LLM inference parallelization configuration decision method and system based on machine learning. Background Technology

[0002] With the explosive growth in the number of parameters and computational requirements of large language models (LLMs), their inference services have become a core application consuming huge amounts of computing resources. In order to meet the requirements of low latency and high throughput inference, distributed parallel technology, especially tensor parallelism (TP) and pipeline parallelism (PP), has become the standard paradigm for deploying LLM inference services. However, in actual production environments, deciding on a parallelization strategy for a specific LLM model is a very challenging problem. Currently, there are two main approaches in industry and academia: (1) Simulation-based search systems: These systems construct sophisticated cost models and perform enumeration or heuristic search in the policy space. However, their cost models are often based on simplified assumptions and are difficult to adapt to the memory management and scheduling strategies specific to different inference engines, resulting in biased decision results in real systems. At the same time, their online search overhead may be large. (2) Rule-based expert experience or grid search: Engineers determine a set of "relatively good" fixed configurations based on experience or through small-scale testing. This approach cannot adapt to different workloads, and in order to find the global optimal solution, a large amount of time-consuming actual benchmark testing is required, resulting in high trial-and-error costs and a lack of scalability. Summary of the Invention

[0003] Purpose of the invention: The purpose of this invention is to provide a machine learning-based method and system for parallel configuration decision-making in dense LLM inference, which can accurately, efficiently, and automatically recommend the optimal parallel strategy and hardware resource configuration for dense large language model inference services.

[0004] Technical solution: The present invention provides a machine learning-based parallelized configuration decision-making method for dense LLM inference, comprising the following steps:

[0005] Receive a user's query request, which includes the input length and output length of the target model; the target model is a dense large language model to be configured.

[0006] Enumerate all feasible parallelization strategies for the target model, input the parallelization strategies into the trained hybrid prediction model, and predict the throughput of each parallelization strategy under the query request constraint; the parallelization strategies include tensor parallelism, pipeline parallelism, and the number of GPUs.

[0007] Based on the predicted throughput, the optimal parallelization strategy is selected as the parallelization strategy for the target model inference.

[0008] Furthermore, the hybrid prediction model includes a first sub-model and a second sub-model;

[0009] The first sub-model is a regression model, used to obtain the first predicted throughput according to the parallelization strategy;

[0010] The second sub-model is a decision tree model, which is used to predict the residual of the first predicted throughput to obtain the first predicted residual;

[0011] The predicted throughput is obtained by adding the first predicted throughput to the first predicted residual.

[0012] Furthermore, the training method for the hybrid prediction model includes:

[0013] Obtain several configuration combinations of the sample model and the actual throughput of each configuration combination. The configuration combinations include input length, output length, tensor parallelism, pipeline parallelism, and number of GPUs.

[0014] The sample model includes several dense large language models;

[0015] The feature multinomial constructed according to the configuration combination is input into the first sub-model to train the first sub-model, and the first sub-model outputs the second prediction throughput.

[0016] The feature polynomial constructed according to the configuration combination is input into the second sub-model. The second sub-model is trained with the difference between the first predicted throughput and the actual operating throughput as the objective. The second sub-model outputs the second prediction residual. The second predicted throughput and the second prediction residual are added together to obtain the predicted throughput result.

[0017] Furthermore, methods for obtaining actual operational throughput include:

[0018] The sample model was obtained through several configuration combinations, and the actual throughput of each configuration combination was recorded.

[0019] The sample model is divided into model fragments based on the tensor parallelism and pipeline parallelism in the configuration combination;

[0020] Configure the GPUs to load the model fragments based on the number of GPUs in the configuration combination;

[0021] Set a random ID sequence based on the input and output lengths in the configuration combination and perform autoregressive generation, recording the total number of tokens and the total time for token computation.

[0022] The actual throughput is calculated based on the total number of tokens and the total time for token computation.

[0023] Furthermore, the first sub-model is a Ridge regression model, and the second sub-model is a gradient boosting decision tree model.

[0024] Furthermore, the query request also includes the maximum number of GPUs, and the feasible parallelization strategy satisfies that the product of tensor parallelism and pipeline parallelism is not greater than the maximum number of GPUs, and both tensor parallelism and pipeline parallelism are powers of 2.

[0025] Furthermore, the query request also includes a minimum throughput;

[0026] When the query request includes a minimum throughput requirement, the parallelization strategy with the smallest number of GPUs is selected from the throughput prediction results that meet the minimum throughput requirement as the optimal parallelization strategy.

[0027] Otherwise, the parallelization strategy corresponding to the maximum predicted throughput is selected as the optimal parallelization strategy.

[0028] Based on the same inventive concept, the decision system of the dense large language model inference parallelization strategy described in this invention includes:

[0029] The request receiving unit is used to receive a user's query request, which includes the input length and output length of the target model; the target model is a dense large language model to be configured.

[0030] The parallelization strategy decision unit is used to enumerate all feasible parallelization strategies for the target model, input the parallelization strategies into the trained hybrid prediction model, and predict the throughput of each parallelization strategy under the query request constraint. The parallelization strategy includes tensor parallelism, pipeline parallelism, and the number of GPUs. Based on the throughput prediction results, the optimal parallelization strategy is selected as the parallelization strategy for the target model inference.

[0031] The electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the machine learning-based dense LLM inference parallelization configuration decision method.

[0032] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the machine learning-based dense LLM inference parallelization configuration decision method.

[0033] Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows:

[0034] (1) This invention incorporates the number of GPUs as an optimizable decision variable into a unified framework. It can work flexibly in different scenarios such as “optimal performance given resources” or “minimum resources given performance” depending on whether the user provides resource constraints. It outputs the optimal package containing parallel strategies and hardware scale at one time, solving the core planning problem in actual deployment and has significant practical value.

[0035] (2) The present invention adopts a two-stage hybrid model of regression prediction and residual correction. This model has the ability to capture global trends and fit local details, and can accurately model the complex nonlinear relationship between inference throughput and multidimensional configuration parameters.

[0036] (3) The training data of the present invention comes from the actual running results of the real inference engine under saturated batch, and the prediction model directly learns the data pattern. Therefore, the decision result of the present invention can fully consider the actual factors such as memory management and scheduling strategy of a specific engine, avoid the difference between the pure simulation model and the real system, and the optimal configuration obtained by the decision is closer to the actual performance peak of the production environment.

[0037] (4) The decision-making method of the present invention is fully automated, requiring no human intervention from data collection and model training to online decision-making. The online decision-making stage only requires milliseconds of model prediction time to evaluate a large number of candidate configurations, replacing the traditional actual testing or complex simulation that requires hours or even days, greatly improving decision-making efficiency and reducing the optimization threshold and cost. Attached Figure Description

[0038] Figure 1 This is a flowchart of a machine learning-based dense LLM inference parallelization configuration decision method according to an embodiment of the present invention.

[0039] Figure 2 This is a flowchart illustrating the process of collecting performance data and building a dataset according to an embodiment of the present invention.

[0040] Figure 3 This is a schematic diagram illustrating the principle of the hybrid prediction model in an embodiment of the present invention. Detailed Implementation

[0041] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0042] like Figure 1 As shown, the dense LLM inference parallelization configuration decision method based on machine learning includes the following steps:

[0043] Receive a user's query request, which includes the input length and output length of the target model; the target model is a dense large language model to be configured.

[0044] Enumerate all feasible parallelization strategies for the target model, input the parallelization strategies into the trained hybrid prediction model, and predict the throughput of each parallelization strategy under the query request constraint; the parallelization strategies include tensor parallelism, pipeline parallelism, and the number of GPUs.

[0045] Based on the predicted throughput, the optimal parallelization strategy is selected as the parallelization strategy for the target model inference.

[0046] Specifically, the method described in this embodiment is divided into an offline training phase and an online decision-making phase. These will be described separately below.

[0047] (a) Offline training phase

[0048] The offline training phase includes the following steps.

[0049] S1, Performance Data Acquisition.

[0050] Specifically, the parallel strategy (TP, PP) and the number of GPUs (N) gpu These factors together constitute a discrete, high-dimensional decision space. For a given total number of GPUs, there are multiple feasible (TP, PP) decomposition methods (e.g., 8 GPUs can be configured as TP=8 / PP=1, TP=4 / PP=2, TP=2 / PP=4, TP=1 / PP=8). Different workload characteristics (such as input sequence length L) also play a role. in and output sequence length L out This significantly affects the ratio of computational to communication overhead, causing the optimal configuration to change dynamically with the workload. Therefore, this embodiment automatically collects training data, including:

[0051] Several sample LLM models, such as LLaMA2-7B, Qwen3-8B, etc.;

[0052] Hardware configuration space: Number of GPUs The following conditions must be met: tensor parallelism (1, 2, 4, 8, etc.), tensor parallelism TP (1, 2, 4, 8, etc.), and pipeline parallelism PP (1, 2, 4, 8, etc.). ;

[0053] Workload space: length of input sequence (128, 512, 1024, 2048, etc.), output sequence length (128, 512, 1024, 2048, etc.).

[0054] This embodiment applies to each configuration combination (model, TP, PP) for each LLM model. , , A single inference task is launched on a real GPU cluster. Using this configuration, the LLM model is split and deployed, and standard forward propagation computation is performed. Text generation quality is ignored; only computational throughput is measured. Token throughput (tokens / second) is calculated by measuring the total computation time and the total number of generated tokens. All measurement results are structured and stored as a dataset, with each record containing input features (TP, PP, ...). , , (Model identifier) ​​and target throughput.

[0055] like Figure 2 As shown, collecting performance data and building a dataset specifically includes the following steps.

[0056] Step S101: Initialize the configuration list. Based on the input model, hardware configuration space, and workload space, generate the Cartesian product of all configuration combinations to be tested.

[0057] Step S102: Process each configuration in a loop. For each configuration in the list... :

[0058] (1) Prepare the environment. According to The target LLM model is split using the TP and PP in the model parallel framework (such as vLLM) script.

[0059] (2) Deployment and loading. In the specified... The inference process is started on each GPU, and the pre-splittered model fragments are loaded.

[0060] (3) Perform reasoning and measurement. Generate and The length corresponds to a random input ID sequence, and execution is performed. The autoregressive generation of steps is performed. A high-precision timer is used to record the wall clock time from the start of the first word calculation to the end of the last word calculation. .

[0061] (4) Calculate throughput. Here That is, the total number of generated lexical units.

[0062] (5) Save the record. Save {configuration parameters, throughput} to the dataset JSON file.

[0063] Step S103: Dataset Preparation. After all configuration tests are completed, the dataset is standardized and divided into training, validation, and test sets.

[0064] S2, Hybrid Prediction Model Construction and Training.

[0065] The performance of LLM inference is influenced by a complex nonlinear relationship among multiple factors, including model structure, hardware specifications (GPU computing power, memory bandwidth, interconnect topology), parallelization strategy, workload characteristics, and inference engine scheduling mechanism. Traditional analytical models based on theoretical computational complexity and communication costs struggle to accurately characterize the interactions between these factors. Therefore, this embodiment trains a two-stage hybrid prediction model based on the dataset obtained in S1, such as... Figure 3 As shown, the hybrid prediction model includes a first sub-model and a second sub-model.

[0066] The first sub-model uses a regression model to capture trends. In this embodiment, Ridge regression (L2 regularized linear regression) is used. This regression model is a high-order polynomial expansion of the input features (e.g., second- or third-order interaction terms). In this embodiment, the input of the first sub-model is based on the feature vector. Constructed polynomial characteristics This model is designed to capture the global, smooth trend of throughput changes with configuration. The first output of the model is the predicted throughput value.

[0067] The second sub-model uses a decision tree model to correct the residuals. In this embodiment, a gradient boosting decision tree model (GBDT), such as XGBoost or LightGBM, is used. The input to this decision tree model is multinomial features. The training objective is not throughput, but the prediction residual of the first sub-model on the training set. Prediction residual = actual throughput - prediction value of the first sub-model.

[0068] The first sub-model outputs the residual prediction value, and the residual prediction value is added to the prediction value of the first sub-model to obtain the final throughput prediction value.

[0069] Specifically, since ridge regression can capture trends but is not accurate enough in fitting the data, while GBDT fits the data accurately but cannot capture trends, this embodiment combines the two, which can both fit the data well and perceive the overall trend of data change.

[0070] Reference Figure 3 The training process of the hybrid prediction model includes the following steps.

[0071] Suppose the training set has m samples and the feature matrix is... Target vector .

[0072] Training the first sub-model:

[0073] S201, using A polynomial expansion of X yields a new characteristic matrix. .

[0074] S202, for Standardize (subtract the mean and divide by the standard deviation).

[0075] S203, Training the Ridge Regression Model: .

[0076] S204, Calculate the predicted values ​​on the training set. and residual .

[0077] Training the second sub-model:

[0078] S205, Input Feature Matrix ,by To train the target GBDT model (e.g.) ): .

[0079] Model combination:

[0080] S206, the final prediction function is:

[0081] .

[0082] S207 adjusts the alpha parameter of Ridge and hyperparameters such as tree depth and learning rate of GBDT on the validation set to minimize the mean squared error of prediction.

[0083] (ii) Online decision-making stage

[0084] This embodiment is not limited to a fixed number of GPUs when making decisions, but rather considers a maximum number of GPUs. The system dynamically optimizes the number of GPUs under constraints to find the optimal balance between performance and cost. The online decision-making phase includes the following steps.

[0085] S3 receives user query requests, in the format: target model, user. ,user Maximum number of available GPUs Optionally, minimum throughput requirements may also be included. .

[0086] S4, enumerate all possible configurations ,in And TP and PP are powers of 2.

[0087] S5, for each feasible configuration, invoke the trained hybrid prediction model to predict the target model in... and The throughput.

[0088] S6, Filter and sort according to the optimization objective:

[0089] If the user specifies Then return all that satisfy the condition. In the configuration Minimal configuration (optimal cost).

[0090] Otherwise, return to Within the constraints, the configuration with the highest predicted throughput (optimal performance).

[0091] S7, Output decision result: optimal And its predicted throughput.

[0092] The method described in this invention will be verified through a specific experiment below.

[0093] In this experiment, we assume the user's query request is: model=LLaMA-7B. , , .

[0094] Enumerate all possible configurations for LLaMA2-7B. The total video memory size must be large enough to accommodate the entire model. In this experiment, a feasible configuration is assumed to be:

[0095] ,

[0096] ,

[0097] ,

[0098] ,

[0099] ,

[0100] ,

[0101] ,

[0102] ,

[0103] ,

[0104] .

[0105] For each configuration, the hybrid model is invoked to predict throughput. Assume the prediction result is:

[0106] (1,1,1): 525 tokens / s;

[0107] (2,1,2): 1361 tokens / s;

[0108] (1,2,2): 1266 tokens / s;

[0109] (4,1,4): 3131 tokens / s;

[0110] (1,4,4): 2688 tokens / s;

[0111] (2,2,4): 2849 tokens / s;

[0112] (8,1,8): 1361 tokens / s;

[0113] (1,8,8): 3467 tokens / s;

[0114] (4,2,8): 2401 tokens / s;

[0115] (2,4,8): 2663 tokens / s.

[0116] Specifically, for the case configured as (1,1,1), the prediction results are explained in detail: number of cards = 1, parallel type _TP = 0, parallel type _PP = 0, parallel type _single card = 1; the predicted value of the first sub-model output is 399720904, the corrected value of the second sub-model output is -399720379, and the final throughput prediction value is 525.

[0117] The dense LLM inference parallelization configuration decision system based on machine learning described in this invention includes:

[0118] The request receiving unit is used to receive a user's query request, which includes the input length and output length of the target model; the target model is a dense large language model to be configured.

[0119] The parallelization strategy decision unit is used to enumerate all feasible parallelization strategies for the target model, input the parallelization strategies into the trained hybrid prediction model, and predict the throughput of each parallelization strategy under the query request constraint. The parallelization strategy includes tensor parallelism, pipeline parallelism, and the number of GPUs. Based on the throughput prediction results, the optimal parallelization strategy is selected as the parallelization strategy for the target model inference.

[0120] The electronic device of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is loaded onto the processor, it implements the machine learning-based dense LLM inference parallelization configuration decision method.

[0121] The computer program product of the present invention includes a computer program that, when executed by a processor, implements the machine learning-based dense LLM inference parallelization configuration decision method.

[0122] The computer-readable storage medium may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store program code in the form of instructions or data structures and is accessible by a computer.

[0123] The processor is used to execute a computer program stored in memory to implement the various steps in the methods described in the above embodiments.

Claims

1. A method for machine learning-based dense LLM inference parallelization configuration decision, characterized in that, Includes the following steps: Receive a user's query request, which includes the input length and output length of the target model; the target model is a dense large language model to be configured. Enumerate all feasible parallelization strategies for the target model, input the parallelization strategies into the trained hybrid prediction model, and predict the throughput of each parallelization strategy under the query request constraint; the parallelization strategies include tensor parallelism, pipeline parallelism, and the number of GPUs. Based on the predicted throughput, the optimal parallelization strategy is selected as the parallelization strategy for the target model inference. The hybrid prediction model includes a first sub-model and a second sub-model; The first sub-model is a regression model, used to obtain the first predicted throughput according to the parallelization strategy; The second sub-model is a decision tree model, which is used to predict the residual of the first predicted throughput to obtain the first predicted residual; The predicted throughput is obtained by adding the first predicted throughput to the first predicted residual. The training method for the hybrid prediction model includes: Obtain several configuration combinations of the sample model and the actual throughput of each configuration combination. The configuration combinations include input length, output length, tensor parallelism, pipeline parallelism, and number of GPUs. The sample model includes several dense large language models; The feature multinomial constructed according to the configuration combination is input into the first sub-model to train the first sub-model, and the first sub-model outputs the second prediction throughput. The feature polynomial constructed according to the configuration combination is input into the second sub-model. The second sub-model is trained with the difference between the second predicted throughput and the actual operating throughput as the objective. The second sub-model outputs the second prediction residual. The second predicted throughput and the second prediction residual are added together to obtain the predicted throughput result.

2. The dense LLM inference parallelization configuration decision method based on machine learning according to claim 1, characterized in that, Methods for obtaining actual operational throughput include: The sample model was obtained through several configuration combinations, and the actual throughput of each configuration combination was recorded. The sample model is divided into model fragments based on the tensor parallelism and pipeline parallelism in the configuration combination; Configure the GPUs to load the model fragments based on the number of GPUs in the configuration combination; Set a random ID sequence based on the input and output lengths in the configuration combination and perform autoregressive generation, recording the total number of tokens and the total time for token computation. The actual throughput is calculated based on the total number of tokens and the total time for token computation.

3. The dense LLM inference parallelization configuration decision method based on machine learning according to claim 1, characterized in that, The first sub-model is a Ridge regression model, and the second sub-model is a gradient boosting decision tree model.

4. The dense LLM inference parallelization configuration decision method based on machine learning according to claim 1, characterized in that, The query request also includes the maximum number of GPUs. The feasible parallelization strategy satisfies that the product of tensor parallelism and pipeline parallelism is not greater than the maximum number of GPUs, and both tensor parallelism and pipeline parallelism are powers of 2.

5. The machine learning-based dense LLM inference parallelization configuration decision method according to claim 1 or 4, characterized in that, The query request also includes a minimum throughput; When the query request includes a minimum throughput requirement, the parallelization strategy with the smallest number of GPUs is selected from the throughput prediction results that meet the minimum throughput requirement as the optimal parallelization strategy. Otherwise, the parallelization strategy corresponding to the maximum predicted throughput is selected as the optimal parallelization strategy.

6. A dense LLM inference parallelization configuration decision system based on machine learning, using the method of claim 1, characterized in that, include: A request receiving unit is used to receive a user's query request, the query request including the input length and output length of the target model; The target model is a dense large language model to be configured; The parallelization strategy decision unit is used to enumerate all feasible parallelization strategies for the target model, input the parallelization strategies into the trained hybrid prediction model, and predict the throughput of each parallelization strategy under the query request constraint. The parallelization strategy includes tensor parallelism, pipeline parallelism, and the number of GPUs. Based on the throughput prediction results, the optimal parallelization strategy is selected as the parallelization strategy for the target model inference.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the dense LLM inference parallelization configuration decision method based on machine learning as described in any one of claims 1-5.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the dense LLM inference parallelization configuration decision method based on machine learning according to any one of claims 1-5.

Images