Large language model performance prediction system, device and equipment based on data driving

Abstract

The invention relates to a big language model performance prediction system, device and equipment based on data driving, and the method comprises the steps: generating a sample data set for training a performance prediction model according to the existing data in the system, obtaining a unified numerical input feature vector through converting all samples in the sample data set, and obtaining a prediction result. And generating a trained prediction model, converting the input parameters of the scene to be evaluated, obtaining the feature vectors, inputting the feature vectors into the prediction model, and generating a performance evaluation result. According to the method, a high-coverage training set can be constructed through multi-source data fusion and an automatic sample expansion capability, accurate mapping from configuration to performance is realized through unified feature conversion and a tree model nonlinear mining capability, and prediction reliability can be ensured through a feature processing consistency guarantee mechanism; and generating confidence early warning through the data space coverage analysis capability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model, in particular to a data-driven large language model performance prediction system, device and equipment. BACKGROUND

[0002] The field of efficient large language models is rapidly developing, but there is still no reliable framework to analyze the deployment performance of various models on different hardware. Currently, models with hundreds of billions of parameters are becoming more and more common, and even the most advanced hardware devices have bottlenecks in practical applications.

[0003] In actual large language model deployment, it is necessary to evaluate the execution performance of a model on target hardware. The traditional evaluation process relies on model actual measurement, which requires actual loading of the model, preparation of input, and collection of log data, and is time-consuming and resource-intensive. In a platform-heterogeneous, model-diverse, and flexible scheduling environment, repeated actual measurement is costly. Especially for small and medium-sized companies with relatively insufficient computing resources, this further limits their technical exploration and investment in large language models.

[0004] Although there are tools (such as optimum-benchmark, DLProf of NVIDIA, or common FLOPs static analyzers) that attempt to estimate performance based on model computation graphs or theoretical computation, such methods often ignore multiple key factors in actual applications, resulting in limited prediction accuracy. For example, when INT8 quantization is enabled in large models such as LLaMA-2-7B, the actual throughput can be improved by about 30%-40%, but existing tools still estimate based on the BF16 path, ignoring the kernel execution efficiency differences of quantization operators; in addition, the prefill and decode stages differ significantly in operator structure and compute-to-bandwidth ratio, with the former dominated by large matrix multiplication and the latter bottlenecked by memory bandwidth, and FLOPs values cannot reflect the actual execution delay in some scenarios. Some existing prediction methods focus too much on computing power or only perform linear fitting on hardware parameters, and often lack adaptability when facing changes in actual deployment environments.

[0005] Therefore, there is an urgent need for a performance prediction mechanism that takes hardware parameters and running configurations as input to achieve fast and accurate performance evaluation without deployment, supporting deployment strategy formulation and resource scheduling optimization. SUMMARY

[0006] The present application provides a data-driven large language model performance prediction system, characterized in that it comprises: According to the existing data in the system, a sample data set for training the performance prediction model is generated; By transforming all samples in the sample data set, a unified numerical input feature vector is obtained, and a trained prediction model is generated; By transforming the input parameters of the scene to be evaluated, a feature vector is obtained and input into the prediction model, and a performance evaluation result is generated.

[0007] Optionally, the sample data set for training the performance prediction model is generated according to existing data in the system, comprising: The existing data of the system comes from the results generated by the simulator, the historical running logs of the cloud platform, and the measured data collected by the user in actual deployment; Obtain the model information field and performance indicators in the existing data to form samples and constitute a sample data set; The formed samples are automatically expanded to obtain configuration combinations that do not exist.

[0008] Optionally, the uniform numerical input feature vector is obtained by transforming all samples in the sample data set, and the trained prediction model is generated, comprising: Obtain the original data features of each sample in the sample data set and transform them into numerical feature vectors that support model processing; The numerical feature vector is used to train a decision tree model to generate a prediction model.

[0009] Optionally, the performance evaluation result is generated by transforming the input parameters of the scene to be evaluated, obtaining a feature vector, and inputting the prediction model, comprising: Obtain the data features of the un-evaluated scene provided by the user and transform them into normalized numerical feature vectors consistent with the training; The normalized numerical feature vector is predicted by the generated prediction model to generate an output core indicator and a confidence evaluation result.

[0010] Optionally, the formed samples are automatically expanded to obtain configuration combinations that do not exist, comprising: The system supports expanding the sample data in the sample data set; The expansion method is to change the values of part of the parameters of the fixed sample to quickly generate configuration combinations that are not covered; The configuration combinations that are not covered are preliminarily performance estimated to generate synthetic sample data with a predicted performance label.

[0011] Optionally, the numerical feature vector is used to train a decision tree model to generate a prediction model, comprising: The decision tree model used is a decision tree prediction model or a gradient boosting decision tree model; Through a decision tree prediction model or a gradient boosting decision tree model, a structure is recursively constructed using a split node to mine the nonlinear relationship between the numerical feature vector and the performance. After the training is completed, the normalization parameters and the input feature templates used during the training are saved to form a serialized file.

[0012] Optionally, the normalized numerical feature vector is predicted through the generated prediction model to generate an output core indicator and a confidence evaluation result, including: The confidence evaluation result generated by the prediction model is used to evaluate the coverage relationship between the input configuration and the training data space, and a confidence label is generated through distribution state analysis; When the input configuration falls outside the training sample space, the result is marked as low confidence, and a prompt is issued to the user.

[0013] The application also provides a data-driven large language model performance prediction device, characterized in that the device comprises: A data set construction expansion module is configured to generate a sample data set for training a performance prediction model according to existing data in the system; A model training module is configured to obtain a unified numerical input feature vector by converting all samples in the sample data set, and generate a trained prediction model; A performance prediction module is configured to obtain a feature vector by converting input parameters of a scene to be evaluated and input the prediction model to generate a performance evaluation result.

[0014] Optionally, the model training module comprises: A feature conversion unit is configured to convert the sample data set into a numerical feature vector that supports model processing; A model training unit is configured to train the numerical feature vector through a decision tree model to generate a prediction model.

[0015] The application also provides an electronic device, characterized by being used to implement any of the data-driven large language model performance prediction systems, comprising: A processor is configured to perform all computationally intensive tasks and implement the data-driven large language model performance prediction system; A memory is configured to store normalization parameters, input feature templates, and prediction models for the processor to call during prediction; A calling interface is configured to support local and remote access and implement the general deployment of the prediction function.

[0016] The application has the advantages of multi-source heterogeneous data fusion and automatic sample expansion, can solve the problem of data scarcity, construct a high-coverage training set, realize accurate mapping of complex configuration to performance indicators through unified feature vector conversion and nonlinear mining of tree models, ensure the reuse of normalization parameters and feature templates in the training phase during online prediction through a training and prediction feature processing consistency guarantee mechanism, and guarantee the reliability of deployment, and establish a prediction result credibility evaluation system through coverage analysis of data space. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description are briefly introduced as follows. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0018] Figure 1 A flowchart of an embodiment of a data-driven large language model performance prediction system disclosed by the present application is shown; Figure 2 A structural block diagram of an embodiment of a data-driven large language model performance prediction device disclosed by the present application is shown. DETAILED DESCRIPTION

[0019] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference signs in the drawings represent functionally identical or similar elements. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0020] Among them, the terms "first", "second" are only for description purpose, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0021] The word "exemplary" is used herein in the sense of being an example, illustration, or demonstration. Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0022] In addition, for a better illustration of the present application, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present application can be practiced without certain specific details. In some instances, well-known methods, procedures, elements, and circuits have not been described in detail in order to emphasize the principles of the present application.

[0023] The present application proposes a data-driven large language model performance prediction system. The system can predict the training throughput or inference delay of a model on a target platform by inputting the basic parameters of the model (such as model size, batch size, data type, execution stage type, etc.) and the core indicators of the target hardware platform (such as TFLOPS, memory bandwidth, communication bandwidth, etc.) without actually running the target model. The system is based on machine learning methods such as decision trees to build a predictor, which has better generalization. Through this method, early performance evaluation and deployment scheme screening can be realized, and it is widely applicable to cloud service platforms, multi-model automatic scheduling systems, and other scenarios.

[0024] Embodiment 1 As shown in Figure 1 The flowchart of the data-driven large language model performance prediction system of an embodiment of the present application, specifically includes the following contents: S100, according to the existing data in the system, generate a sample data set for training the performance prediction model.

[0025] Specifically, the system obtains existing data such as simulation results, cloud platform historical running logs, and actual deployment collected measured data, extracts the model information field and performance indicators from the existing data to form samples, and through the aggregation of samples, the sample data set for training the model is formed. By fixing part of the parameters of the sample and modifying the remaining parameters, the coverage of the sample is expanded.

[0026] S200, by transforming all samples in the sample data set, obtain a unified numerical input feature vector, and generate a trained prediction model.

[0027] Specifically, the sample data set generated in S100 is normalized, which is the original sample data transformed into a numerical feature vector that can be processed by the model. Then, the numerical feature vector is input into the decision tree model for model training to generate the prediction model. After the training is completed, the prediction model is saved together with the normalization parameters and input feature templates as a serialized file.

[0028] S300, by transforming the input parameters of the scene to be evaluated, obtain a feature vector and input it into the prediction model to generate a performance evaluation result.

[0029] Specifically, the user-provided input parameters of the to-be-evaluated deployment scenario are converted into a numerical feature vector through an input feature template, and then input into the prediction model generated in S200 to perform prediction. If the input configuration falls within the training sample space, an output core indicator is generated. If the input configuration falls outside the training sample space, it is marked as low confidence and the user is prompted.

[0030] To sum up, the data-driven large language model performance prediction system proposed in the present application can first construct a training dataset covering a multi-dimensional configuration space of model parameter quantity, parallel strategy, and hardware platform through multi-source heterogeneous data fusion and automatic sample expansion capability. Secondly, through a unified numerical feature vector conversion interface and tree model nonlinear relationship mining capability, the original configuration parameters are converted into normalized feature vectors, and the complex mapping rules between features and performance are automatically captured through gradient boosting decision tree recursive splitting nodes. Then, through the training and prediction feature processing consistency guarantee mechanism, the normalized parameters and feature templates of the training phase are reused during online prediction to prevent prediction deviation caused by feature misplacement. Then, through data space coverage relationship analysis, the deviation degree of the input configuration from the training data distribution is identified, and the prediction results beyond the coverage range are marked with low confidence warning. Finally, the present application has a modular pipeline architecture, which can realize end-to-end automation services from data construction, model training to performance prediction on electronic devices, significantly reducing the deployment trial and error cost of large language models.

[0031] As an implementable manner, in step S100, sample data sets for training the performance prediction model are generated according to existing data in the system, including: S110, the existing data of the system comes from the simulator generated results, cloud platform historical running logs and the measured data collected by the user in actual deployment.

[0032] Specifically, the performance data under different configuration combinations generated by the simulator, the task logs actually running on the cloud service platform, and a small amount of data accurately measured and reported by the user in the actual deployment environment are used as the source of existing data. Information extraction is performed to form samples.

[0033] S120, the model information field and performance indicators in the existing data are obtained to form samples and constitute a sample data set.

[0034] Specifically, the existing data collected in S101 are extracted for field and performance indicators under corresponding configurations. The model name, model running configuration parameters, target hardware platform indicators, and actual measured performance indicators under the configuration are obtained. By summarizing these main parameter information, samples are formed, and the sample data set is generated by summarizing all samples of the existing data.

[0035] The model running configuration parameters are specifically model actual parameter quantity, model activation parameter quantity, batch size, parameter data type, execution stage type, expert parallel number, tensor parallel number, and data parallel number. The model actual parameter quantity, i.e., the total data quantity of the parameters to be trained, is, for example, 8B (billion) for Llama3.1-8B and 671B for DeepSeek-V3. The model activation parameter quantity is the actual parameter quantity for a dense model such as Llama3.1, while the activation parameter quantity of an MOE model is generally much smaller than the actual parameter quantity, for example, 37B for DeepSeek-V3. The parameter data type includes the following commonly used data types, i.e., bf16, fp16, fp8, int8, etc. The execution stage type mainly includes training, inference prefill, and inference decode stages. The computing network AllReduce bandwidth is the measured inter-node AllReduce communication bandwidth. The computing network AlltoAll bandwidth is the measured inter-node AllReduce communication bandwidth, and AlltoAll is mainly used for MOE inference and training.

[0036] The target hardware platform indicators are specifically theoretical peak TFLOPS, total accelerator memory, single-card accelerator memory, accelerator bandwidth, inter-card communication bandwidth, and computing network bandwidth. The theoretical peak TFLOPS is the theoretical peak computing power of all computing cards, which is mainly obtained from the data manual of Nvidia. The total accelerator memory is the total memory size of the accelerator participating in training / inference. The single-card accelerator memory is the memory size of a single card of the accelerator. The accelerator bandwidth is the memory access bandwidth of the accelerator. The inter-card communication bandwidth is the PCIE / NvLink communication bandwidth of the accelerator in the node.

[0037] The actually measured performance indicators are specifically average delay and token throughput. The average delay mainly refers to the average delay of completing inference. For large language model inference tasks, two delay indicators are generally used, i.e., TTFT (Time to first token) and TPOT (Time per output token). The TTFT refers to the time from receiving a request to outputting the first token, mainly measuring the time efficiency of inference prefill. The TPOT refers to the time between tokens, mainly measuring the time efficiency of inference decode. The token throughput mainly refers to the number of tokens that can be processed per unit time during training / inference.

[0038] S130, automatically expanding the formed sample to obtain a configuration combination that does not exist.

[0039] Specifically, the system supports automatic expansion of sample data, generates uncovered configuration combinations by keeping part of the variables unchanged and replacing other parameter combinations, and preliminarily estimates the performance of the configuration combinations by means of the simulator, so as to enhance the coverage of the training samples and the generalization ability of the model.

[0040] As an implementable manner, in step S130, the formed samples are automatically expanded to obtain configuration combinations that are not possessed, including: S131, the system supports expansion of sample data in the sample data set, and the expansion mode is to take values of part of parameters of a fixed sample, and to quickly generate uncovered configuration combinations by changing the remaining parameters.

[0041] Specifically, by selecting parameters that need to be kept unchanged in the sample, for example, fixing the hardware type as GPU_A100 and the model architecture as LLaMA. Then, the expandable dimensions are identified, and the full combination of the variable parameters is generated according to the preset step length to output the uncovered configuration set.

[0042] In addition, invalid configuration filtering needs to be performed, which can be verified by physical rules, such as multiplying the model parameter quantity and the byte number to verify whether the video memory capacity is sufficient, or filtered by business logic, such as excluding disabled combinations and skipping the performance invalid area, to realize output of the valid configuration set to be estimated.

[0043] S132, preliminary performance estimation is performed on the uncovered configuration combinations to generate synthetic sample data with predicted performance labels.

[0044] Specifically, for the uncovered configuration set generated in S131, corresponding simulators and formulas are used according to different application scenarios to output predicted performance indicators and mark the data source, forming synthetic sample data.

[0045] As an implementable manner, in step S200, by transforming all samples in the sample data set, a unified numerical input feature vector is obtained, and a trained prediction model is generated, including: S201, obtaining the original data features of each sample in the sample data set, and transforming them into numerical feature vectors supported by the model.

[0046] Specifically, the numerical features, categorical features and combined features in the sample original data of the generated sample data set are obtained, the numerical features are directly retained, the categorical features are converted into numbers, and the combined features are split into independent fields, so that the model can identify different parameter types. Then, the maximum and minimum values of each numerical feature in the training set are counted, and linear transformation is performed on each numerical feature column to realize normalization of the numerical values, and a unified numerical input feature vector is generated.

[0047] The features include model size, batch size, data type, execution stage type (represented by one-hot encoding as train, inference prefill, or inference decode), TP (Tensor Parallel: a way to run large-scale model training or inference on a GPU server. When the model weight exceeds the capacity of a single card, the matrix or weight to be operated is split and placed on different acceleration cards), EP (Expert Parallel: a way to run large-scale model training or inference on a GPU server. When the model weight exceeds the capacity of a single card, the experts of the MOE layer are distributed to different acceleration cards to realize model parallelism), and DP (Data Parallel: a way to run large-scale model training or inference on a GPU server. By splitting the data and sending it to different cards, the throughput value during training and inference is improved), GPU computing power, and communication bandwidth, and the like platform parameters.

[0048] In S202, a decision tree model is used to train a prediction model based on the numerical feature vector.

[0049] Specifically, the decision tree model used is a decision tree prediction model or a gradient boosting decision tree model. The decision tree prediction model or the gradient boosting decision tree model is used to recursively construct a structure using split nodes to mine the nonlinear relationship between the numerical feature vector and the performance. After training, the normalization parameters and input feature templates used during training are saved to form a serialized file.

[0050] The decision tree prediction model is selected because it has extremely fast training speed and strong model interpretability, and is suitable for small-scale data sets with less than ten thousand samples for rapid prototyping verification. The gradient boosting decision tree model is selected because it can improve prediction accuracy through Boosting integration and automatically process feature interactions, and can be used in production environments to solve the complex nonlinear mapping problem between large language model configuration parameters and performance.

[0051] The normalization parameters are statistical values of each numerical feature of the sample training set, which can be used as a basis for prediction consistency guarantee and confidence detection. The input feature template is metadata of feature processing including feature field order, category feature encoding rules, and combined feature splitting method, which can be used to ensure that the prediction input and the field order and encoding during training are completely consistent, and to automatically apply the template to convert new data. The serialized file is a binary persistent file formed by combining the normalization parameters, input feature templates, and prediction model, which can be used for cross-platform deployment.

[0052] As an implementable manner, in step S300, the performance evaluation result is generated by converting the input parameters of the scene to be evaluated, obtaining the feature vector and inputting the prediction model, including: S301, obtaining the data features of the un-evaluated scene provided by the user and converting them into normalized numerical feature vectors consistent with the training.

[0053] Specifically, according to the input parameters of the deployment scene to be evaluated provided by the user, including model running configuration and target hardware platform indicators, the user input is converted into a feature vector using an input feature template, and the same normalization parameter recorded in the training phase is applied to the numerical features for the same normalization processing, and the same encoding processing is performed on the category features (data type, execution phase) to generate a normalized numerical feature vector consistent with the training.

[0054] S302, predicting the normalized numerical feature vector through the generated prediction model to generate the output core indicators and confidence evaluation results.

[0055] Specifically, the normalized numerical feature vector generated in S301 is input into the trained regression model to perform prediction, and the key performance indicators under the scene are obtained, including the average delay and token throughput rate per unit time under the scene, and the coverage relationship between the input configuration and the training data space is evaluated to generate a confidence label through distribution state analysis. When the input configuration falls outside the training sample space, the result is marked as low confidence, and a prompt is sent to the user to perform supplementary measurement to optimize the prediction coverage.

[0056] Among them, the confidence evaluation is to calculate the distance (such as Euclidean distance, Mahalanobis distance) between the input feature vector and all training sample feature vectors, if the minimum distance exceeds the threshold or the nearest neighbor sample is too small, it is determined as an external sample, and it is determined as low confidence.

[0057] Embodiment 2 Based on the same principle as the foregoing method, a data-driven large language model performance prediction system is also proposed, see Figure 2 The data-driven large language model performance prediction device 100 of the embodiment of the present disclosure comprises: The data set construction expansion module 110 is configured to generate a sample data set for training the performance prediction model according to the existing data in the system; The model training module 120 obtains a unified numerical input feature vector by converting all samples in the sample data set, and generates a trained prediction model; The performance prediction module 130 generates a performance evaluation result by converting the input parameters of the scene to be evaluated, obtaining the feature vector and inputting the prediction model.

[0058] As an optional implementation of the present application, the model training module 120, optionally, comprises: a feature conversion unit 121, configured to convert the sample data set into a numerical feature vector supported by the model processing; a model training unit 122, configured to train the numerical feature vector through a decision tree type model to generate a prediction model.

[0059] Obviously, those skilled in the art should understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the above-mentioned embodiment of each control method. The above-mentioned modules or steps of the present application can be realized by a general computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be respectively manufactured into each integrated circuit module, or a plurality of modules or steps among them can be manufactured into a single integrated circuit module to realize. Thus, the present application is not limited to any specific hardware and software combination.

[0060] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the above-mentioned embodiment of each control method. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), etc. The storage medium can also include a combination of the above-mentioned types of memories.

[0061] Embodiment 3 Furthermore, the present application proposes an electronic device, characterized by being used to implement any of the data-driven large language model performance prediction systems, comprising: a processor, configured to perform all computationally intensive tasks and implement the data-driven large language model performance prediction system; a memory, configured to store normalization parameters, input feature templates and prediction models, for calling by the processor in prediction; a calling interface, configured to support local and remote access and realize the general deployment of the prediction function.

[0062] The electronic device of the embodiments of the present disclosure includes a processor and a memory for storing processor-executable instructions. Wherein the processor is configured to implement the method for implementing expert pruning of a large-scale hybrid expert model based on a small number of examples as described above when executing the executable instructions.

[0063] It should be noted that the number of processors can be one or more. Meanwhile, the electronic device of the embodiments of the present disclosure can also include an input device and an output device. Wherein the processor, the memory, the input device and the output device can be connected through a bus, or connected through other means, which is not limited here.

[0064] The memory as a computer readable storage medium of the performance prediction system of the large language model based on data driving can be used to store software programs, computer executable programs and various modules, such as programs or modules corresponding to the performance prediction system of the large language model based on data driving of the embodiments of the present disclosure. The processor executes the software programs or modules stored in the memory, thereby performing various functional applications and data processing of the electronic device.

[0065] The input device can be used to receive input numbers or signals. Wherein the signal can be a key signal generated in relation to the user settings and function control of the device / terminal / server. The output device can include a display device such as a display screen.

[0066] The calling interface supports the following functions: batch input of input parameters, suitable for parallel computing resource scheduling scenarios; submission of query requests based on Python SDK, the system supports synchronous and asynchronous prediction modes; provides a prediction request log recording and analysis module for prediction error tracking, model regression evaluation and version comparison; the system supports multiple version models coexistence and dynamic switching mechanism to ensure system stability during model updating.

[0067] The above has described the embodiments of the present application, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles, practical application or improvement of technology in the market of the embodiments, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A data-driven performance prediction system for large language models, characterized in that, include: Based on the existing data in the system, generate a sample dataset for training the performance prediction model; By transforming all samples in the sample dataset, a unified numerical input feature vector is obtained, and a trained prediction model is generated. By transforming the input parameters of the scenario to be evaluated, the feature vector is obtained and input into the prediction model to generate performance evaluation results.

2. The data-driven large language model performance prediction system as described in claim 1, characterized in that, The step of generating a sample dataset for training a performance prediction model based on existing data in the system includes: The system's existing data comes from simulator-generated results, cloud platform historical operation logs, and actual test data collected by users during deployment. Obtain model information fields and performance metrics from existing data, form samples, and construct a sample dataset; The generated samples are automatically expanded to obtain configuration combinations that are not yet available.

3. The data-driven large language model performance prediction system as described in claim 1, characterized in that, The step of transforming all samples in the sample dataset to obtain a unified numerical input feature vector and generating a trained prediction model includes: Obtain the original data features of each sample in the sample dataset and transform them into numerical feature vectors that support model processing; A predictive model is generated by training a decision tree model using numerical feature vectors.

4. The data-driven large language model performance prediction system as described in claim 1, characterized in that, The process of transforming the input parameters of the scenario to be evaluated, obtaining feature vectors, inputting them into the prediction model, and generating performance evaluation results includes: Obtain data features from unevaluated scenarios provided by the user and transform them into normalized numerical feature vectors consistent with the training. The normalized numerical feature vector is predicted using the generated prediction model, and the core output indicators and confidence assessment results are generated.

5. The data-driven large language model performance prediction system as described in claim 2, characterized in that, The automatic expansion of the formed samples to obtain unavailable configuration combinations includes: The system supports expanding the sample data in the sample dataset; The expansion method involves fixing some parameter values ​​of a sample and quickly generating uncovered configuration combinations by changing the remaining parameters. For uncovered configuration combinations, preliminary performance estimates are performed, generating synthetic sample data with predicted performance labels.

6. The data-driven large language model performance prediction system as described in claim 3, characterized in that, The step of training a prediction model using a decision tree model with numerical feature vectors includes: The decision tree model used is either a decision tree prediction model or a gradient boosting decision tree model; By using decision tree prediction models or gradient boosting decision tree models, and recursively constructing structures with split nodes, the nonlinear relationship between numerical feature vectors and performance can be explored. After training is complete, the normalized parameters and input feature templates used during training are saved to form a serialized file.

7. The data-driven large language model performance prediction system as described in claim 4, characterized in that, The process of predicting normalized numerical feature vectors using the generated prediction model to generate core output indicators and confidence assessment results includes: The confidence assessment results generated by the prediction model are used to generate confidence labels by evaluating the coverage relationship between the input configuration and the training data space through distribution state analysis. If the input configuration falls outside the training sample space, the result is marked as low confidence and a prompt is issued to the user.

8. A data-driven large language model performance prediction device, characterized in that, The device includes: The dataset building extension module is used to generate sample datasets for training performance prediction models based on existing data in the system. The model training module transforms all samples in the sample dataset to obtain a unified numerical input feature vector and generates a completed prediction model. The performance prediction module transforms the input parameters of the scenario to be evaluated, obtains feature vectors, inputs them into the prediction model, and generates performance evaluation results.

9. The data-driven large language model performance prediction device according to claim 8, wherein the model training module comprises: The feature transformation unit is used to transform the sample dataset into numerical feature vectors that support model processing. The model training unit is used to train numerical feature vectors through a decision tree model to generate a prediction model.

10. An electronic device, characterized in that, For implementing the data-driven large language model performance prediction system according to any one of claims 1 to 7, comprising: The processor is used to perform all computationally intensive tasks and implement a data-driven large language model performance prediction system. The memory is used to store normalization parameters, input feature templates, and prediction models for the processor to access during prediction. The API is used to support local and remote access, enabling a generalized deployment of prediction functionality.

Images