Method for predicting performance of pre-training model in downstream task
By plotting the relationship between the loss function value and performance of a large language model and fitting a regression model, the problem of the inability to predict the performance of downstream tasks in existing technologies is solved, and accurate prediction and optimized deployment of model performance at different scales are achieved.
Patent Information
- Application Number
- CN202410637917.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-22
- Publication Date
- 2025-12-02
AI Technical Summary
Existing technologies cannot effectively predict the performance of large language models in downstream tasks, especially when computing resources and time costs are limited, and cannot determine the model parameter size and training dataset size to meet the requirements of downstream tasks.
The initial model is trained by acquiring pre-training corpus, the relationship curve between loss function value and model performance is plotted, a regression model is fitted to predict model performance, and the prediction model is used to predict the performance in downstream tasks based on the loss function value.
It enables accurate prediction of the performance of pre-trained models in downstream tasks under different parameter scales and training data scales, helping users choose appropriate model deployment schemes to optimize cost-effectiveness.
Smart Images
Figure CN121051401A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of model performance prediction technology, and specifically relates to a method for predicting the performance of a pre-trained model in a downstream task. Background Technology
[0002] Large language models, such as GPT, PaLM, GLM, and LlaMA, have demonstrated powerful performance on downstream tasks due to their large parameter size and training dataset size, supporting applications like ChatGPT. However, pre-training large models consumes significant computational resources and time, and it's difficult to predict their performance on downstream tasks before training is complete. In many scenarios, we need to obtain models that meet required downstream task performance under certain constraints (e.g., the model's parameter size cannot exceed a certain maximum to reduce deployment costs). In such cases, we need to predict the minimum amount of data required for training to satisfy the downstream task performance requirements; however, current research on large model pre-training methods does not support this scenario.
[0003] Current research on pre-training of large models focuses on the relationship between the loss function value of the pre-trained large model and the model's parameter size and training dataset size; this extension law can only predict the loss function value of the pre-training model, but cannot predict the performance of the model in downstream tasks. Summary of the Invention
[0004] In view of the above analysis, the embodiments of the present invention aim to provide a method, apparatus, device and medium for predicting the performance of a pre-trained model in a downstream task, so as to solve one or more of the above-mentioned problems existing in the prior art.
[0005] The objective of this invention is achieved as follows:
[0006] A first aspect of the present invention provides a method for predicting the performance of a pre-trained model in a downstream task, comprising:
[0007] Obtain pre-training corpus;
[0008] The initial model is trained using the pre-training corpus to obtain the pre-trained model and loss function values;
[0009] The pre-trained model is evaluated on a specific downstream task to obtain model performance;
[0010] Plot the relationship curve between the loss function value and the model performance;
[0011] By fitting the relationship curve, a prediction model is obtained;
[0012] The prediction model is used to predict the performance of the pre-trained model in downstream tasks.
[0013] Further, training an initial model using the pre-training corpus includes: setting a pre-training corpus of a specific training data size, and a first initial model set including initial models of different parameter sizes; training the initial models in the first initial model set using the pre-training corpus to obtain a first pre-training model set, and the loss function value corresponding to each model in the set.
[0014] Furthermore, training the initial model using the pre-training corpus also includes: setting a corpus set containing pre-training corpora of different training data sizes, and a second initial model set consisting of initial models of different parameter sizes; training the initial models in the second initial model set using the pre-training corpora in the corpus set to obtain a second pre-training model set, and the loss function value corresponding to each model in the set.
[0015] Further, evaluating the pre-trained model on a specific downstream task includes: acquiring a dataset for the specific downstream task and performing data preprocessing and labeling; selecting an instruction format for evaluation; evaluating the pre-trained models in the first pre-trained model set and the second pre-trained model set using the dataset; and obtaining the model performance of the pre-trained model on the downstream task.
[0016] Further, plotting the relationship curve between the loss function value and the model performance includes: plotting the relationship curve between the loss function value and the corresponding model performance of the pre-trained models in the first pre-trained model set and the relationship curve between the loss function value and the corresponding model performance of the pre-trained models in the second pre-trained model set using visualization tools.
[0017] Further, fitting the relationship curve includes: selecting a regression model; fitting the relationship curve using the regression model to obtain a fitted model; and optimizing the fitted model to obtain the prediction model.
[0018] Furthermore, predicting the performance of the pre-trained model in a downstream task using the prediction model includes: obtaining the loss function value of the model to be predicted; inputting the loss function value of the model to be predicted into the prediction model; and the prediction model outputting the performance prediction values of the model to be predicted in the downstream task for different parameter scales and training data scales.
[0019] A second aspect of the present invention provides an apparatus for predicting the performance of a pre-trained model in a downstream task, comprising:
[0020] The acquisition module is used to acquire pre-training corpora.
[0021] The pre-training module is used to train an initial model using the pre-training corpus to obtain the pre-trained model and loss function values;
[0022] An evaluation module is used to evaluate the pre-trained model on a specific downstream task to obtain model performance;
[0023] The curve plotting module is used to plot the relationship curve between the loss function value and the model performance.
[0024] The fitting module is used to fit the relationship curve to obtain a prediction model;
[0025] The prediction module is used to predict the performance of the pre-trained model in downstream tasks using the prediction model.
[0026] A third aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements the method for predicting the performance of a pre-trained model in a downstream task as described in any embodiment.
[0027] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for predicting the performance of a pre-trained model in a downstream task as described in any embodiment.
[0028] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:
[0029] The method for predicting the performance of a pre-trained model in a downstream task provided by this invention fits a prediction model based on the relationship between the loss function value of the pre-trained model and the model performance in the downstream task, thereby predicting the performance of the pre-trained model in the downstream task through the loss function value. Attached Figure Description
[0030] To more clearly illustrate the technical solutions in the embodiments of this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings.
[0031] Figure 1 This is a flowchart of a method for predicting the performance of a pre-trained model in a downstream task, as provided in Embodiment 1 of the present invention.
[0032] Figure 2 This is a schematic diagram of the device for predicting the performance of a pre-trained model in a downstream task, as provided in Embodiment 2 of the present invention.
[0033] Figure 3This is a schematic diagram of the electronic device architecture provided in Embodiment 3 of the present invention. Detailed Implementation
[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be noted that, unless otherwise specified, the implementation methods and features in the implementation methods in this disclosure can be combined, separated, interchanged, and / or rearranged. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0035] Example 1
[0036] A specific embodiment of the present invention, such as Figure 1 As shown, a method for predicting the performance of a pre-trained model in a downstream task is disclosed, comprising the following steps:
[0037] S1. Obtain pre-training corpus;
[0038] In this embodiment, the loss function value of the model on the pre-training corpus needs to represent the loss function value of the model on general text content. Therefore, the pre-training corpus includes mixed Chinese and English corpus from multiple sources such as web pages, encyclopedias, books, and papers.
[0039] Furthermore, the model's loss function should be calculated based on the probability of the model predicting the next character in the training data, so that the loss function can represent the accuracy of the model's correct character predictions, such as Transformer, autoregressive language models, etc.; for example, the Exact Match (EM) metric commonly used in question-answering tasks, for input x, when the model predicts the next character... The index is 1 when it is exactly the same as the label value y, and 0 otherwise. Its expectation can be expressed as:
[0040]
[0041] in, Indicates predicted value The EM index, P, is related to the label value y. LM (yx) represents the conditional probability. This represents the loss function.
[0042] S2. Train the initial model using the pre-training corpus to obtain the pre-trained model and loss function value;
[0043] In this embodiment, step S2 includes the following steps:
[0044] S201, Set a pre-training corpus with a specific training data size, and a first set of initial models including initial models with different parameter sizes.
[0045] S202. Train the initial models in the first initial model set using the pre-training corpus to obtain the first pre-trained model set and the loss function value corresponding to each model in the set.
[0046] For example, a pre-training corpus of 3T with a relatively large training data size is set up, and a first set of initial models including initial models with parameter sizes of 1.5B, 6B and 32B is set up. The initial models in the first set of initial models are trained using the pre-training corpus to obtain the first set of pre-trained models and the loss function values of the pre-trained models in the set.
[0047] In some embodiments, step S2 further includes:
[0048] S203. Set up a corpus set including pre-trained corpora of different training data sizes, and a second initial model set consisting of initial models of different parameter sizes.
[0049] S204. Train the initial model in the second initial model set using the pre-trained corpus in the corpus set to obtain the second pre-trained model set and the loss function value corresponding to each model in the set.
[0050] To further analyze the impact of different parameter sizes and training data sizes on the performance of pre-trained models in downstream tasks, for example, a corpus set consisting of several pre-training corpora with training data sizes between 33B and 500B is set up, and a second initial model set including initial models with parameter sizes of 300M, 500M, 1B, 1.5B, 3B, and 6B is set up. After training the initial models in the second initial model set using the pre-training corpora in the corpus set, a second pre-trained model set is obtained, along with the loss function values of each pre-trained model in the set.
[0051] S3. Evaluate the pre-trained model on a specific downstream task to obtain the model performance;
[0052] In this embodiment, step S3 includes the following steps:
[0053] S301. Obtain the dataset for a specific downstream task and perform data preprocessing and labeling.
[0054] For example, open-source datasets such as TriviaQA, Hellaswag, MMLU, C-Eval, GSM8K, and GSM8K-Chinese are selected as datasets for specific downstream tasks. These datasets represent different types of natural language understanding tasks, including question answering and reasoning. Data cleaning, word segmentation, and word vectorization are performed as preprocessing, and data annotation is carried out according to the needs of the task.
[0055] S302. Select the instruction format to be used for evaluation, evaluate the pre-trained models in the first pre-trained model set and the second pre-trained model set through the dataset, and obtain the model performance of the pre-trained models on the downstream task.
[0056] Specifically, this step determines the appropriate instruction format based on the needs of the downstream task, including zero-shot instructions, few-shot instructions, etc., and then evaluates the pre-trained models in the first pre-trained model set and the second pre-trained model set through the dataset, such as accuracy, recall, F1 score, etc. The specific performance indicators depend on the nature of the downstream task.
[0057] S4. Plot the relationship curve between the loss function value and the model performance;
[0058] In this embodiment, step S4 specifically involves the following process:
[0059] The relationship curves between the loss function values of the pre-trained models in the first pre-trained model set and their corresponding model performance are plotted using visualization tools, as well as the relationship curves between the loss function values of the pre-trained models in the second pre-trained model set and their corresponding model performance. The visualization tools include Matplotlib, Seaborn, etc.
[0060] S5. Fit the relationship curve to obtain the prediction model;
[0061] In this embodiment, step S5 specifically includes the following steps:
[0062] S501. Select a regression model.
[0063] For example, the regression model can be linear regression, multinomial regression, or neural network, and the appropriate regression model can be selected based on the loss function value and model performance.
[0064] S502. Fit the relationship curve using the regression model to obtain the fitted model.
[0065] Specifically, the loss function values and corresponding model performance data obtained in steps S2 and S3 are used as the training set. The regression model is trained using this training set to fit these data points in order to obtain the fitted model.
[0066] S503. Optimize the fitting model to obtain the prediction model.
[0067] Specifically, by adjusting the model's hyperparameters and using techniques such as cross-validation to optimize the fitted model, a predictive model with sufficient predictive power can be obtained to predict the performance of the pre-trained model in downstream tasks.
[0068] S6. Predict the performance of the pre-trained model in downstream tasks using the prediction model.
[0069] Compared with the prior art, the method for predicting the performance of a pre-trained model in a downstream task provided in this embodiment fits a prediction model based on the relationship between the loss function value of the pre-trained model and the model performance in the downstream task, thereby predicting the performance of the pre-trained model in the downstream task through the loss function value.
[0070] In this embodiment, step S6 specifically includes the following steps:
[0071] S601. Obtain the loss function value of the model to be predicted.
[0072] Specifically, when it is necessary to predict the performance of a pre-trained model on a downstream task, it is necessary to first obtain the loss function value of the model during the pre-training stage.
[0073] S602. Input the loss function value of the model to be predicted into the prediction model.
[0074] Specifically, the loss function value obtained in step S601 is input into the prediction model obtained in step S503 for prediction.
[0075] S603, The prediction model outputs the performance prediction values of the model to be predicted in downstream tasks with different parameter scales and training data scales.
[0076] Specifically, after receiving the loss function value and processing it accordingly, the prediction model calculates and outputs the performance prediction value of the pre-trained model corresponding to the loss function value in different downstream tasks under different parameter sizes and training data sizes. This allows users to select a model with appropriate parameter sizes and training data sizes based on the output prediction results, and to intuitively understand the model's performance in downstream tasks, and further analyze the cost-effectiveness of deploying the pre-trained model in downstream tasks.
[0077] Example 2
[0078] This embodiment provides an apparatus for predicting the performance of a pre-trained model in a downstream task, such as... Figure 2 As shown, it includes:
[0079] The acquisition module is used to acquire pre-training corpora.
[0080] The pre-training module is used to train an initial model using the pre-training corpus to obtain the pre-trained model and loss function values;
[0081] An evaluation module is used to evaluate the pre-trained model on a specific downstream task to obtain model performance;
[0082] The curve plotting module is used to plot the relationship curve between the loss function value and the model performance.
[0083] The fitting module is used to fit the relationship curve to obtain a prediction model;
[0084] The prediction module is used to predict the performance of the pre-trained model in downstream tasks using the prediction model.
[0085] Example 3
[0086] This embodiment provides an electronic device, such as... Figure 3 As shown, it includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method for predicting the performance of a pre-trained model in a downstream task as described in any of the above embodiments.
[0087] Example 4
[0088] This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method for predicting the performance of a pre-trained model in a downstream task as described in any of the above embodiments.
[0089] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0090] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.
[0091] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0092] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for predicting the performance of a pre-trained model in a downstream task, characterized in that, include: Obtain pre-training corpus; The initial model is trained using the pre-training corpus to obtain the pre-trained model and loss function values; The pre-trained model is evaluated on a specific downstream task to obtain model performance; Plot the relationship curve between the loss function value and the model performance; By fitting the relationship curve, a prediction model is obtained; The prediction model is used to predict the performance of the pre-trained model in downstream tasks.
2. The method for predicting the performance of a pre-trained model in a downstream task according to claim 1, characterized in that, Training an initial model using the pre-training corpus includes: Set up a pre-training corpus with a specific training data size, and a first set of initial models including initial models with different parameter sizes; The initial models in the first initial model set are trained using the pre-training corpus to obtain the first pre-trained model set and the loss function value corresponding to each model in the set.
3. The method for predicting the performance of a pre-trained model in a downstream task according to claim 2, characterized in that, Training the initial model using the pre-training corpus also includes: Set up a corpus set including pre-trained corpora with different training data sizes, and a second initial model set consisting of initial models with different parameter sizes; The initial models in the second initial model set are trained using the pre-trained corpus in the corpus set to obtain the second pre-trained model set and the loss function value corresponding to each model in the set.
4. The method for predicting the performance of a pre-trained model in a downstream task according to claim 3, characterized in that, Evaluating the pre-trained model on a specific downstream task includes: Obtain the dataset for a specific downstream task, and perform data preprocessing and labeling; Select the instruction format to be used during evaluation, and evaluate the pre-trained models in the first pre-trained model set and the second pre-trained model set using the dataset; Obtain the model performance of the pre-trained model on downstream tasks.
5. The method for predicting the performance of a pre-trained model in a downstream task according to claim 4, characterized in that, Plotting the relationship curve between the loss function value and the model performance includes: Using visualization tools, plot the relationship curves between the loss function values of the pre-trained models in the first pre-trained model set and their corresponding model performance, as well as the relationship curves between the loss function values of the pre-trained models in the second pre-trained model set and their corresponding model performance.
6. The method for predicting the performance of a pre-trained model in a downstream task according to claim 5, characterized in that, Fitting the relationship curve includes: Choose a regression model; The regression model is used to fit the relationship curve to obtain the fitted model; The fitting model is optimized to obtain the prediction model.
7. The method for predicting the performance of a pre-trained model in a downstream task according to claim 6, characterized in that, Predicting the performance of the pre-trained model in downstream tasks using the prediction model includes: Obtain the loss function value of the model to be predicted; Input the loss function value of the model to be predicted into the prediction model; The prediction model outputs performance predictions of the model under test in downstream tasks with different parameter scales and training data scales.
8. An apparatus for predicting the performance of a pre-trained model in a downstream task, characterized in that, The device includes: The acquisition module is used to acquire pre-training corpora. The pre-training module is used to train an initial model using the pre-training corpus to obtain the pre-trained model and loss function values; An evaluation module is used to evaluate the pre-trained model on a specific downstream task and obtain the model performance; The curve plotting module is used to plot the relationship curve between the loss function value and the model performance. The fitting module is used to fit the relationship curve to obtain a prediction model; The prediction module is used to predict the performance of the pre-trained model in downstream tasks using the prediction model.
9. An electronic device, characterized in that, The system includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method for predicting the performance of a pre-trained model in a downstream task as described in any one of claims 1-7.
10. A storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method for predicting the performance of a pre-trained model in a downstream task as described in any one of claims 1-7.