A model hyperparameter determination method and device, electronic equipment and medium
By using a specified optimization algorithm for iterative search and supervised learning training, the efficiency and accuracy issues of hyperparameter selection for large language models are addressed, achieving efficient hyperparameter determination in specific task scenarios and improving model performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG LAB
- Filing Date
- 2025-07-23
- Publication Date
- 2026-05-12
AI Technical Summary
Existing technologies struggle to meet user needs in terms of efficiency and accuracy of hyperparameter selection for specific task scenarios within large language models, especially in high-frequency recommendation tasks where performance declines.
By iteratively searching the hyperparameter space of the target large model using a specified optimization algorithm, a labeled data sequence is constructed, and an initial sequence model is trained through supervised learning to obtain the target hyperparameters.
It enables the rapid and accurate determination of hyperparameters of a target large model in specific task scenarios, avoiding the decrease in optimization algorithm efficiency caused by the increase in data sequence and improving model performance.
Smart Images
Figure CN120893528B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device and medium for determining model hyperparameters. Background Technology
[0002] With the development of artificial intelligence technology, large language models are increasingly being widely applied to complex information processing tasks across various fields. However, currently used large language models perform poorly for specific domain tasks, such as intelligence analysis where accuracy, real-time performance, and security are critical, making it difficult to meet actual user needs.
[0003] To improve the performance of large models in specific task scenarios, fine-tuning can be performed during training to allow the model to better understand user intent. In practice, the reasonable, efficient, and accurate selection of hyperparameters during training is crucial to the final model performance.
[0004] Currently, reinforcement learning algorithms are mainly used to efficiently recommend hyperparameters of large models. However, as historical data increases, the complexity of inference by reinforcement learning algorithms increases, leading to a decline in efficiency and accuracy in large-scale problems or high-frequency recommendation tasks.
[0005] Therefore, how to accurately and efficiently obtain the hyperparameters of large models to meet the needs of different downstream tasks is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0006] In view of this, one aspect of this application provides a method for determining model hyperparameters, the method comprising:
[0007] Obtain the pre-built initial sequence model and the target large model;
[0008] The hyperparameter space of the target large model is iteratively searched using a specified optimization algorithm in order to construct a label data sequence based on the hyperparameter space.
[0009] The target sequence model is obtained by supervising the training of the initial sequence model using the labeled data sequence.
[0010] The target hyperparameters of the target large model are determined using the target sequence model.
[0011] Optionally, the step of iteratively searching the hyperparameter space of the target large model using a specified optimization algorithm to construct a label data sequence based on the hyperparameter space includes:
[0012] Obtain the test dataset of the target large model, the preset initial hyperparameters, and the initial model accuracy corresponding to the initial hyperparameters;
[0013] Use the initial hyperparameters and the initial model accuracy as the current data sequence;
[0014] Input the current data sequence into the specified optimization algorithm to obtain the algorithm's recommended hyperparameters;
[0015] After setting the hyperparameters of the target large model to the hyperparameters recommended by the algorithm, the target large model is fine-tuned and trained using the test dataset to determine the current model accuracy of the target large model.
[0016] The sequence consisting of the hyperparameters recommended by the algorithm and the accuracy of the current model is added to the current data sequence, and the step of inputting the current data sequence into the specified optimization algorithm to obtain the hyperparameters recommended by the algorithm is returned to perform the iterative search until the preset iteration condition is reached.
[0017] The label data sequence is constructed based on the current data sequence after the preset iteration conditions are met.
[0018] Optionally, the specified optimization algorithm is a Bayesian optimization algorithm.
[0019] Optionally, determining the target hyperparameters of the target large model through the target sequence model includes:
[0020] Obtain the historical data sequence and test dataset of the target large model; the historical data sequence consists of historical hyperparameters and historical model accuracy.
[0021] Input the historical data sequence into the target sequence model to obtain the model's recommended hyperparameters;
[0022] After setting the hyperparameters of the target large model to the recommended hyperparameters of the model, the target large model is fine-tuned and trained using the test dataset to determine the target model accuracy of the target large model;
[0023] The sequence consisting of the model's recommended hyperparameters and the target model's accuracy is added to the historical data sequence, and the process of inputting the historical data sequence into the target sequence model to obtain the model's recommended hyperparameters is repeated until the target model's accuracy reaches the accuracy threshold.
[0024] Optionally, the target sequence model is a model based on the Transformer architecture.
[0025] Optionally, the target sequence model includes a first multilayer perceptual mechanism, a second multilayer perceptual mechanism, and a multilayer attention mechanism; the step of inputting the historical data sequence into the target sequence model to obtain the model recommendation hyperparameters includes:
[0026] The historical data sequence is encoded into a target vector through the first multi-layer sensing mechanism;
[0027] The target vector is input into the multi-layer attention mechanism to generate a global aggregate representation;
[0028] The global aggregation representation is input into the second multilayer perception mechanism to obtain the model's recommended hyperparameters.
[0029] Optionally, before performing supervised learning training on the initial sequence model using the labeled data sequence, the following steps are included:
[0030] Obtain the expert-recommended hyperparameters and the test dataset of the target large model; wherein, there are multiple expert-recommended hyperparameters;
[0031] After setting the hyperparameters of the target large model to the expert-recommended hyperparameters, the target large model is fine-tuned and trained using the test dataset to determine the accuracy of the target model corresponding to the expert-recommended hyperparameters.
[0032] The label data sequence is constructed based on the expert-recommended hyperparameters and the accuracy of the target model.
[0033] Another aspect of this application provides a device for determining model hyperparameters, the device comprising:
[0034] The first acquisition module is used to acquire the pre-built initial sequence model and the target large model;
[0035] The label data sequence generation module is used to iteratively search the hyperparameter space of the target large model using a specified optimization algorithm, so as to construct a label data sequence based on the hyperparameter space;
[0036] The supervised learning module is used to perform supervised learning training on the initial sequence model using the labeled data sequence to obtain the target sequence model;
[0037] The target hyperparameter determination module is used to determine the target hyperparameters of the target large model through the target sequence model.
[0038] Another aspect of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the computer program to implement the steps of the method for determining the model hyperparameters.
[0039] Another aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for determining the model hyperparameters.
[0040] The method, apparatus, electronic device, and medium for determining model hyperparameters provided in this application have the following beneficial effects: by iteratively searching the hyperparameter space of a target large model using a specified optimization algorithm, a labeled data sequence including hyperparameter labels is obtained. The initial sequence model is then trained using the labeled data sequence through supervised learning, enabling the initial sequence model to learn the hyperparameter recommendation strategy of the specified optimization algorithm. This allows the target sequence model to be used directly to quickly and accurately determine the hyperparameters of the target large model, avoiding the continuous increase of the data sequence during iterative search, which would reduce the efficiency of the optimization algorithm in recommending hyperparameters. Attached Figure Description
[0041] Figure 1 A flowchart illustrating a method for determining model hyperparameters provided in an embodiment of this application;
[0042] Figure 2 A schematic diagram illustrating the principle of a method for determining model hyperparameters provided in an embodiment of this application;
[0043] Figure 3 A schematic diagram illustrating the principle of a method for determining model hyperparameters provided in another embodiment of this application;
[0044] Figure 4 A schematic diagram of a device for determining model hyperparameters provided in an embodiment of this application;
[0045] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.
[0046] The reference numerals in the attached figures are as follows: 40 is the first acquisition module, 41 is the tag data sequence generation module, 42 is the supervised learning module, 43 is the target hyperparameter determination module, 50 is the memory, 51 is the processor, 52 is the display screen, 53 is the input / output interface, 54 is the communication interface, 55 is the power supply, 56 is the communication bus, 501 is the computer program, 502 is the operating system, and 503 is the data. Detailed Implementation
[0047] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0048] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."
[0049] Figure 1 This is a flowchart illustrating a method for determining model hyperparameters provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:
[0050] S10: Obtain the pre-built initial sequence model and target large model;
[0051] In a specific embodiment, in order to efficiently and accurately determine the target hyperparameters of the target large model, it can be achieved by a pre-built initial sequence model. The initial sequence model refers to a machine learning model used to process sequence data, and the initial sequence model may include, but is not limited to, RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory Network), Transformer-based models, and CNN (One-Dimensional Convolutional Network).
[0052] It should be noted that the target large model provided in the embodiments of this application may include, but is not limited to, the BERT (Bidirectional Encoder Representations from Transformers) model, the GPT series, and the Tongyi 1000 Questions.
[0053] S11: Iteratively search the hyperparameter space of the target large model by specifying an optimization algorithm, so as to construct a label data sequence based on the hyperparameter space;
[0054] Understandably, optimization algorithms can systematically search for the hyperparameters of a large target model, avoiding local optima from manual parameter tuning, and enabling the model to achieve higher accuracy or lower loss on a specific task. Therefore, in one optional embodiment, to ensure that the pre-built initial sequence model can accurately determine the target hyperparameters of the large target model, a labeled data sequence can be obtained first by specifying an optimization algorithm.
[0055] Furthermore, the initial sequence model is trained using labeled data sequences through supervised learning. This allows the initial sequence model to learn the recommendation strategy for hyperparameters from the specified optimization algorithm. Therefore, the target hyperparameters can be quickly determined based on the optimization algorithm's recommendation strategy, while also avoiding the decrease in hyperparameter recommendation efficiency caused by the continuous increase in data sequences in large-scale data sequence scenarios. It should be noted that, in specific embodiments, in addition to supervised learning training, reinforcement learning, maximizing the acquisition function, and other strategies can also be used to train the initial sequence model; this application does not limit the scope of the application.
[0056] It is worth noting that the hyperparameter space refers to the multidimensional space composed of all possible values of the hyperparameters of the target large model. The hyperparameters of the target large model may include, but are not limited to, the learning rate, gradient accumulation, batch size, and truncation length. This application does not limit these parameters and they can be determined according to the actual target large model and application scenario.
[0057] A labeled data sequence refers to a data sequence that includes labeled data. Specifically, the labeled data sequence includes the hyperparameter space and the model accuracy corresponding to each hyperparameter in the hyperparameter space. The model accuracy is the accuracy of the target large model.
[0058] S12: Supervised learning training of the initial sequence model is performed using the labeled data sequence to obtain the target sequence model;
[0059] S13: Determine the target hyperparameters of the target large model through the target sequence model.
[0060] Furthermore, after obtaining the labeled data sequence, the initial sequence model is trained using supervised learning based on the labeled data sequence, enabling the final target sequence model to possess the same hyperparameter recommendation capability as a specified optimization algorithm. Therefore, in a specific embodiment, the target hyperparameters of the target large model can be directly determined through the target sequence model.
[0061] In one optional embodiment, when the initial sequence model is trained using supervised learning with labeled data sequences, the training process aims to maximize consistency with the Bayesian optimization recommendation results. This consistency includes, but is not limited to, consistency in point selection, consistency in sampling and ranking, and consistency in maximum boosting. In other words, using the decisions of a specified optimization algorithm as teacher signals, a lightweight model (i.e., the initial sequence model) is trained through supervised learning or reinforcement learning, ensuring that the initial sequence model is consistent with the specified optimization algorithm in point selection, ranking, and boosting performance.
[0062] Figure 2 This is a schematic diagram illustrating the principle of a method for determining model hyperparameters provided in an embodiment of this application. To enable those skilled in the art to better understand the technical solution provided in this application, the following will be combined with... Figure 2 Further explanation is needed.
[0063] like Figure 2 As shown, in a specific embodiment, the label data sequence is obtained by specifying an optimization algorithm. In the initial moment, the target large model hyperparameters can be manually set, and the initial model accuracy of the target large model under the hyperparameters can be determined. Furthermore, the initial hyperparameters and the corresponding initial model accuracy are optimized and iteratively searched by the target optimization algorithm to obtain the label data sequence.
[0064] Furthermore, the labeled data sequence is input into the initial sequence model for supervised learning training, so that the initial sequence model learns the recommendation capability of the specified optimization algorithm based on the labeled data sequence, thus obtaining the target sequence model. Therefore, the current target hyperparameters of the target large model can be directly determined based on the target sequence model.
[0065] It is worth noting that the method for determining model hyperparameters provided in this application can be applied to real-world scenarios requiring rapid response, such as edge devices, high-frequency request scenarios, and intelligence analysis. Furthermore, the method provided in this application is also applicable to tasks such as black-box objective function optimization, experimental condition recommendation, and automatic parameter adjustment; however, this application does not limit its application to these areas.
[0066] Therefore, the method for determining model hyperparameters provided in this application iteratively searches the hyperparameter space of the target large model using a specified optimization algorithm, thereby obtaining a labeled data sequence including hyperparameter labels. The initial sequence model is then trained using the labeled data sequence through supervised learning, enabling the initial sequence model to learn the hyperparameter recommendation strategy of the specified optimization algorithm. This allows the target sequence model to be used directly to quickly and accurately determine the hyperparameters of the target large model, avoiding the continuous increase of data sequences during iterative searches, which would reduce the efficiency of the optimization algorithm in recommending hyperparameters.
[0067] In one alternative embodiment, an optimization algorithm is used to iteratively search the hyperparameter space of the target large model in order to construct a label data sequence based on the hyperparameter space, including:
[0068] Obtain the test dataset of the target large model, the preset initial hyperparameters, and the initial model accuracy corresponding to the initial hyperparameters;
[0069] Use the initial hyperparameters and initial model accuracy as the current data sequence;
[0070] Input the current data sequence into the specified optimization algorithm to obtain the algorithm's recommended hyperparameters;
[0071] After setting the hyperparameters of the target large model to the hyperparameters recommended by the algorithm, the target large model is fine-tuned and trained using a test dataset to determine the current model accuracy.
[0072] The sequence consisting of the algorithm's recommended hyperparameters and the current model's accuracy is added to the current data sequence, and the steps of inputting the current data sequence into the specified optimization algorithm to obtain the algorithm's recommended hyperparameters are returned for iterative search until the preset iteration conditions are met.
[0073] Based on the current data sequence after the preset iteration conditions are met, a label data sequence is constructed.
[0074] In one alternative embodiment, the target large model can be an intelligence analysis model. In a specific embodiment of constructing the labeled data sequence, the model fine-tuning architecture of LLaMA-Factory can be used to train and evaluate the target large model.
[0075] In a specific embodiment, a test dataset for the intelligence analysis model is obtained. In one optional embodiment, the test dataset may be text data including questions and answers. Furthermore, in order to iteratively search the hyperparameters of the intelligence analysis model using a specified optimization algorithm, preset initial hyperparameters also need to be obtained.
[0076] In one optional embodiment, the initial hyperparameters are denoted as a vector x1 (which may include, but is not limited to, learning rate, gradient accumulation, batch size, and truncation length). Before fine-tuning the target large model using the test dataset, the Train module in LLaMA-Factory sets this initial hyperparameter x1. Further, using the test dataset, the Evaluate&Predict module in LLaMA-Factory evaluates and predicts the model, thereby obtaining the initial model accuracy f(x1) of the target large model corresponding to the initial hyperparameters. At this point, the sequence consisting of the initial hyperparameters x1 and the initial model accuracy f(x1) is used as the current data sequence, i.e., as the initial data sequence.
[0077] Furthermore, the current data sequence is input into the optimization algorithm, which calculates the next recommended hyperparameter, i.e., the algorithm-recommended hyperparameter x2. Similarly, after replacing the current hyperparameters of the target model with the algorithm-recommended hyperparameter x2, the target model is fine-tuned using the test dataset to obtain the current model accuracy f(x2).
[0078] Furthermore, the algorithm-recommended hyperparameter x2 and the current model accuracy f(x2) are added to the current data sequence, which is now [(x1, f(x1)), (x2, f(x2))]. Simultaneously, the process returns to the step of inputting the current data sequence into the specified optimization algorithm to obtain the algorithm's recommended hyperparameters, thus yielding the next recommended hyperparameter x3. This iterative search continues until the preset iteration conditions are met.
[0079] The preset iteration conditions can be a preset number of iterations, or they can be determined based on the effect of the recommendation strategy of the specified optimization algorithm learned by the target sequence model. This application does not limit this.
[0080] After the above iterative search, after t iterations, the current data sequence obtained is [(x1,f(x1)),(x2,f(x2)),……,(x t ,f(x t It is understandable that, in order for the initial sequence model to learn the recommendation ability of the specified optimization algorithm, the data sequence used to train the initial sequence model is a labeled data sequence that includes labeled data. Here, labeled data refers to the algorithm-recommended hyperparameters obtained by the specified optimization algorithm after calculating each pair of hyperparameters and model accuracy.
[0081] Therefore, based on the current data sequence [(x1,f(x1)),(x2,f(x2)),……,(x t ,f(x t [)] can construct a label data sequence including label data. Correspondingly, the label data sequence can be represented as [[(x1,f(x1)),x2],[(x2,f(x2)),x3],……, ... t ,f(x t )),x t+1 ]).
[0082] It should be noted that, considering factors such as computational resources, efficiency, and hyperparameter calculation accuracy, Bayesian optimization algorithms are chosen as the preferred choice in one optional embodiment due to their advantages, including high efficiency, high sample efficiency, good balance between exploration and utilization, high flexibility, lack of gradient information requirement, and strong adaptability. Correspondingly, in specific embodiments, the label data sequence may, but is not limited to, Bayesian optimization algorithms such as Gaussian Process Regression (GP), Tree-structured Parzen Estimator (TPE), and Upper Confidence Bound (UCB).
[0083] As can be seen, the method for determining model hyperparameters provided in this application provides corresponding algorithm-recommended hyperparameters (i.e., label data) for different hyperparameters and model accuracy in the obtained label data sequence, so that the subsequent initial sequence model can learn the recommendation strategy of the specified optimization algorithm from the label data sequence.
[0084] In one optional embodiment, the target hyperparameters of the target large model are determined through the target sequence model, including:
[0085] Obtain the historical data sequence and test dataset of the target large model; the historical data sequence consists of historical hyperparameters and historical model accuracy.
[0086] Input the historical data sequence into the target sequence model to obtain the model's recommended hyperparameters;
[0087] After setting the hyperparameters of the target large model to the recommended hyperparameters, the target large model is fine-tuned and trained using a test dataset to determine the target model accuracy.
[0088] The sequence consisting of the model's recommended hyperparameters and the target model's accuracy is added to the historical data sequence. The process of inputting the historical data sequence into the target sequence model to obtain the model's recommended hyperparameters is repeated until the target model's accuracy reaches the accuracy threshold.
[0089] In a specific embodiment, such as Figure 2 As shown, in the embodiment of determining the actual target hyperparameters of the target large model, a test dataset that can be used to train the target large model is obtained. The test dataset can be text or images, and this application does not limit this. However, it should be noted that the test dataset is a dataset that includes questions and answers.
[0090] In addition, such as Figure 2As shown, it is also necessary to obtain historical data sequences. These historical data sequences are composed of historical hyperparameters and historical model accuracy. In an optional embodiment, the historical data sequences can be represented as [(x1,f(x1)),(x2,f(x2)),……,(x...]. i ,f(x i ))).
[0091] It should be noted that the historical data sequence is a data sequence obtained in the actual downstream task scenario of the target large model (e.g., intelligence analysis scenario), and is different from the labeled data sequence obtained through a specified optimization algorithm. Furthermore, it should be noted that at the initial moment of the downstream task scenario, an initial hyperparameter can be generated by the target sequence model, and the initial model accuracy corresponding to the initial hyperparameter can be obtained. The initially obtained hyperparameter and model accuracy are then used as the historical data sequence for further processing. Figure 2 The loop iteration is shown.
[0092] like Figure 2 As shown, inputting historical data sequences into the target sequence model yields recommended hyperparameters. Further, the current hyperparameters of the target model are set to the recommended hyperparameters, and the target model is fine-tuned using a test dataset. Simultaneously, the accuracy of the target model corresponding to the current recommended hyperparameters is obtained. Further, the historical data sequences are updated using the recommended hyperparameters and the target model accuracy. This process is repeated until the target model accuracy reaches an accuracy threshold.
[0093] Therefore, the method for determining model hyperparameters provided in this application continuously calculates the target hyperparameters based on historical sequence data to obtain the target hyperparameters corresponding to the expected model accuracy, based on the target sequence model.
[0094] Figure 3 This is a schematic diagram illustrating the principle of a method for determining model hyperparameters according to another embodiment of this application. In one optional embodiment, the target sequence model is a Transformer-based model. Of course, in another optional embodiment, it could also be a variant of the Transformer architecture, including but not limited to a Key-only Attention structure.
[0095] Based on the above embodiments, such as Figure 3 As shown, the target sequence model includes a first multilayer perception mechanism, a second multilayer perception mechanism, and a multilayer attention mechanism. In an optional embodiment, historical data sequences are input into the target sequence model to obtain the model's recommended hyperparameters, including:
[0096] The historical data sequence is encoded into a target vector through the first multi-layer sensing mechanism;
[0097] The target vector is input into a multi-layer attention mechanism to generate a global aggregate representation;
[0098] The global aggregate representation is input into the second multilayer perceptual mechanism to obtain the model's recommended hyperparameters.
[0099] In a specific embodiment, the target sequence modulo obtains the historical data sequence (x). i ,f(x i After that, the historical data sequence (x) is processed through a multilayer sensing mechanism (MLP). i ,f(x i The target vector is encoded and input into a multi-layer attention mechanism. After processing through several layers of attention mechanisms, the output model recommends hyperparameters through another MLP layer.
[0100] It is understandable that, in specific embodiments, in order to meet the accuracy requirements of different downstream processing tasks, further, such as Figure 3 As shown, the current parameters of the target large model are set as the recommended hyperparameters, and fine-tuning is performed using a test dataset. The historical data sequence is continuously updated until the optimal target hyperparameters for the target large model are obtained. Therefore, based on a given historical data sequence, recommended hyperparameters can be obtained directly by inputting the model, without the need to build a posterior model. Furthermore, thanks to the efficient computational characteristics of the attention mechanism, it can achieve rapid response even when dealing with large-scale data, supporting real-time online application requirements.
[0101] Furthermore, the target hyperparameters and their corresponding model accuracies are stored. This allows the target hyperparameters to be directly retrieved and used in subsequent downstream tasks to meet their requirements. Alternatively, in one optional embodiment, actual hyperparameters and model accuracies can be continuously acquired and added to the historical data sequence during downstream tasks. This updates the historical data sequence to continuously and accurately determine the target hyperparameters for different downstream tasks.
[0102] In one alternative embodiment, prior to supervised learning training of the initial sequence model using labeled data sequences, the following steps are included:
[0103] Obtain the test dataset for the expert-recommended hyperparameters and the target large model; among them, there are multiple expert-recommended hyperparameters;
[0104] After setting the hyperparameters of the target large model to the expert-recommended hyperparameters, the target large model is fine-tuned and trained using a test dataset to determine the accuracy of the target model corresponding to the expert-recommended hyperparameters.
[0105] A label data sequence is constructed based on expert-recommended hyperparameters and the accuracy of the target model.
[0106] Building upon the above embodiments, to further improve the accuracy of model hyperparameter determination, in an optional embodiment, manually recommended hyperparameters can be introduced. In a specific embodiment, expert-recommended hyperparameters are obtained, and the hyperparameters of the target large model are sequentially set to these expert-recommended hyperparameters. Then, the target large model is fine-tuned using a test dataset to determine the accuracy of the target model corresponding to the expert-recommended hyperparameters.
[0107] Based on the current accuracy of the target model, obtain the next expert-recommended hyperparameters after adjustment by human experience, and return to the step of setting the hyperparameters of the target model to the expert-recommended hyperparameters. This process is repeated until a stop command is received from the user.
[0108] Therefore, after iterative iteration, a data sequence generated by expert recommendations can be obtained, and a label data sequence can also be generated based on this data sequence. For example... Figure 2 As shown, the labeled data sequence includes sequences obtained through a specified optimization algorithm and sequences obtained through experts, further improving the accuracy of the target sequence model in calculating target hyperparameters. In this embodiment, during training, special emphasis is placed on recommended hyperparameter data from domain experts. By adding this labeled data sequence, it is ensured that the initial sequence model can effectively learn from the expert's experience, further improving the accuracy of hyperparameter calculation.
[0109] It should be noted that, in one optional embodiment, the method for determining model hyperparameters provided in this application supports processing multiple input data sequences simultaneously, or continuously receiving new data sequences for updating predictions, greatly improving the system's flexibility and practicality. In other words, the method provided in this application can simultaneously input labeled data sequences obtained through a specified optimization algorithm, labeled data sequences obtained through professional experience, historical data sequences generated by the target large model in actual downstream tasks, and historical data sequences obtained through test datasets.
[0110] Therefore, the hyperparameter determination method provided in this application does not require an explicit acquisition function. Instead, it simulates the sampling strategy of the optimization algorithm (i.e., the hyperparameter recommendation strategy) through a trainable attention mechanism. A multi-layer attention network is introduced to fuse function values, positions, and sparsity, and it supports end-to-end training, possessing strong generalization capabilities and adaptability to different optimization tasks. Furthermore, it requires no external query, relying entirely on existing historical information for inference. Its flexible architecture allows it to be embedded in meta-learning frameworks for multi-task optimization.
[0111] In the above embodiments, the method for determining model hyperparameters has been described in detail. This application also provides an embodiment of a device for determining model hyperparameters.
[0112] Figure 4 This is a schematic diagram of the structure of a model hyperparameter determination device provided in an embodiment of this application, as shown below. Figure 4 As shown, the device includes:
[0113] The first acquisition module 40 is used to acquire the pre-built initial sequence model and the target large model;
[0114] The label data sequence generation module 41 is used to iteratively search the hyperparameter space of the target large model by specifying an optimization algorithm, so as to construct the label data sequence based on the hyperparameter space;
[0115] Supervised learning module 42 is used to train the initial sequence model using labeled data sequences to obtain the target sequence model.
[0116] The target hyperparameter determination module 43 is used to determine the target hyperparameters of the target large model through the target sequence model.
[0117] Furthermore, the model hyperparameter determination device provided in this application embodiment also includes:
[0118] The second acquisition module is used to acquire the test dataset of the target large model, the preset initial hyperparameters, and the initial model accuracy corresponding to the initial hyperparameters;
[0119] The current sequence determination module is used to take the initial hyperparameters and initial model accuracy as the current data sequence;
[0120] The algorithm recommendation hyperparameter acquisition module is used to input the current data sequence into the specified optimization algorithm to obtain the algorithm recommendation hyperparameters;
[0121] The first training module is used to set the hyperparameters of the target large model to the hyperparameters recommended by the algorithm, and then fine-tune the target large model with instructions through the test dataset to determine the current model accuracy of the target large model.
[0122] The current data sequence update module is used to add the sequence consisting of the algorithm's recommended hyperparameters and the current model accuracy to the current data sequence, and return the steps of inputting the current data sequence into the specified optimization algorithm to obtain the algorithm's recommended hyperparameters for iterative search until the preset iteration conditions are met.
[0123] The first construction module of the sequence is used to construct a label data sequence based on the current data sequence after the preset iteration conditions are met.
[0124] The third acquisition module is used to acquire the historical data sequence and test dataset of the target large model; the historical data sequence consists of historical hyperparameters and historical model accuracy.
[0125] The model recommendation hyperparameter determination module is used to input historical data sequences into the target sequence model to obtain the model recommendation hyperparameters;
[0126] The second training module is used to set the hyperparameters of the target large model to the recommended hyperparameters of the model, and then fine-tune the target large model with instructions through the test dataset to determine the target model accuracy of the target large model.
[0127] The historical data sequence update module is used to add the sequence consisting of the model's recommended hyperparameters and the target model's accuracy to the historical data sequence, and return the steps of inputting the historical data sequence into the target sequence model to obtain the model's recommended hyperparameters, until the target model's accuracy reaches the accuracy threshold.
[0128] The encoding module is used to encode historical data sequences into target vectors through a first multi-layer sensing mechanism;
[0129] The global aggregation representation acquisition module is used to input the target vector into a multi-layer attention mechanism to generate a global aggregation representation;
[0130] The input module is used to input the global aggregate representation into the second multilayer perceptron to obtain the model's recommended hyperparameters.
[0131] The fourth acquisition module is used to acquire the expert-recommended hyperparameters and the test dataset of the target large model; among them, there are multiple expert-recommended hyperparameters.
[0132] The third training module is used to set the hyperparameters of the target large model to the expert-recommended hyperparameters, and then fine-tune the target large model using the test dataset to determine the accuracy of the target model corresponding to the expert-recommended hyperparameters.
[0133] The second construction module of the sequence is used to construct the label data sequence based on the expert-recommended hyperparameters and the accuracy of the target model.
[0134] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 5 As shown, the electronic device includes: a memory 50 for storing computer programs;
[0135] The processor 51 is used to implement the steps of the method for determining model hyperparameters as described in the above embodiments when executing a computer program.
[0136] The electronic devices provided in this embodiment may include, but are not limited to, laptops or desktop computers.
[0137] The processor 51 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 51 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 51 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 51 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 51 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.
[0138] The memory 50 may include one or more computer-readable storage media, which may be non-transitory. The memory 50 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 50 is used to store at least the following computer program 501, which, after being loaded and executed by the processor 51, is capable of implementing the relevant steps of the model hyperparameter determination method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 50 may also include an operating system 502 and data 503, and the storage method may be temporary storage or permanent storage. The operating system 502 may include Windows, Unix, Linux, etc. The data 503 may include, but is not limited to, the relevant data involved in the model hyperparameter determination method.
[0139] In some embodiments, the electronic device may further include a display screen 52, an input / output interface 53, a communication interface 54, a power supply 55, and a communication bus 56.
[0140] Those skilled in the art will understand that Figure 5 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.
[0141] The electronic device provided in this application includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the method for determining the model hyperparameters in the above embodiments.
[0142] It should be noted that although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
Claims
1. A control method for intelligence analysis, characterized in that, The method includes: Obtain the pre-built initial sequence model and the target large model; The hyperparameter space of the target large model is iteratively searched using a specified optimization algorithm in order to construct a label data sequence based on the hyperparameter space. The target sequence model is obtained by supervising the training of the initial sequence model using the labeled data sequence. The target sequence model is used to determine the target hyperparameters of the target large model, so as to perform intelligence analysis through the target large model; the sequence model is a model based on the Transformer architecture; the target large model is an intelligence analysis module. The step of iteratively searching the hyperparameter space of the target large model using a specified optimization algorithm to construct a label data sequence based on the hyperparameter space includes: Obtain the test dataset of the target large model, the preset initial hyperparameters, and the initial model accuracy corresponding to the initial hyperparameters; the test dataset is text data including questions and answers; Use the initial hyperparameters and the initial model accuracy as the current data sequence; Input the current data sequence into the specified optimization algorithm to obtain the algorithm's recommended hyperparameters; After setting the hyperparameters of the target large model to the hyperparameters recommended by the algorithm, the target large model is fine-tuned and trained using the test dataset to determine the current model accuracy of the target large model. The sequence consisting of the hyperparameters recommended by the algorithm and the accuracy of the current model is added to the current data sequence, and the step of inputting the current data sequence into the specified optimization algorithm to obtain the hyperparameters recommended by the algorithm is returned to perform the iterative search until the preset iteration condition is reached. The label data sequence is constructed based on the current data sequence after the preset iteration conditions are met.
2. The intelligence analysis control method as described in claim 1, characterized in that, The specified optimization algorithm is the Bayesian optimization algorithm.
3. The intelligence analysis control method as described in claim 1, characterized in that, The step of determining the target hyperparameters of the target large model through the target sequence model includes: Obtain the historical data sequence and test dataset of the target large model; the historical data sequence consists of historical hyperparameters and historical model accuracy. Input the historical data sequence into the target sequence model to obtain the model's recommended hyperparameters; After setting the hyperparameters of the target large model to the recommended hyperparameters of the model, the target large model is fine-tuned and trained using the test dataset to determine the target model accuracy of the target large model; The sequence consisting of the model's recommended hyperparameters and the target model's accuracy is added to the historical data sequence, and the process of inputting the historical data sequence into the target sequence model to obtain the model's recommended hyperparameters is repeated until the target model's accuracy reaches the accuracy threshold.
4. The intelligence analysis control method as described in claim 3, characterized in that, The target sequence model includes a first multilayer perception mechanism, a second multilayer perception mechanism, and a multilayer attention mechanism; the step of inputting the historical data sequence into the target sequence model to obtain the model's recommended hyperparameters includes: The historical data sequence is encoded into a target vector through the first multi-layer sensing mechanism; The target vector is input into the multi-layer attention mechanism to generate a global aggregate representation; The global aggregation representation is input into the second multilayer perception mechanism to obtain the model's recommended hyperparameters.
5. The control method for intelligence analysis as described in claim 1, characterized in that, Before performing supervised learning training on the initial sequence model using the labeled data sequence, the following steps are included: Obtain the expert-recommended hyperparameters and the test dataset of the target large model; wherein, there are multiple expert-recommended hyperparameters; After setting the hyperparameters of the target large model to the expert-recommended hyperparameters, the target large model is fine-tuned and trained using the test dataset to determine the accuracy of the target model corresponding to the expert-recommended hyperparameters. The label data sequence is constructed based on the expert-recommended hyperparameters and the accuracy of the target model.
6. A control device for intelligence analysis, characterized in that, The device includes: The first acquisition module is used to acquire the pre-built initial sequence model and the target large model; The label data sequence generation module is used to iteratively search the hyperparameter space of the target large model using a specified optimization algorithm, so as to construct a label data sequence based on the hyperparameter space; The supervised learning module is used to perform supervised learning training on the initial sequence model using the labeled data sequence to obtain the target sequence model; The target hyperparameter determination module is used to determine the target hyperparameters of the target large model through the target sequence model, so as to perform intelligence analysis through the target large model; the sequence model is a model based on the Transformer architecture; the target large model is the intelligence analysis module; The second acquisition module is used to acquire the test dataset of the target large model, the preset initial hyperparameters, and the initial model accuracy corresponding to the initial hyperparameters; the test dataset is text data including questions and answers; The current sequence determination module is used to take the initial hyperparameters and the initial model accuracy as the current data sequence; The algorithm recommendation hyperparameter acquisition module is used to input the current data sequence into the specified optimization algorithm to obtain the algorithm recommendation hyperparameters; The first training module is used to set the hyperparameters of the target large model to the hyperparameters recommended by the algorithm, and then perform instruction fine-tuning training on the target large model using the test dataset to determine the current model accuracy of the target large model. The current data sequence update module is used to add the sequence consisting of the algorithm-recommended hyperparameters and the current model accuracy to the current data sequence, and return the step of inputting the current data sequence into the specified optimization algorithm to obtain the algorithm-recommended hyperparameters, so as to perform the iterative search until the preset iteration condition is reached; The first sequence construction module is used to construct the label data sequence based on the current data sequence after the preset iteration conditions are met.
7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the intelligence analysis control method according to any one of claims 1 to 6.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the intelligence analysis control method according to any one of claims 1 to 6.