Hyperparameter optimization method and device based on large language model and electronic equipment
By constructing a hyperparameter search space using a large language model and combining it with Bayesian optimization, the problem of low efficiency in hyperparameter optimization in existing technologies is solved. This achieves efficient and adaptive hyperparameter optimization, improves model performance, and shortens the development cycle. It is applicable to computer vision and natural language processing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI ELECTRICGROUP CORP
- Filing Date
- 2026-04-08
- Publication Date
- 2026-07-03
Smart Images
Figure CN122334547A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the technical field of machine learning model training, specifically relating to a hyperparameter optimization method, device, electronic device, computer-readable medium, and computer program product based on a large language model. Background Technology
[0002] In machine learning and deep learning tasks, model performance is highly dependent on hyperparameter settings, including learning rate, batch size, number of network layers, and regularization coefficients. Traditional hyperparameter tuning methods include grid search, random search, and automated tools based on Bayesian optimization (such as Optuna, Hyperopt, and Ray Tune). While these tools can automatically search for optimal hyperparameter combinations, they still have the following problems: Existing systems require users to manually define the range of values for each hyperparameter, and the reasonable search space setting relies on expert experience. Novice users are prone to setting the range too wide or too narrow, resulting in low search efficiency or missing the optimal solution. Traditional systems cannot dynamically adjust their search strategies based on task type (such as image classification or text generation) and data characteristics (sample size, dimensionality, and category distribution).
[0003] Therefore, a new hyperparameter optimization method is urgently needed to address the shortcomings of the existing technologies. Summary of the Invention
[0004] The main purpose of this application is to provide a hyperparameter optimization method, device, electronic device, computer-readable medium, and computer program product based on a large language model, in order to solve the problems in the prior art where the search space of hyperparameters depends on manual setting and the search results of hyperparameters are difficult to determine the optimal solution.
[0005] To address the aforementioned technical problems, in a first aspect, this application provides a hyperparameter optimization method based on a large language model, comprising: a hyperparameter search space construction step, wherein the large language model receives a target task, data features, and candidate models input by a user; the large language model determines a hyperparameter search space corresponding to the target task, data features, and candidate models, wherein the candidate models are machine learning models distinct from the large language model; and the hyperparameter search space is a set containing the hyperparameters of the candidate models and their constraints; an initial optimization step, wherein the hyperparameter search space is initialized based on an initial optimization method, and prior information, including indicator thresholds, is determined based on the initial search results; a prompt word generation step, wherein the prior information is determined to be prompt word information of the large language model; an optimized hyperparameter generation step, wherein the large language model generates optimized hyperparameters based on context awareness based on the prompt word information; a model indicator determination step, wherein the large language model trains the candidate models based on the optimized hyperparameters to determine the model indicators obtained by the candidate models after training; an indicator judgment step, wherein the model indicator is judged to be less than the indicator threshold; and a hyperparameter generation step, wherein if the model indicator is less than the indicator threshold, the optimized hyperparameters are determined to be the hyperparameters of the candidate models.
[0006] Furthermore, the initial optimization steps specifically include: an initial search step, where the initial optimization method includes Bayesian optimization, which is used to initialize the hyperparameter search space to obtain the initial hyperparameters and the index thresholds of the candidate models after the initial search; and a prior information determination step, which determines the prior information based on the initial hyperparameters, the constraints of the hyperparameters, the data features, and the index thresholds.
[0007] Furthermore, after the indicator judgment step, there is an iterative optimization step: if the model indicator is greater than or equal to the indicator threshold, the optimized hyperparameters and model indicators are updated to the prompt word information of the large language model, and the optimized hyperparameter generation step is executed again.
[0008] Furthermore, candidate models include the decision tree model LightGBM, the deep learning model Transformer with attention mechanism, the convolutional neural network model CNN, and the linear regression model.
[0009] Secondly, this application also provides a hyperparameter optimization device based on a large language model, comprising: a hyperparameter search space construction module, configured to receive a target task, data features, and candidate models input by a user from the large language model; the large language model determining a hyperparameter search space corresponding to the target task, data features, and candidate models; wherein the candidate models are machine learning models distinct from the large language model; and the hyperparameter search space is a set containing the hyperparameters of the candidate models and their constraints; and an initial optimization module, configured to perform an initial search on the hyperparameter search space based on an initial optimization method, and based on the results of the initial search... The system comprises the following modules: a priori information module, which determines prior information including indicator thresholds; a prompt word generation module, configured to determine prompt word information for the large language model; an optimized hyperparameter generation module, configured to generate optimized hyperparameters based on the prompt word information and the large language model's context-aware capabilities; a model indicator determination module, configured to train candidate models based on the optimized hyperparameters and determine the model indicators obtained after training; an indicator judgment module, configured to determine whether the model indicator is greater than the indicator threshold; and a hyperparameter generation module, configured to determine the optimized hyperparameters as the hyperparameters of the candidate model if the model indicator is greater than the indicator threshold.
[0010] Thirdly, this application provides an electronic device including a processor and a memory for storing processor-executable instructions, wherein the processor executes the instructions to implement the steps of the method described in the first aspect.
[0011] Fourthly, this application provides a computer-readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method as described in the first aspect.
[0012] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, implements the steps of the method as described in the first aspect.
[0013] Compared with the prior art, this application has the following advantages: By integrating the initial optimization process of Bayesian optimization with the contextual reasoning capabilities of a large language model, this method achieves efficient and task-adaptive hyperparameter optimization. Unlike traditional hyperparameter optimization methods, this approach supports reasoning by combining historical optimization performance with task-specific structures within the large language model, resulting in more targeted and adaptive hyperparameters. This effectively improves the output of candidate models and the final performance of the models, broadly supports automated hyperparameter tuning in fields such as computer vision and natural language processing, and significantly shortens the model development cycle. Attached Figure Description
[0014] Figure 1This is a flowchart illustrating a hyperparameter optimization method based on a large language model according to an embodiment of this application. Figure 2 This is a flowchart illustrating the initial optimization steps in an embodiment of this application; Figure 3 This is a schematic diagram of a hyperparameter optimization device based on a large language model according to an embodiment of this application. Detailed Implementation
[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. However, the embodiments described below are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application. Unless obvious from the context or otherwise, the same reference numerals in the figures represent the same structures or operations.
[0016] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.
[0017] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously. Furthermore, other operations may be added to these processes, or one or more steps may be removed from these processes.
[0018] Please refer to Figure 1 The first aspect of the embodiments of this application provides a hyperparameter optimization method based on a large language model, including: S1. Hyperparameter search space construction steps: The large language model receives the target task, data features and candidate models input by the user. The large language model determines the hyperparameter search space corresponding to the target task, data features and candidate models. Among them, the candidate model is a machine learning model that is different from the large language model. The hyperparameter search space is a set that includes the hyperparameters of the candidate models and the constraints of the hyperparameters. S2. Initial optimization step: Based on the initial optimization method, the hyperparameter search space is initialized and searched. Based on the results of the initial search, the prior information is determined, including the index threshold. S3, Prompt word generation step: Determine the prior information as prompt word information of the large language model; S4. Optimize the hyperparameter generation step: Based on prompt word information, the large language model generates optimized hyperparameters based on context awareness capabilities. S5. Model index determination steps: The large language model trains candidate models based on optimized hyperparameters to determine the model indexes obtained after training the candidate models. S6. Indicator Judgment Steps: Determine whether the model indicator is greater than the indicator threshold. S7. Hyperparameter generation step: If the model index is greater than the index threshold, determine the optimized hyperparameters as the hyperparameters of the candidate model.
[0019] This application generates a hyperparameter search space based on candidate models, data features, and target tasks through a Large Language Model (LLM). The hyperparameters are searched through initial optimization to determine the prior information of the hyperparameters. This prior information is used as the corpus of prompt words for the Large Language Model. Then, the context awareness capability of the Large Language Model is used to automatically generate optimized hyperparameters that meet the requirements of the candidate models and perform automatic verification. This can effectively improve the training results and final performance of the candidate models.
[0020] In some specific embodiments, the large language model used in this application may include Qwen, ChatGPT, CodeLlama, Claude, etc. The candidate model in this application may be one or more of the following: decision tree model LightGBM, deep learning model with attention mechanism Transformer, convolutional neural network model CNN, and linear regression model, depending on the user's specific functional requirements.
[0021] Users input the target task, data features, and candidate models into the Large Language Model (LLM). The LLM maps these elements into a hyperparameter search space S, which is a set containing the hyperparameters of the candidate models and their constraints. S=L(T,D,M)={hp i ~ U(a i b i )|i=1,2,3...k} Among them, hp i Let U(a) represent the i-th hyperparameter. i b i ) represents the constraint condition of the hyperparameter, which may include the range of values of the hyperparameter and the constraint function.
[0022] Data features include the sample size, feature dimensions, and feature types of the dataset: Sample size refers to the total number of data samples; Feature dimension refers to the length of the vectors included in the data. For example, a two-dimensional vector [x1, x2] has two feature dimensions, while a three-dimensional vector [x1, x2, x3] has three feature dimensions. Generally speaking, the feature dimensions need to match the sample size; Feature types include numerical features, categorical features, text features, image features, and time features. Taking numerical features as an example, they can be further subdivided into continuous features and discrete features.
[0023] Large Language Models (LLMs) can determine the range of hyperparameter values based on data features. Specifically, first, sample classification rules are defined. For example, the sample size can include three levels: small samples (N < 1000), medium samples (1000 ≤ N ≤ 10000), and large samples (N > 10000). Second, the approximate range of preset values is determined based on the sample size. For example, if the sample size N = 800 corresponds to small samples, the learning rate can be set to a range of (10...). -5 10 -2 ).
[0024] To avoid invalid combinations, large language models (LLMs) can further define constraint functions c based on data features to ensure the validity of the candidate model's dataset. For example, for the deep learning model Transformer, the constraint function c needs to be defined as "d_model mod n_heads=0", which means that the remainder of the model's feature dimension (d_model) divided by the number of attention heads (n_heads) must be equal to 0.
[0025] This setting ensures that the constraints of the hyperparameters match the target task, data features, and candidate models, thereby guaranteeing the generalization ability of the large language model to the hyperparameters and ensuring the effectiveness of data fitting during model training.
[0026] Please refer to Figure 2 Step S2, “Initial Optimization Steps,” specifically includes: S21. Initial search step: The initial optimization method includes Bayesian optimization. Bayesian optimization is used to initialize the hyperparameter search space to obtain the initial hyperparameters and the index thresholds of the candidate models after the initial search. S22, Prior information determination step: Based on the initial hyperparameters, the constraints of the hyperparameters, the data characteristics, and the index thresholds, determine the prior information.
[0027] In some specific embodiments, Bayesian optimization is first used to initialize the hyperparameter search space to obtain initial hyperparameters, and at the same time, the index threshold of the candidate model (such as the root mean square error RMSE in a linear regression model) is also obtained. It can be understood that the initial hyperparameters are the best-performing results obtained based on Bayesian optimization. Using the initial hyperparameters to train the candidate model, the prediction error of the candidate model should be minimized, that is, the corresponding index threshold should be minimized.
[0028] Therefore, by combining the aforementioned initial hyperparameters, corresponding index thresholds, hyperparameter constraints, and data features, we can determine the prior information of the large language model. This prior information serves as the "domain meta-knowledge" for the large language model to understand the candidate model. By defining this prior information as the prompt word information of the large language model, we can generate optimized hyperparameters based on this prior information, thereby further reducing the prediction error of the candidate model, based on the context awareness capability of the large language model.
[0029] Please continue to refer to Figure 1 Step S6, "Indicator Judgment Step," further includes: S8. Iterative optimization step: If the model index is greater than or equal to the index threshold, update the optimized hyperparameters and model index to the prompt word information of the large language model, and execute the optimized hyperparameter generation step again.
[0030] In some specific embodiments, after the large language model trains the candidate model based on optimized hyperparameters, it determines the model metric (e.g., root mean square error (RMSE) in a linear regression model) obtained after training and judges the relationship between the model metric and the metric threshold. If the model metric is less than the metric threshold, it means that the accuracy of the optimized hyperparameters generated by the large language model is better than the accuracy of the initial hyperparameters, and the optimized hyperparameter is used as the final selected hyperparameter for the candidate model. If the model metric is greater than or equal to the metric threshold, it means that the accuracy of the optimized hyperparameter still needs to be improved, and the optimized hyperparameter and model metric are updated to the prompt word information of the large language model, and the optimized hyperparameter generation step is executed again. This iterative cycle continues until the optimized hyperparameter is less than the metric threshold.
[0031] This application achieves efficient and task-adaptive hyperparameter optimization by integrating the initial optimization process of Bayesian optimization with the contextual reasoning capabilities of a large language model. Unlike traditional hyperparameter optimization methods, this method supports reasoning by combining historical optimization performance with task-specific structures within the large language model, resulting in more targeted and adaptive hyperparameters. This effectively improves the output of candidate models and the final performance of the models, widely supports automated hyperparameter tuning in fields such as computer vision and natural language processing, and significantly shortens the model development cycle.
[0032] Please see Figure 3 The second aspect of this application provides a hyperparameter optimization apparatus, comprising: a hyperparameter search space construction module, configured to receive a target task, data features, and candidate models input by a user from a large language model; the large language model determining a hyperparameter search space corresponding to the target task, data features, and candidate models; wherein the candidate models are machine learning models distinct from the large language model; and the hyperparameter search space is a set containing the hyperparameters of the candidate models and their constraints; and an initial optimization module, configured to perform an initial search on the hyperparameter search space based on an initial optimization method, and, based on the result of the initial search, determine... The system comprises the following modules: a prior information module, which defines a threshold value for a metric; a prompt word generation module, configured to determine the prompt word information for the large language model; an optimization hyperparameter generation module, configured to generate optimization hyperparameters based on the prompt word information and the large language model's context-aware capabilities; a model metric determination module, configured to train candidate models using the optimization hyperparameters and determine the model metric obtained after training; a metric judgment module, configured to determine whether the model metric is greater than the metric threshold; and a hyperparameter generation module, configured to determine the optimization hyperparameters as the hyperparameters of the candidate model if the model metric is greater than the metric threshold.
[0033] For details of other operations performed by each module in this embodiment, please refer to the foregoing embodiments, which will not be elaborated here.
[0034] The hyperparameter optimization device in this application embodiment can be a device, or it can be a component, integrated circuit, or chip in a terminal. The hyperparameter optimization device in this application embodiment can be a chip, including FPGA (Field Programmable Gate Array), MCU (Microcontroller Unit), etc., but this application embodiment does not specifically limit the specific implementation.
[0035] The embodiments of this application also provide an electronic device, including a processor and a memory for storing processor-executable instructions. When the processor executes the instructions, it implements the various processes of the embodiments of the hyperparameter optimization method based on the large language model described above, and can achieve the same technical effect. To avoid repetition, it will not be described in detail here.
[0036] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.
[0037] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the hyperparameter optimization method based on a large language model in the embodiments of this application. The processor executes various functional applications and data classification by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the hyperparameter optimization method based on a large language model in the above method embodiments.
[0038] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0039] One or more modules are stored in memory and, when executed by a processor, execute the hyperparameter optimization method based on a large language model in the implementation example.
[0040] The embodiments of this application also provide a computer-readable medium storing a program or instructions that, when executed by a processor, implement the various processes of the above-described embodiments of the hyperparameter optimization method based on a large language model and achieve the same technical effect. To avoid repetition, further details are omitted here.
[0041] Embodiments of this application also provide at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes a computer program that, when executed by a processor, implements the various processes of the above-described embodiments of the hyperparameter optimization method based on a large language model and achieves the same technical effect. To avoid repetition, further details are omitted here.
[0042] Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus.
[0043] When the hyperparameter optimization method based on a large language model of this application is implemented as a computer program, it can also be stored as an article of manufacture in a computer-readable storage medium. For example, computer-readable storage media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical discs (e.g., compact discs (CDs), digital multifunction discs (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memory (EEPROM), cards, sticks, key drives). Furthermore, the various storage media described herein can represent one or more devices and / or other machine-readable media used for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media (and / or storage media) capable of storing, containing, and / or carrying code and / or instructions and / or data.
[0044] For those skilled in the art, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application and therefore remain within the spirit and scope of the exemplary embodiments of this application.
[0045] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of scope in some embodiments of this application are approximate values, in specific embodiments, such values are set as precisely as feasible.
[0046] Although this application has been described with reference to specific embodiments, those skilled in the art should recognize that the above embodiments are only used to illustrate this application, and various equivalent changes or substitutions can be made without departing from the spirit of this application. Therefore, any changes or modifications to the above embodiments within the scope of the essential spirit of this application will fall within the scope of this application.
Claims
1. A large language model-based hyperparameter optimization method, characterized in that, include: The hyperparameter search space construction step involves a large language model receiving a target task, data features, and candidate models input by a user. The large language model then determines a hyperparameter search space corresponding to the target task, data features, and candidate models. The candidate models are machine learning models distinct from the large language model, and the hyperparameter search space is a set containing the hyperparameters of the candidate models and the constraints of the hyperparameters. The initial optimization step involves performing an initial search on the hyperparameter search space based on the initial optimization method, and determining prior information based on the results of the initial search, including index thresholds. The prompt word generation step determines that the prior information is the prompt word information of the large language model; The hyperparameter generation step is optimized by generating optimized hyperparameters based on the prompt word information and the context-aware capability of the large language model. The model metric determination step involves training the candidate model based on the optimized hyperparameters using the large language model, and determining the model metric obtained after training the candidate model. The indicator judgment step involves determining whether the model indicator is less than the indicator threshold. In the hyperparameter generation step, if the model index is less than the index threshold, the optimized hyperparameter is determined as the hyperparameter of the candidate model.
2. The large language model based hyperparameter optimization method according to claim 1, characterized in that, The initial optimization steps specifically include: The initial search step includes an initial optimization method comprising Bayesian optimization, which is used to initialize the hyperparameter search space to obtain the initial hyperparameters and the index threshold of the candidate model after the initial search. The prior information determination step determines the prior information based on the initial hyperparameters, the constraints of the hyperparameters, the data features, and the index thresholds. 3.The method of claim 1, wherein, Following the indicator judgment step, the method further includes: In the iterative optimization step, if the model index is greater than or equal to the index threshold, the optimization hyperparameters and the model index are updated to the prompt word information of the large language model, and the optimization hyperparameter generation step is executed again.
4. The hyperparameter optimization method based on a large language model according to any one of claims 1-3, characterized in that, The candidate models include the decision tree model LightGBM, the deep learning model Transformer with attention mechanism, the convolutional neural network model CNN, and the linear regression model. 5.A device for hyperparameter optimization based on a large language model, characterized in that, include: The hyperparameter search space construction module is configured to receive a target task, data features, and candidate models from a user input by a large language model. The large language model determines the hyperparameter search space corresponding to the target task, the data features, and the candidate models. The candidate models are machine learning models that are different from the large language model. The hyperparameter search space is a set that includes the hyperparameters of the candidate models and the constraints of the hyperparameters. The initial optimization module is configured to perform an initial search on the hyperparameter search space based on the initial optimization method, and determine prior information based on the results of the initial search, wherein the prior information includes index thresholds. The prompt word generation module is configured to determine that the prior information is the prompt word information of the large language model; The optimized hyperparameter generation module is configured to generate optimized hyperparameters based on the prompt word information, and the large language model generates optimized hyperparameters based on context awareness capabilities. The model metric determination module is configured to train the candidate model based on the optimized hyperparameters of the large language model, and determine the model metric obtained by the candidate model after training. The indicator judgment module is configured to determine whether the model indicator is greater than the indicator threshold. The hyperparameter generation module is configured to determine the optimized hyperparameter as the hyperparameter of the candidate model if the model index is greater than the index threshold.
6. An electronic device, comprising: include: A processor and a memory for storing processor-executable instructions, wherein the processor, when executing the instructions, implements the steps of the method according to any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program or instructions that, when executed by a processor, implement the method as described in any one of claims 1-4.
8. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-4.