A general search method for training hyperparameters powered by large language models

By refactoring the training hyperparameter search task into a candidate hyperparameter ranking problem, and utilizing large language models (LLM) to learn the mapping relationship between hyperparameter encoding and performance metrics, this approach addresses the lack of flexibility of large language models in training hyperparameter search tasks in existing technologies, and achieves cross-task generalization and efficient automated hyperparameter search.

CN123117262APending Publication Date: 2026-11-17THE CHINESE UNIVERSITY OF HONG KONG
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510628332.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-15
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing large language models (LLMs) lack flexibility and adaptability in training hyperparameter search tasks, struggle to generalize across tasks, and require extensive manual suggestion engineering, limiting their application in new search spaces and datasets.

Method used

The training hyperparameter search task is restructured into a ranking problem of candidate hyperparameters. The mapping relationship between hyperparameter encoding and performance metrics is learned through a large language model (LLM). Fine-tuning is performed using a NAS benchmark dataset, historical hyperparameter-performance pairs are dynamically updated, and the hyperparameter with the best performance is output through an iterative search strategy, avoiding reliance on prior knowledge of the search space and network topology.

Benefits of technology

It achieves cross-task generalization of large language models in zero-shot scenarios, breaks through the dependence on specific search spaces and datasets, improves search efficiency and robustness, and demonstrates efficient, general and scalable automated hyperparameter search capabilities in complex optimization scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117262A_ABST
    Figure CN123117262A_ABST
Patent Text Reader

Abstract

A general hyperparameter search method for training supported by a large language model includes: S1, reconstructing the training hyperparameter search task into a ranking problem of candidate hyperparameters, and obtaining performance data containing historical hyperparameters and text encodings of candidate hyperparameters; S2, sampling from a local region of the target search space to generate training data, wherein the training data contains multiple historical hyperparameter-performance pairs and corresponding candidate hyperparameter sets; S3, using the training data to fine-tune the LLM, enabling it to learn the mapping relationship between hyperparameter encodings and performance metrics; S4, in zero-shot scenarios, dynamically updating historical hyperparameter-performance pairs through an iterative search strategy, and ranking candidate hyperparameters based on the fine-tuned LLM, outputting the hyperparameter with the best performance. The training hyperparameter search method of this invention can achieve cross-task generalization in zero-shot scenarios, and can be directly applied to diverse visual tasks without target domain fine-tuning, demonstrating efficient, general, and scalable automated hyperparameter search capabilities.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to artificial intelligence and machine learning technologies, and in particular to a general search method for training hyperparameters supported by a large language model. Background Technology

[0002] Recent advances in large language models (LLMs) [29,38] have significantly improved their capabilities across a wide range of applications, including instruction following

[37] , creative writing, and style control. Furthermore, advanced LLMs have demonstrated superior performance in more challenging tasks requiring strong reasoning capabilities, such as solving challenging mathematical or coding problems [13,43,45].

[0003] Large language models (LLMs) have been pre-trained on large datasets and have shown excellent performance in learning scenarios with a small number of samples [2]. The field has witnessed the rapid rise of LLM development and has made significant contributions, including models such as LLaMA

[39] , PaLM [6], and GPT-4

[29] . Subsequent research has focused on fine-tuning these models for specialized or synthetic datasets, especially open-source models such as LLaMA. As a result, significant performance improvements have been achieved in areas such as code generation

[33] and mathematical reasoning [11,42].

[0004] However, despite these advances, challenging problems such as training hyperparameter search, which require complex optimization, have largely failed to be solved by LLM. Training hyperparameter search faces unique challenges, requiring not only a deep understanding of neural network topology but also the ability to optimize hyperparameters based on specific performance metrics or delay constraints. While traditional search methods (including reinforcement learning (RL)

[51] and evolutionary algorithm (EA)

[31] ) have achieved great success, they are often inflexible and adaptable, requiring significant redesign or retraining when applied to new search spaces or target datasets. Although current LLMs have impressive reasoning capabilities, their application to training hyperparameter search problems is limited to specific search spaces and datasets. Jawahar et al.

[18] used GPT-4

[29] to estimate the performance of different hyperparameters and used these estimates to guide the initialization of hyperparameter search. Similarly, EvoPrompting [3] and LLMatic

[28] used LLMs to perform hyperparameter search at the code level. Other studies (such as Song et al.

[34] and Liu et al.

[26] ) have explored the potential of LLM for black-box optimization in the context of hyperparameter search. GENIUS

[48] used GPT-4 to provide model hyperparameter suggestions during the search phase. However, these methods are generally limited by some constraints, such as the need for a lot of manual hint engineering, making them difficult to transfer to other tasks.

[0005] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a general search method for training hyperparameters supported by a large language model.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A general search method for training hyperparameters supported by a large language model includes the following steps:

[0009] S1. Reconstruct the training hyperparameter search task into a ranking problem of candidate hyperparameters, obtain performance data including historical hyperparameters and text encoding of candidate hyperparameters, and then sort the performance of candidate hyperparameters using the Large Language Model (LLM).

[0010] S2. Sample training data from a local region of the target search space, wherein the training data includes multiple historical hyperparameter-performance pairs and corresponding candidate hyperparameter sets;

[0011] S3. Use the training data to fine-tune the LLM so that it learns the mapping relationship between hyperparameter encoding and performance metrics;

[0012] S4. In the zero-sample scenario, the historical hyperparameter-performance pairs are dynamically updated through an iterative search strategy, and the candidate hyperparameters are ranked based on the fine-tuned LLM to output the hyperparameter with the best performance.

[0013] Furthermore, in step S1, the text encoding of the candidate hyperparameter is a numerical code generated based on search space options or topology index. The numerical code is used to uniquely identify hyperparameter features and capture core network features, avoiding reliance on prior knowledge of the search space and network topology.

[0014] Furthermore, in step S1, the ranking task is achieved by transforming hyperparameter generation into a classification problem, and LLM identifies the optimal hyperparameter based on the candidate hyperparameters represented by numerical codes.

[0015] Furthermore, in step S2, the local region of the target search space is generated by fixing some hyperparameters and the sampling area is reduced by setting the option to discard probability.

[0016] Furthermore, in step S2, the training data originates from a NAS benchmark dataset containing diverse hyperparameter configurations. Each sample includes historical hyperparameter-performance pairs sampled from the same local region and a candidate hyperparameter set, and the candidate set contains at least the hyperparameter with the best performance in the current local region.

[0017] Further, in step S4, the iterative search strategy includes: randomly sampling and evaluating a fixed number of hyperparameters at the start of the search to establish initial historical data; randomly sampling a fixed number of candidate hyperparameters from the search space in each iteration, and inputting them along with the current historical data into the fine-tuned LLM to predict the optimal hyperparameters; evaluating the performance of the selected hyperparameters and updating the historical data, while dynamically truncating the amount of historical data and retaining some hyperparameter-performance pairs in the "history" to adapt to the context window constraints of the LLM; and outputting the hyperparameter with the highest performance after the iteration continues for a set number of times.

[0018] Furthermore, in step S4, the performance evaluation of the candidate hyperparameters is based on a fast verification metric for the proxy task, which includes model accuracy, computational efficiency, or task-specific score.

[0019] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the general search method for training hyperparameters supported by the large language model.

[0020] A computer program product includes a computer program that, when executed by a processor, implements the general search method for training hyperparameters supported by the large language model.

[0021] A training hyperparameter search device based on a large language model, comprising:

[0022] Memory, used to store computer programs;

[0023] A processor is used to implement the general search method for training hyperparameters supported by the large language model when executing the computer program.

[0024] The present invention has the following beneficial effects:

[0025] This invention proposes a general hyperparameter search method for training supported by a large language model (LLM). By reconstructing the training hyperparameter search task into a ranking problem of candidate hyperparameters, and utilizing compact text encoding (such as numerical codes for options or topology indices) to represent the core features of network hyperparameters, it avoids the complex hyperparameter generation and explicit topological description in traditional methods, significantly reducing the difficulty of processing high-density information by large language models (LLMs). Based on the fine-tuned LLM, it achieves cross-task generalization in zero-shot scenarios, and can be directly applied to diverse tasks such as image classification, segmentation, and generation without target domain fine-tuning, breaking through the dependence of traditional training hyperparameter search methods on specific search spaces and datasets. Furthermore, performance evaluation can be accelerated by quickly verifying metrics (such as CLIP text-image alignment scores) through surrogate tasks. Finally, with reduced human intervention, this invention achieves efficient, general, and scalable automated hyperparameter search, with performance surpassing traditional reinforcement learning, evolutionary algorithms, and methods that directly apply large language models such as GPT-4o, demonstrating robustness in complex optimization scenarios and industrial-grade deployment potential. Furthermore, this embodiment of the invention generates hyperparameter-performance pairs containing subtle performance differences by dynamically sampling training data from the local search space, fixing some configurations and setting option dropout probabilities, and dynamically updating historical data in conjunction with an iterative search strategy and truncating the LLM context window constraints to effectively improve the model's sensitivity to performance differences and search efficiency.

[0026] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0027] Figure 1 This is an example diagram of hyperparameter encoding based on local search space and LLM training samples in an embodiment of the present invention.

[0028] Figure 2 The overall flowchart shows the general search method for training hyperparameters supported by the large language model of this invention. Detailed Implementation

[0029] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0030] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0031] While large language models (LLMs) have demonstrated impressive performance across various tasks, their application in complex search problems such as training hyperparameter search remains limited. Directly applying advanced LLMs (such as GPT-4o) to hyperparameter search benchmarks yields poor results, highlighting the challenge of directly optimizing training hyperparameters using LLMs. This invention aims to leverage the potential of LLMs as a general-purpose training hyperparameter search model that requires no fine-tuning. This invention proposes a general-purpose training hyperparameter search method powered by large language models, called LM-Searcher. This novel approach cleverly utilizes LLMs for training hyperparameter search by reformulating hyperparameter search as a ranking problem, thereby allowing tasks to better leverage the advantages of LLMs. This invention further enhances the functionality of LM-Searcher by fine-tuning datasets selected from NAS benchmark datasets (such as NAS-Bench-301), improving LLM's ability to discern finer performance differences. The results show that LM-Searcher, trained solely on NAS-Bench-301, achieves strong performance on multiple real-world vision tasks without requiring fine-tuning of the target hyperparameters or benchmarks, and demonstrates excellent flexibility and generalizability across various search spaces and datasets.

[0032] Compared to previous methods that typically rely on knowledge distillation

[15] , requiring a larger model to transfer knowledge to a smaller model, the method of this invention can achieve powerful zero-shot generalization in different training hyperparameter search tasks, thus setting a new benchmark for LLM performance in this field. The method of this invention regards LLM as a general search model for training hyperparameter search, and learns the relationship between network hyperparameters and corresponding performance indicators by training it, thereby achieving zero-shot generalization without needing to make cue adjustments for specific tasks.

[0033] See Figure 2 This invention provides a method for searching training hyperparameters based on large language models (LLM), comprising the following steps:

[0034] Step S1, Task Restructuring: The training hyperparameter search task is transformed into a candidate hyperparameter ranking task, where each candidate hyperparameter represents its feature configuration through a compact text code; LLM will rank a set of candidate hyperparameters based on historically sampled hyperparameters and their performance, rather than directly generating hyperparameter configurations.

[0035] Step S2, Data Generation: Locally sample hyperparameters from the search space of a proxy-based NAS benchmark (such as NAS-Bench-301) to generate a training dataset; each training sample contains two parts: a set of historically evaluated hyperparameters and their performance metrics, i.e., historical hyperparameter-performance pairs, used to provide historical information for LLM; and a set of candidate hyperparameters whose performance has not yet been evaluated; LLM will predict the performance of these candidate hyperparameters based on historical information and select the hyperparameter most likely to perform best.

[0036] Step S3, Model Training: Fine-tune the LLM based on the training dataset to enable it to learn the mapping relationship between hyperparameter code and performance; through fine-tuning, the LLM can better understand and predict hyperparameter performance, thus exhibiting strong generalization ability on new tasks.

[0037] Step S4, Zero-Shot Search: Input the current historical hyperparameter-performance pairs and candidate hyperparameter sets into the LLM, causing the LLM to output the ranking results of the candidate hyperparameters, and select the optimal hyperparameter based on the ranking results; add the selected optimal hyperparameter and its performance to the historical dataset, and repeat the above steps iteratively until the preset search termination condition is reached. Finally, report the hyperparameter with the highest performance observed throughout the entire search process as the final result.

[0038] In some embodiments, in step S1, when the training hyperparameter search task is reconstructed into a ranking problem of candidate hyperparameters, the text encoding of the candidate hyperparameters is a numerical code generated based on search space options or topology indexes, used to uniquely identify hyperparameter features and avoid reliance on prior knowledge of the search space and network topology; the obtained text encoding is a sampled hyperparameter encoded in compact text code form, and the Large Language Model (LLM) ranks the performance of candidate hyperparameters based on these encodings.

[0039] The process of reconstructing the training hyperparameter search task into a ranking problem of candidate hyperparameters specifically includes transforming the hyperparameter generation task into a classification problem, providing the LLM with a set of randomly sampled hyperparameters represented by numerical codes, so that the task of the LLM is to identify the hyperparameters that are expected to produce the best performance.

[0040] In some embodiments, in step S2, when generating training data by sampling from a local region of the target search space, the sampling region is narrowed by setting a dropout probability for each option in the search space. This allows the collected training data to better reflect the subtle relationships between hyperparameter performance, thereby improving the quality of the training data. The local region of the target search space is generated by fixing some hyperparameter configurations to narrow the search range and enhance subtle performance differences between candidate hyperparameters. The training data originates from a NAS benchmark dataset containing diverse hyperparameter configurations, and hyperparameter pairs with subtle performance differences are generated through a local sampling strategy. As an example, the target search space may include the NAS benchmark NAS-Bench-301.

[0041] Each sample in the training data includes: historical hyperparameter-performance pairs sampled from the same local region; and a candidate hyperparameter set, which contains at least the hyperparameter with the best performance in the current local region.

[0042] In some embodiments, in step S3, each sample in the training dataset contains multiple randomly selected hyperparameter-performance pairs. These pairs are randomly divided into two subsets: "historical" hyperparameter-performance pairs and "candidate" hyperparameters. The hyperparameter with the highest performance in the original hyperparameter-performance pairs is located in the "candidate" set. Based on this, LLM is trained.

[0043] In some embodiments, in step S4, when dynamically updating historical hyperparameter-performance pairs through an iterative search strategy, a fixed number of candidate hyperparameters are randomly sampled from the search space in each iteration, and these are provided to the fine-tuned LLM along with the previously evaluated hyperparameter-performance pairs in the current "history" to predict the best performance hyperparameters. To adapt to the limited context window of the LLM, the number of historical hyperparameter-performance pairs is dynamically truncated in the iterative search strategy to fit the context window limitation of the LLM, retaining only a certain number of hyperparameter-performance pairs in the "history". The iteration continues until the set number of iterations is completed, and the hyperparameter with the highest performance during the search process is output.

[0044] In some embodiments, the performance evaluation of the candidate hyperparameters is based on rapid validation metrics for the proxy task, including model accuracy, computational efficiency, or task-specific scores. The task-specific scores include text-image alignment scores or concept retention scores.

[0045] The principles, features, and experimental verification of specific embodiments of the present invention are further described below.

[0046] This invention addresses a challenging problem: leveraging the inherent reasoning capabilities of LLMs to construct a general search model for hyperparameter search that can adapt to different search spaces and datasets without fine-tuning. A pilot study was first conducted using an off-the-shelf LLM (GPT-4o) on an existing hyperparameter search task. Following the iterative approach used in previous RL-based methods, GPT-4o was prompted to improve hyperparameter configurations based on historically sampled hyperparameters and their performance. The search terminated after a fixed number of samples, and the best-performing hyperparameters were reported. The inventors observed that GPT-4o effectively identifies better hyperparameters in simple search spaces. However, when applied to larger and more complex search spaces, it fails to improve upon random search methods.

[0047] There are two main limitations to using LLM for hyperparameter search. One limitation is that LLMs tend to focus only on the information provided in the "utilize" hints rather than actively "exploring" a broader search space, which limits their ability to discover diverse or novel solutions. The other limitation is that LLMs are primarily trained on natural language corpora, making it difficult to effectively handle large amounts of high-density information, thus inhibiting their performance in complex or data-intensive tasks.

[0048] To address these issues, this invention proposes LM-Searcher, a novel approach that leverages LLM to perform hyperparameter search by reformulating the problem, thereby better utilizing the advantages of LLM. This invention does not directly suggest that LM-Searcher generate specific hyperparameter configurations through code or language, as pilot studies have shown this to be challenging. Therefore, the practical approach requires LM-Searcher to rank a set of candidate hyperparameters based on historically sampled hyperparameters and their performance. In other words, the hyperparameter search problem is transformed into a ranking task, where LM-Searcher receives some sampled hyperparameters encoded in compact text code and needs to identify the best-performing hyperparameters solely based on these encodings. The proposed method utilizes the capabilities of LLM to establish the relationship between network hyperparameters (represented in compact text code) and their performance, thus avoiding the need for explicit encoding of detailed hyperparameters and the search space.

[0049] To further enhance the ability of general LLM to solve complex search problems, a large dataset was selected from existing hyperparameter search benchmarks to fine-tune LLM (e.g., LLaMA). Specifically, a dataset containing 200,000 training samples was formed using the massive amount of data available in NAS-Bench-301

[47] (a comprehensive benchmark for agent-based NAS), with each sample containing a set of candidate hyperparameters and corresponding performance metrics. However, the inventors found that simply sampling random hyperparameters from NAS-Bench-301 was insufficient, as the performance differences between these hyperparameters could be too large and too obvious for the model to learn effectively. Instead, they focused on generating training examples from the local search space while keeping some hyperparameter configurations fixed, thereby reducing the challenge of the learning task and requiring the model to learn subtle performance differences. By focusing on these subtle changes, the model's ability to discern subtle performance differences was greatly enhanced.

[0050] By training on this carefully selected dataset, the LM-Searcher of this invention learns how to rank hyperparameters, effectively becoming a powerful and general-purpose training hyperparameter estimator for new tasks. It is demonstrated that LM-Searcher trained solely on NAS-Bench-301 achieves superior performance in image classification and exhibits excellent generalization ability in image segmentation and image generation independent of the original NAS-Bench-301, surpassing the performance of directly applying GPT-4o and even exceeding mature traditional NAS methods. Importantly, unlike previous methods which typically require specialized optimization or fine-tuning of the target search space and dataset, LM-Searcher can be directly applied to various hyperparameters, search spaces, datasets, and tasks without requiring specialized customization, thus demonstrating strong flexibility and generalization ability.

[0051] While large language models (LLMs) have demonstrated powerful reasoning capabilities in mathematical or coding problems, the inventors found that directly applying them to training hyperparameter search tasks failed to match traditional methods due to the inherent complexity of the hyperparameter search problem. In fact, GPT-4o, in the pilot study of this invention, even performed worse than random search. The LM-Searcher of this invention better utilizes the reasoning capabilities of LLMs to solve hyperparameter search problems, exhibiting strong generalization search capabilities across various NAS tasks.

[0052] Regarding the proposed LM-Searcher framework, the following details the task formulation and dataset used to train the general LM-Searcher. First, the task formulation is defined to better adapt LLM to NAS problems. Then, how to collect a large dataset to train the LM-Searcher is introduced. Finally, how to apply the LM-Searcher to different NAS tasks with zero-shot capability is presented.

[0053] Task Representation Formula

[0054] NAS (Neural Architecture Search) involves finding the optimal training hyperparameters from a predefined search space, aiming to maximize performance for a specific task while satisfying constraints such as latency and model size. Formally, given a search space S, the goal is to determine the hyperparameters a* that maximize performance:

[0055]

[0056] Where P(a) represents the target performance metric (e.g., accuracy or throughput).

[0057] While traditional methods

[51] have achieved success in NAS, they often lack adaptability to new search spaces and datasets, requiring retraining when applied to new NAS tasks. With the advent of LMMs, which possess strong zero-shot and inference capabilities, recent research [28,48] has attempted to leverage LMMs to address NAS problems, primarily involving prompting LMMs to iteratively output better hyperparameter configurations. Typically, the LMM receives manually designed prompts that define its role and specify the search space as programming code (e.g., PyTorch code). Based on these prompts, the LMM samples a new hyperparameter for sampling and evaluation. However, these works require significant expertise and manual prompting adjustments for different search tasks, limiting their widespread application.

[0058] To enhance the hyperparameter search process, the method of this invention utilizes the capabilities of an LLM in a search space-independent manner. This method differs significantly from previous methods, which required detailed definition of the search space and explicit hyperparameter configurations (e.g., in the form of PyTorch code). The method of this invention simplifies the input provided to the LLM by representing each hyperparameter solely as hyperparameter word codes (i.e., options or topology indexes), thereby capturing the core characteristics of the network while avoiding reliance on prior knowledge of the search space and network topology.

[0059] Taking the MNASNet

[36] search space as an example, each hyperparameter is compiled into a series of indices. Each index in this sequence represents a specific choice of a cell configuration, such as Figure 1As shown on the left. Figure 1 Examples of training samples are shown. This is illustrated using a simplified version of the DARTS

[24] search space. Hyperparameter-performance pairs are sampled from the local search space. Searchable features are encoded as numerical codes, while fixed features are discarded (shown in gray). LLM needs to reason and learn from the “history”, rank the “candidates”, and predict which “candidate” hyperparameter is likely to perform best.

[0060] Furthermore, this invention simplifies the hyperparameter generation task by transforming it into a classification problem. Specifically, instead of directly requesting the LLM to generate hyperparameter configurations, it provides the LLM with a set of randomly sampled hyperparameters represented by numerical codes. The LLM's task is to identify the hyperparameters that are expected to produce optimal performance. This method offers flexibility and better adaptability to different NAS tasks because it minimizes the need for training specific hyperparameters and cumbersome hint engineering.

[0061] LM-Searcher training

[0062] While such task reconstruction avoids limiting trained models to specific hyperparameters and enhances the generalization ability of LLMs to NAS, the inventors observed that directly applying off-the-shelf LLMs still lags behind advanced specialized NAS methods. A key reason for this performance gap is that existing LLMs are not explicitly trained to find optimal solutions from historical information, thus limiting their effectiveness in this domain.

[0063] To address this limitation, a new dataset containing 200,000 training samples was selected to fine-tune the LLaMA model to the LM-Searcher model of this invention. Specifically, this invention collected a large dataset by sampling from the existing NAS benchmark NAS-Bench-301, containing a large number of hyperparameter-performance pairs. Each sample in this dataset contains n (i.e., 100-200) randomly selected such pairs, which are then randomly split into two subsets: n-10 "historical" hyperparameter-performance pairs, and 10 "candidate" hyperparameters without performance data. Figure 1 (Right) illustrates a training sample. This [history, candidate] format is designed to enable the model to reason and learn from "historical" data and rank "candidate" hyperparameters based on the learned reasoning ability. Importantly, it ensures that the highest-performing hyperparameter among the original n pairs is within the "candidate" set. The LLM is then trained based on the provided "history" hyperparameter-performance pairs to predict which "candidate" hyperparameter is likely to achieve the best performance.

[0064] The hyperparameter search space in NAS-Bench-301 is enormous, with a maximum possible configuration of up to 10. 18 Therefore, designing an effective sampling strategy to generate representative training data is crucial. The inventors observed that performance differences between hyperparameters randomly sampled from the entire search space are often too significant for LLM to effectively learn meaningful relationships. To address this issue, this invention proposes a novel local sampling strategy involving sampling n hyperparameter-performance pairs from a local region of the search space (rather than the entire design space). Figure 1 As shown on the left, in the NAS-Bench-301 search space, each option is set with a 50% probability of being discarded. This narrows the sampling area and highlights more subtle performance differences in the collected data. This local sampling strategy improves the quality of the training dataset and also enables the LLM to learn the subtle relationship between hyperparameter configurations and their corresponding performance, thereby enhancing the hyperparameter selection capability of the LLM.

[0065] Zero-shot hyperparameter search using LM-Searcher

[0066] We apply LM-Search to different hyperparameter search tasks in a zero-shot manner. Specifically, we employ an iterative search strategy to find better hyperparameters. At the start of the search process, we randomly sample and evaluate 10 hyperparameters to establish the initial "history" of the LM-Searcher. In each subsequent iteration, we randomly sample 10 candidate hyperparameters from the search space. We then provide these 10 candidate hyperparameters, along with the current "history" of previously evaluated (hyperparameter-performance) pairs, to the LM-Searcher, asking it to predict which of these 10 candidate hyperparameters is likely to achieve the best performance. We then evaluate the performance of the selected hyperparameter on the target task and add the resulting (hyperparameter-performance) pair to the "history." To accommodate the limited context window of LLM, we retain only the top 200 hyperparameter-performance pairs in the "history." This iterative process continues for a fixed number of iterations. Finally, we report the hyperparameter with the highest performance observed throughout the entire search process as the final result.

[0067] In summary, the method proposed in this invention effectively leverages the potential of Large Language Models (LLMs) in training hyperparameter search. Preliminary investigations show that while LLMs possess powerful inference capabilities, their direct applications (e.g., GPT-4o) still struggle to handle the inherently complex optimizations of hyperparameter search. To address this issue, this invention proposes LM-Searcher, reformulating the training hyperparameter search task as a ranking task, thereby better utilizing the advantages of LLMs. Fine-tuning LM-Searcher on a selected dataset derived from NAS-Bench-301 demonstrates its ability to learn fine-grained performance differences and its effective generalization across various search spaces and datasets. Notably, LM-Searcher achieves performance comparable to, or even surpassing, traditional NAS methods while maintaining generalization capabilities without requiring specific optimizations or retraining for different search spaces and datasets.

[0068] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0069] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0070] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0071] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk drive or magnetic tape drive. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0072] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0073] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0074] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0075] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0076] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0077] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0078] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0079] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0080] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

[0081] References:

[0082] [1]RohanAsthana,Joschua Conrad,YoussefDawoud,Maurits Ortmanns,andVasileios Belagiannis.Multi-conditioned graph diffusion for neuralarchitecture search.arXiv preprint arXiv:2403.06020,2024

[0083] [2]Tom Brown,Benjamin Mann,Nick Ryder,Melanie Subbiah,Jared D Kaplan,Prafulla Dhariwal,Arvind Neelakantan,Pranav Shyam,Girish Sastry,AmandaAskell,et al.Language models are few-shot learners.Advances in neuralinformationprocessing systems,33:1877–1901,2020

[0084] [3]Angelica Chen,David Dohan,and David So.Evoprompting:Languagemodels for code-level neural architecture search.Advances in NeuralInformation Processing Systems,36,2024

[0085] [4]Tianrun Chen,Lanyun Zhu,Chaotao Deng,Runlong Cao,Yan Wang,Shangzhan Zhang,Zejian Li,Lingyun Sun,Ying Zang,and Papa Mao.Sam-adapter:Adapting segment anything in underperformed scenes.In Proceedings of theIEEE / CVF International Conference on Computer Vision,pages 3367–3375,2023

[0086] [5]Wuyang Chen,Xinyu Gong,and Zhangyang Wang.Neural architecturesearch on imagenet in four gpu hours:A theoretically inspiredperspective.arXiv preprint arXiv:2102.11535,2021

[0087] [6]Aakanksha Chowdhery,Sharan Narang,Jacob Devlin,Maarten Bosma,Gaurav Mishra,Adam Roberts,Paul Barham,Hyung Won Chung,Charles Sutton,Sebastian Gehrmann,et al.Palm:Scaling language modeling with pathways.JournalofMachine Learning Research,24(240):1–113,2023

[0088] [7]Noel Codella,Veronica Rotemberg,Philipp Tschandl,M Emre Celebi,Stephen Dusza,David Gutman,Brian Helba,Aadi Kalloo,Konstantinos Liopyris,Michael Marchetti,et al.Skin lesion analysis toward melanoma detection 2018:Achallenge hosted by the international skin imaging collaboration(isic).arXivpreprint arXiv:1902.03368,2019

[0089] [8]Noel CF Codella,David Gutman,M Emre Celebi,Brian Helba,Michael AMarchetti,Stephen W Dusza,Aadi Kalloo,Konstantinos Liopyris,Nabin Mishra,Harald Kittler,et al.Skin lesion analysis toward melanoma detection:Achallenge at the 2017 international symposium on biomedical imaging(isbi),hosted by the international skin imaging collaboration(isic).In2018 IEEE 15thinternational symposium on biomedical imaging(ISBI 2018),pages 168–172.IEEE,2018

[0090] [9]Xuanyi Dong and Yi Yang.Nas-bench-201:Extending the scope ofreproducible neural architecture search.arXiv preprint arXiv:2001.00326,2020

[0091]

[10] Abhimanyu Dubey,Abhinav Jauhri,Abhinav Pandey,Abhishek Kadian,Ahmad Al-Dahle,Aiesha Letman,Akhil Mathur,Alan Schelten,Amy Yang,Angela Fan,et al.The llama 3 herd of models.arXiv preprint arXiv:2407.21783,2024

[0092]

[11] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Minlie Huang, NanDuan, Weizhu Chen, et al. Tora: A toolintegrated reasoning agent for mathematical problem solving. In The Twelfth International Conference on LearningRepresentations.

[0093]

[12] Yuchao Gu, 15890–15902,2023.

[0094]

[13] Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196,2024.

[0095]

[14] Jack Hessel,Ari Holtzman,Maxwell Forbes,Ronan Le Bras,and YejinChoi.Clipscore:A reference-free evaluation metric for image captioning.InProceedings of the 2021 Conference on Empirical Methods in Natural LanguageProcessing,pages 7514–7528,2021.

[0096]

[15] Geoffrey Hinton.Distilling the knowledge in a neuralnetwork.arXiv preprint arXiv:1503.02531,2015.

[0097]

[16] Neil Houlsby,Andrei Giurgiu,Stanislaw Jastrzebski,Bruna Morrone,Quentin De Laroussilhe,Andrea Gesmundo,MonaAttariyan,and SylvainGelly.Parameter-efficient transfer learning for nlp.In Internationalconference on machine learning,pages 2790–2799.PMLR,2019.

[0098]

[17] Edward J Hu,Yelong Shen,Phillip Wallis,ZeyuanAllenZhu,Yuanzhi Li,Shean Wang,Lu Wang,and Weizhu Chen.Lora:Low-rank adaptation of large languagemodels.arXiv preprint arXiv:2106.09685,2021.

[0099]

[18] Ganesh Jawahar,Muhammad Abdul-Mageed,Laks VS Lakshmanan,andDujian Ding.LLM performance predictors are good initializers for architecturesearch.arXiv preprint arXiv:2310.16712,2023.

[0100]

[19] Debesh Jha,Pia H Smedsrud,Michael A Riegler, Halvorsen,ThomasDe Lange,Dag Johansen,and D Johansen.Kvasir-seg:A segmented polypdataset.In MultiMedia modeling:26th international conference,MMM 2020,Daejeon,South Korea,January 5–8,2020,proceedings,part II 26,pages 451–462.Springer,2020.

[0101]

[20] Alexander Kirillov,Eric Mintun,Nikhila Ravi,Hanzi Mao,ChloeRolland,Laura Gustafson,Tete Xiao,Spencer Whitehead,Alexander C Berg,Wan-YenLo,et al.Segment anything.In Proceedings of the IEEE / CVF internationalconference on computer vision,pages 4015–4026,2023.

[0102]

[21] Nupur Kumari,Bingliang Zhang,Richard Zhang,Eli Shechtman,and Jun-Yan Zhu.Multi-concept customization of text-to-image diffusion.In Proceedingsof the IEEE / CVF conference on computer vision and pattern recognition,pages1931–1921,2023.

[0103]

[22] Trung-Nghia Le,Tam V Nguyen,Zhongliang Nie,Minh-Triet Tran,andAkihiro Sugimoto.Anabranch network for camouflaged objectsegmentation.Computer vision and image understanding,184:45–56,2019.

[0104]

[23] Dongze Lian,Daquan Zhou,Jiashi Feng,and Xinchao Wang.Scaling&shifting your features:A new baseline for efficient model tuning.Advances inNeural Information Processing Systems,35:109–123,2022.

[0105]

[24] Hanxiao Liu,Karen Simonyan,and Yiming Yang.Darts:Differentiablearchitecture search.arXiv preprint arXiv:1806.09055,2018.

[0106]

[25] Haotian Liu,Chunyuan Li,Qingyang Wu,and Yong Jae Lee.Visualinstruction tuning.Advances in neural information processing systems,36,2024.

[0107]

[26] Tennison Liu,Nicolás Astorga,Nabeel Seedat,and Mihaela van derSchaar.Large language models to enhance bayesian optimization.arXiv preprintarXiv:2402.03921,2024.

[0108]

[27] Jovita Lukasik,Steffen Jung,and Margret Keuper.Learning where tolook–generative nas is surprisingly efficient.In European Conference onComputerVision,pages 257–273.Springer,2022.

[0109]

[28] Muhammad Umair Nasir,Sam Earle,Julian Togelius,Steven James,andChristopher Cleghorn.Llmatic:neural architecture search via large languagemodels and quality diversity optimization.In Proceedings of the Genetic andEvolutionary Computation Conference,pages 1110–1118,2024.

[0110]

[29] OpenAI.https: / / openai.com / index / gpt-4 / .,2023.

[0111]

[30] Samyam Rajbhandari,JeffRasley,Olatunji Ruwase,andYuxiong He.Zero:Memory optimizations toward training trillion parameter models.In SC20:International Conference for High Performance Computing,Networking,StorageandAnalysis,pages 1–16.IEEE,2020.

[0112]

[31] Esteban Real,Alok Aggarwal,Yanping Huang,and Quoc VLe.Regularized evolution for image classifier architecture search.InProceedings ofthe aaai conference on artificial intelligence,pages 4780–4789,2019.

[0113]

[32] Robin Rombach,Andreas Blattmann,Dominik Lorenz,Patrick Esser,and Ommer.High-resolution image synthesis with latent diffusion models.InProceedings of the IEEE / CVF conference on computer vision and patternrecognition,pages 10684–10695,2022.

[0114]

[33] Baptiste Roziere,Jonas Gehring,Fabian Gloeckle,Sten Sootla,ItaiGat,Xiaoqing Ellen Tan,Yossi Adi,Jingyu Liu,Tal Remez,Jérémie Rapin,etal.Code llama:Open foundation models for code.arXiv preprint arXiv:2308.12950,2023.

[0115]

[34] Xingyou Song,Yingtao Tian,Robert Tjarko Lange,Chansoo Lee,YujinTang,and Yutian Chen.Position paper:Leveraging foundational models for black-box optimization:Benefits,challenges,and future directions.arXiv preprintarXiv:2405.03547,2024.

[0116]

[35] Yi-Lin Sung,Jaemin Cho,and Mohit Bansal.Lst:Ladder side-tuningfor parameter and memory efficient transfer learning.Advances in NeuralInformation Processing Systems,35:12991–13005,2022.

[0117]

[36] Mingxing Tan,Bo Chen,Ruoming Pang,Vijay Vasudevan,Mark Sandler,Andrew Howard,and Quoc V Le.Mnasnet:Platform-aware neural architecture searchfor mobile.In Proceedings ofthe IEEE / CVF conference on computer vision andpattern recognition,pages 2820–2828,2019.

[0118]

[37] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.

[0119]

[38] Gemini Team,Rohan Anil,Sebastian Borgeaud,Yonghui Wu,Jean-Baptiste Alayrac,Jiahui Yu,Radu Soricut,Johan Schalkwyk,Andrew M Dai,AnjaHauth,et al.Gemini:a family ofhighly capable multimodal models.arXiv preprintXiv:2312.11805,2023.

[0120]

[39] Hugo Touvron,Thibaut Lavril,Gautier Izacard,Xavier Martinet,Marie-Anne Lachaux,Timothy Lacroix,Baptiste Rozier,Naman Goyal,Eric Hambro,Faisal Azhar,et al.Llama:Open and efficient foundation language models.arXivpreprint arXiv:2302.13971,2023.

[0121]

[40] Tomás F Yago Vicente,Le Hou,Chen-Ping Yu,Minh Hoai,and DimitrisSamaras.Large-scale training of shadow detectors with noisily-annotatedshadow examples.In Computer Vision–ECCV 2016:14th European Conference,Amsterdam,The Netherlands,October 11-14,2016,Proceedings,Part VI 14,pages816–832.Springer,2016.

[0122]

[41] Andrey Voynov,Qinghao Chu,Daniel Cohen-Or,and Kfir Aberman.p+:Extended textual conditioning in text-to-image generation.arXiv preprintarXiv:2303.09522,2023.

[0123]

[42] Ke Wang,Houxing Ren,Aojun Zhou,Zimu Lu,Sichun Luo,Weikang Shi,Renrui Zhang,Linqi Song,Mingjie Zhan,and Hongsheng Li.Mathcoder:Seamless codeintegration in llms for enhanced mathematical reasoning.In 12th InternationalConference on Learning Representations(ICLR 2024),2024.

[0124]

[43] Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. Deepseek-prover: Advancing theoremproving in llms through large-scale synthetic data. arXiv preprint arXiv:2405.14333,2024.

[0125]

[44] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, ​​and Hongkai Xiong. Pc-darts: Partial channel connections for memory-efficient architecture search. arXiv preprint arXiv:1907.05737,2019.

[0126]

[45] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen 2.5-mathtechnical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122,2024.

[0127]

[46] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simpleparameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199, 2021.

[0128]

[47] Arber Zela,Julien Siems,Lucas Zimmer,Jovita Lukasik,MargretKeuper,and Frank Hutter.Surrogate nas benchmarks:Going beyond the limitedsearch spaces of tabular nas benchmarks.arXiv preprint arXiv:2008.09777,2020.

[0129]

[48] Mingkai Zheng,Xiu Su,Shan You,Fei Wang,Chen Qian,Chang Xu,andSamuel Albanie.Can gpt-4 perform neural architecture search?arXiv preprintarXiv:2304.10970,2023.

[0130]

[49] Yaowei Zheng,Richong Zhang,Junhao Zhang,Yanhan Ye,Zheyan Luo,Zhangchi Feng,and Yongqiang Ma.Llamafactory:Unified efficient fine-tuning of100+language models.arXiv preprint arXiv:2403.13372,2024.

[0131]

[50] Zihan Zhong,Zhiqiang Tang,Tong He,Haoyang Fang,and ChunYuan.Convolution meets lora:Parameter efficient finetuning for segmentanything model.In The Twelfth International Conference on LearningRepresentations.

[0132]

[51] Barret Zoph and Quoc V Le.Neural architecture search withreinforcement learning.arXiv preprint arXiv:1611.01578,2016.

Claims

1. A general search method for training hyperparameters supported by a large language model, characterized in that, Includes the following steps: S1. Reconstruct the training hyperparameter search task into a ranking problem of candidate hyperparameters, obtain performance data including historical hyperparameters and text encoding of candidate hyperparameters, and then sort the performance of candidate hyperparameters using the Large Language Model (LLM). S2. Sample training data from a local region of the target search space, wherein the training data includes multiple historical hyperparameter-performance pairs and corresponding candidate hyperparameter sets; S3. Use the training data to fine-tune the LLM so that it learns the mapping relationship between hyperparameter encoding and performance metrics; S4. In the zero-sample scenario, the historical hyperparameter-performance pairs are dynamically updated through an iterative search strategy, and the candidate hyperparameters are ranked based on the fine-tuned LLM to output the hyperparameter with the best performance.

2. The method according to claim 1, characterized in that, In step S1, the text encoding of the candidate hyperparameter is a numerical code generated based on search space options or topology index. The numerical code is used to uniquely identify hyperparameter features and capture core network features, avoiding reliance on prior knowledge of search space and network topology.

3. The method according to claim 2, characterized in that, In step S1, the ranking task is achieved by transforming hyperparameter generation into a classification problem, and LLM identifies the optimal hyperparameter based on the candidate hyperparameters represented by numerical codes.

4. The method according to claim 1, characterized in that, In step S2, the local region of the target search space is generated by configuring a fixed portion of hyperparameters, and the sampling area is narrowed by setting the discard probability of the options in the search space.

5. The method according to claim 4, characterized in that, In step S2, the training data is derived from a NAS benchmark dataset containing diverse hyperparameter configurations. Each sample includes historical hyperparameter-performance pairs sampled from the same local region and a candidate hyperparameter set, and the candidate set contains at least the hyperparameter with the best performance in the current local region.

6. The method according to claim 1, characterized in that, In step S4, the iterative search strategy includes: at the start of the search, randomly sampling and evaluating a fixed number of hyperparameters to establish initial historical data; in each iteration, randomly sampling a fixed number of candidate hyperparameters from the search space, inputting them along with the current historical data into the fine-tuned LLM to predict the optimal hyperparameters; evaluating the performance of the selected hyperparameters and updating the historical data, while dynamically truncating the amount of historical data and retaining some hyperparameter-performance pairs in the "history" to adapt to the context window constraints of the LLM; and after the iteration continues for a set number of times, outputting the hyperparameter with the highest performance.

7. The method according to claim 1, characterized in that, In step S4, the performance evaluation of the candidate hyperparameters is based on the fast verification metrics of the proxy task, which include model accuracy, computational efficiency, or task-specific scores.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the general search method for training hyperparameters supported by the large language model as described in any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the general search method for training hyperparameters supported by the large language model as described in any one of claims 1 to 7.

10. A training hyperparameter search device based on a large-scale language model, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the general search method for training hyperparameters supported by a large language model as described in any one of claims 1 to 7.