Automatic neural architecture searching method and system based on large language model
By combining the generation ability of large language models with actual machine learning tasks, cross-and-mutation operations based on few-shot prompts are used to generate new candidate architectures, and fine-tune them through training and evaluation to select the most adaptable architecture for fine-tuning, the existing NAS methods are solved inefficient in large and complex search spaces, and efficient and diversified neural architecture search is achieved.
Patent Information
- Application Number
- CN202411784174.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-04-11
AI Technical Summary
When facing large and complex search spaces, existing NAS methods are inefficient, time-consuming, and computational resources are consumed. The fixed set of operations cannot capture new and efficient network structure variants, limiting the innovation of the architecture.
By combining the generation ability of large language models with actual machine learning tasks, cross-and-mutation operations based on few-shot prompts are adopted to generate new candidate architectures, and fine-tune the architecture with the highest adaptability through training and evaluation to improve the adaptability of the generated architecture.
It significantly improves the efficiency of neural architecture search, improves the search performance of the final model, expands the diversity and adaptability of the search space, reduces the dependence on hyperparameter adjustment, and lowers the threshold for use.
Smart Images
Figure CN119807487A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to an automatic neural architecture search method and system based on a large language model. Background Art
[0002] Neural Architecture Search (NAS) is an automated technology that aims to find the optimal neural network architecture for a specific machine learning task. In recent years, with the rapid development and widespread application of deep learning, NAS technology has received increasing attention. Its core goal is to automatically discover efficient and high-performance network architectures through algorithms in the huge and complex neural network design space, thereby reducing the time and effort required for manual design.
[0003] Traditional NAS methods usually require defining a discrete search space consisting of a series of predefined neural network operations, such as convolutional layers, pooling layers, fully connected layers, and their combinations. The search process of these methods usually relies on heuristic strategies such as random search, Bayesian optimization, or genetic algorithms. These heuristic strategies may perform well in small search spaces, but their efficiency often drops significantly when faced with large and complex search spaces, resulting in a time-consuming search process and high consumption of computing resources. The design of discrete search spaces relies on the researcher's experience and understanding of the task, which limits its flexibility and adaptability. In addition, a fixed set of operations may not capture new and efficient network structure variants, limiting the innovation of potential architectures. Therefore, how to effectively expand and optimize the search space has become an important challenge facing the NAS field.
[0004] With the diversification of deep learning applications, the demand for automated design of more complex network architectures has become increasingly urgent. An ideal NAS system should not only be able to efficiently search for high-quality architectures, but also have adaptive capabilities to meet the needs of different tasks. To this end, this application proposes an automated neural architecture search method based on a large language model, including techniques such as hyperparameter fine-tuning based on a large language model, which successfully combines the generation capability of LLMs with actual machine learning tasks, and realizes an automated neural architecture search process. Summary of the invention
[0005] To address the deficiencies of the prior art, the present invention successfully combines the generation capability of LLMs with the actual machine learning task through the above-mentioned method, and realizes an automated neural architecture search process. This process significantly improves the search efficiency and significantly improves the search performance of the final model.
[0006] To achieve the above objectives, the technical solutions adopted by the present invention include:
[0007] The present invention provides an automatic neural architecture search method based on a large language model, characterized by comprising:
[0008] In the initialization stage, a set of representative and diverse seed architecture sets is selected to initialize the seed architecture set;
[0009] In the hint evolution phase, a few-shot hint is used to perform crossover and mutation operations on the initialized seed architecture set to generate new candidate architectures.
[0010] In the training and evaluation phase, the candidate architecture is trained on the task training set, and its performance is evaluated on the validation set to obtain evaluation results;
[0011] In the selection and prompt adjustment stage, the candidate architecture with the highest fitness is selected according to the evaluation results, and the prompt of the large language model is fine-tuned using the candidate architecture with the highest fitness to improve the adaptability of the generated architecture.
[0012] Exemplarily, the seed architecture includes multiple different types of graph neural network layers and connection patterns to explore different design spaces.
[0013] Exemplarily, in the training and evaluation phase, the candidate architecture is trained on the task training set, and its performance is evaluated on the validation set. Obtaining the evaluation result includes: defining a neural network model of the candidate architecture, defining a cross entropy loss function as an objective function for optimizing the neural network model, performing multiple rounds of iterative training on the training set, and selecting the candidate architecture according to a set validation error threshold.
[0014] Exemplarily, the selection and prompt adjustment stage selects the candidate architecture with the highest fitness according to the evaluation results, and also includes: using the fitness function in combination with the verification error, model size and selection mechanism of the candidate architecture to identify the candidate architecture with the highest fitness.
[0015] Exemplarily, the cross entropy function uses an early stopping method to prevent overfitting.
[0016] Exemplarily, the fine-tuning of the prompt of the large language model using the candidate architecture with the highest applicability includes using all candidate architectures not used in the training and evaluation phases as training data to update the prompt parameters of the large language model.
[0017] The present application also provides an automated neural architecture search system based on a large language model, characterized in that it is used to execute any method described in claims 1 to 6, including:
[0018] An initialization module, used to select a set of representative and diverse seed architecture sets and initialize the seed architecture sets;
[0019] A hint evolution module that uses few-shot hints to perform crossover and mutation operations on an initialized set of seed architectures to generate new candidate architectures;
[0020] A training and evaluation module, used to train the candidate architecture on the task training set and evaluate its performance on the validation set to obtain an evaluation result;
[0021] The selection and prompt adjustment module is used to select the candidate architecture with the highest fitness according to the evaluation results, and use the candidate architecture with the highest fitness to fine-tune the prompt of the large language model.
[0022] The present invention also provides a computer-readable storage medium, characterized in that a computer program is stored on the storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
[0023] The present invention also provides an electronic device, characterized in that it includes a processor and a memory;
[0024] The memory is used to store computer programs;
[0025] The processor is configured to execute the method according to any one of claims 1 to 6 by calling a stored computer program.
[0026] The present invention also relates to a computer program product, comprising a computer program and / or instructions, characterized in that the computer program and / or instructions implement the steps of the above method when executed by a processor.
[0027] The beneficial effects of the present invention are:
[0028] Automated search space expansion: Leveraging the natural language generation capabilities of large language models (LLMs), this paper achieves automated expansion of the search space, allowing exploration of broader and more diverse graph neural network (GNN) architectures, providing more possibilities for architectural innovation.
[0029] Balance between computational efficiency and performance: By introducing evolutionary algorithms and soft-hint adjustment methods, the computational efficiency of the search process is improved, significantly reducing the reliance on large amounts of computing resources, so that a high-performance model architecture can still be obtained in a limited resource environment.
[0030] Enhanced generalization and adaptability: This paper enhances the generalization and adaptability of the GNN-NAS method, enabling it to effectively cope with different types of graph structured data and a wide range of machine learning tasks, rather than just being suitable for specific tasks, providing support for diversified application scenarios.
[0031] Simplify hyperparameter adjustment: Through the automated prompt adjustment process, the complex dependence on hyperparameter adjustment is reduced, so that non-expert users can easily apply the system for GNN architecture search, thereby lowering the usage threshold and promoting the popularization and application of technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 Schematic diagram of the process flow of the method for automatic neural architecture search based on a large language model of the present invention.
[0033] Figure 2 Schematic diagram of the structure of the automatic neural architecture search system based on the large language model of the present invention. DETAILED DESCRIPTION
[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0035] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.
[0036] The first aspect of the present invention relates to a method for automatic neural architecture search based on a large language model. Figure 1 As shown, including:
[0037] In the initialization stage, a set of representative and diverse seed architecture sets is selected to initialize the seed architecture set;
[0038] Specifically, the selected seed architecture should be a well-designed example that covers different types of graph network structures and design principles. The specific requirements are simplicity: the selected architecture should remain simple and clear to facilitate the understanding and generation of large language models (LLMs); and diversity: the seed architecture should include different types of layers and connection modes to effectively explore a wider design space. The initialization phase can provide a good starting point for subsequent architecture searches, ensuring the efficiency and effectiveness of the search process.
[0039] In the hint evolution phase, a few-shot hint is used to perform crossover and mutation operations on the initialized seed architecture set to generate new candidate architectures.
[0040] To create an effective few-shot hint, k models are first randomly selected from the current seed architecture collection as examples. These examples include the source code of the models and their evaluation metrics, such as model size and validation accuracy. Then, LLMs constructs a hint that contains these examples, code operation details of crossover and mutation, and the performance optimization direction of the target model. The goal is to generate a model with fewer parameters and higher accuracy. The hint will contain information like: "Metrics:{'num_params':'Target model size','val_accuracy':'Target accuracy'}". For example, the target model size can be set to 95% of the minimum model size of the parent model, and the target validation accuracy can be set to 102% of the maximum validation accuracy of the parent model.
[0041] Assume that the condition of hint-based LLMs is that each few-shot hint generates n hints, and each round generates m hints in total. p is the candidate parent population to be selected in each generation. The crossover and mutation operation function performed by hint evolution is expressed as CrossMutation(,P,m,k,n). The crossover here comes from the crossover update of each source code in the seed code group.
[0042] In the training and evaluation phase, the candidate architecture is trained on the task training set, and its performance is evaluated on the validation set to obtain evaluation results;
[0043] Specifically, the training process is implemented through the EVALT(c,D) function, which receives the generated code sample c and the training dataset D as input. Each candidate architecture c is trained on the dataset D using a predefined optimizer and hyperparameters. The goal of training is to minimize the model's loss function while ensuring that the model does not overfit the training data. The training process can be formalized using the following mathematical expression: min θ L(f θ (x),y), where θ represents the model parameters, f θ(x) represents the model prediction, y is the true label, and L is the loss function, such as cross entropy loss. A validation set is separated from the entire dataset to evaluate the generalization ability of the model. The preparation of the validation set needs to ensure that it represents the distribution of the training set and is large enough to provide reliable performance estimates. When evaluating candidate architectures, two main indicators are concerned: the validation accuracy and model size of the candidate architecture. Here, candidate architectures with poor performance are excluded according to the preset validation error threshold α. Only when the validation error of the candidate architecture is lower than α will it be retained and enter the next selection stage
[0044] In the selection and prompt adjustment stage, the candidate architecture with the highest fitness is selected according to the evaluation results, and the prompt of the large language model is fine-tuned using the candidate architecture with the highest fitness to improve the adaptability of the generated architecture.
[0045] Specifically, in this patent, the fitness function usually combines the validation error and model size of the candidate architecture.
[0046] The following fitness function can be used: Fitness(c) = -ValError(c) × ModelSize(c). Where ValError(c) is the accuracy of the generated code sample c on the validation set, ModelSize(c) is the size of the code sample c, and the negative sign indicates that we want to maximize the validation accuracy and minimize the candidate architecture size.
[0047] The selection mechanism is responsible for identifying the most fit candidate architecture from the current population. In this application, this is usually achieved through ranking or tournament-based selection. For example, the top p individuals with the highest fitness can be selected for elimination, and the remaining candidate architectures can be used as parents for the next round to maintain population diversity.
[0048] Hint tuning involves using the evaluated individuals as training data to fine-tune the hints of LLMs. In this way, LLMs can learn how to generate architectures that are more adapted to the task.
[0049] The prompt adjustment process can be formalized into the following steps:
[0050] 1. Select p most adaptable individuals (candidate architectures) from the current population;
[0051] 2. Use the population after removing p individuals (candidate architectures) as case data to update the LLMs hints;
[0052] 3. Adjust the parameters of LLMs through the fine-tuning process to better generate an architecture that adapts to the task. The mathematical expression is as follows: θ t+1 ←TRAIN(θ t ,C evaled \P), where θt+1 is the few-shot cue parameter for the next round, θ t is the few-shot hint parameter for this round, TRAIN represents the training process, which uses all previously unselected candidate architectures C that were not generated in the previous round. evaled \P to adjust the next round of prompts, that is, to update θ in LLMs t Prompt parameter.
[0053] Embodiment 2:
[0054] In the initialization phase of this case, the goal is to build a diverse and representative set of seed architectures. To this end, we selected five different graph neural network (GNN) architectures that are optimized for different types of graph data, including the classic graph convolutional network (GCN), graph attention network (GAT), and three custom GNN architectures tailored for specific application scenarios.
[0055] To ensure the simplicity and diversity of seed architectures so that large language models (LLMs) can efficiently understand and generate new architectures. These seed architectures serve as input to LLMs and lay the foundation for the subsequent hint evolution stage, thus providing a starting point for the search algorithm to explore a wide design space.
[0056] In the hint evolution phase, the seed architecture obtained in the initialization phase is crossover and mutation operated by leveraging the few-shot learning capability of LLMs. First, 5 models are randomly selected from the current population as examples, which contain source code and evaluation metrics such as model size and validation accuracy.
[0057] Next, a few-shot hint is constructed that contains these examples and performance optimization directions. For example, we set the target model size to 95% of the smallest model of the parent generation and the target validation accuracy to 102% of the maximum validation accuracy of the parent generation. This hint guides LLMs to generate new architecture variants that aim to reduce the number of parameters while improving the accuracy of the model.
[0058] In the training and evaluation phases, the 20 GNN architectures generated in the hint evolution phase are applied to the standard graph dataset for training. A graph dataset containing 10,000 nodes is used, and the same computing resources and training time are allocated to each architecture.
[0059] The training goal is to minimize the model's loss function while avoiding overfitting. The cross entropy loss function is used, and the early stopping method is used to prevent overfitting. During the training process, the loss of each model and its performance on the independent validation set are recorded.
[0060] The evaluation results show that 18 architectures have higher accuracy than the initial seed architecture on the validation set, and the best architecture has an accuracy improvement of 5% while reducing the model size by 20%. This result verifies the effectiveness of this method in improving model performance and reducing resource consumption. Through this step, 20 new GNN architectures were successfully generated, which were generated by cross-mutation of the original architecture. The new architectures reduce the number of parameters by an average of 15% while maintaining or improving accuracy, fully demonstrating the potential of LLMs in automated architecture innovation.
[0061] In the final selection and prompt adjustment stage, the best 5 architectures were selected based on the performance of the model on the validation set. These architectures not only performed well in terms of accuracy, but also showed significant advantages in terms of model size and training time.
[0062] Using these best architectures as case data, the LLMs’ prompts were adjusted. Through the fine-tuning process, the parameters of the LLMs were updated so that they could generate architectures that better meet the task requirements in the next round of search. This step not only improves the LLMs’ ability to generate high-quality architectures, but also provides a better starting point for future search tasks.
[0063] Through this process, the generation capability of LLMs was successfully combined with the actual machine learning task, realizing an automated neural architecture search process. This process significantly improved the search efficiency and significantly improved the search performance of the final model.
[0064] Another aspect of the present invention also relates to an automated neural architecture search system based on a large language model, which is used to perform any method of the method claims, such as Figure 2 As shown, including,
[0065] An initialization module, used to select a set of representative and diverse seed architecture sets and initialize the seed architecture sets;
[0066] A hint evolution module that uses few-shot hints to perform crossover and mutation operations on an initialized set of seed architectures to generate new candidate architectures;
[0067] A training and evaluation module, used to train the candidate architecture on the task training set and evaluate its performance on the validation set to obtain an evaluation result;
[0068] The selection and prompt adjustment module is used to select the candidate architecture with the highest fitness according to the evaluation results, and use the candidate architecture with the highest fitness to fine-tune the prompt of the large language model.
[0069] By using this system, the above-mentioned operation processing method can be executed and the corresponding technical effects can be achieved.
[0070] An embodiment of the present invention also provides a computer-readable storage medium capable of implementing all the steps in the method in the above embodiment, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all the steps in the method in the above embodiment are implemented.
[0071] An embodiment of the present invention also provides an electronic device for executing the above method. As an implementation device of the method, the electronic device has at least a processor and a memory, and in particular, the memory stores data and related computer programs required for executing the method, such as storage, etc., and the processor calls the data and programs in the memory to execute all the steps of the implementation method and obtains the corresponding technical effect.
[0072] Preferably, the electronic device may include a bus architecture, the bus may include any number of interconnected buses and bridges, and the bus will include various circuits linked together by one or more processors and memories. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are all well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same component, i.e., a transceiver, providing a unit for communicating with various other systems over a transmission medium. The processor is responsible for managing the bus and general processing, while the memory may be used to store data used by the processor when performing operations.
[0073] In addition, the electronic device may further include components such as a communication module, an input unit, an audio processor, a display, and a power supply. The processor (or controller, operation control) used may include a microprocessor or other processor device and / or a logic device, which receives input and controls the operation of various components of the electronic device; the memory may be one or more of a cache, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices, which may store the above-mentioned related data information, and may also store a program for executing related information, and the processor may execute the program stored in the memory to realize information storage or processing, etc.; the input unit is used to provide input to the processor, for example, it may be a key or a touch input device; the power supply is used to provide power to the electronic device; the display is used to display display objects such as images and text, for example, it may be an LCD display. The communication module is a transmitter / receiver that sends and receives signals via an antenna. The communication module (transmitter / receiver) is coupled to the processor to provide input signals and receive output signals, which may be the same as the case of a conventional mobile communication terminal. Based on different communication technologies, multiple communication modules may be provided in the same electronic device, such as a cellular network module, a Bluetooth module and / or a wireless local area network module, etc. The communication module (transmitter / receiver) is also coupled to a speaker and a microphone via an audio processor to provide an audio output via the speaker and receive an audio input from the microphone, thereby realizing a common telecommunication function. The audio processor may include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor is also coupled to a central processor, so that recording can be performed on the machine through the microphone, and the sound stored on the machine can be played through the speaker.
[0074] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0075] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.
[0076] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0077] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in a box or multiple boxes. Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the attached claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0078] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. An automated neural architecture search method based on a large language model, characterized in that: include: In the initialization stage, a set of representative and diverse seed architecture sets is selected to initialize the seed architecture set; In the hint evolution phase, a few-shot hint is used to perform crossover and mutation operations on the initialized seed architecture set to generate new candidate architectures. In the training and evaluation phase, the candidate architecture is trained on the task training set, and its performance is evaluated on the validation set to obtain evaluation results; In the selection and prompt adjustment stage, the candidate architecture with the highest fitness is selected according to the evaluation results, and the prompt of the large language model is fine-tuned using the candidate architecture with the highest fitness to improve the adaptability of the generated architecture.
2. The method according to claim 1, characterized in that The seed architecture includes multiple different types of graph neural network layers and connection patterns to explore different design spaces.
3. The method according to claim 2, characterized in that In the training and evaluation phase, the candidate architecture is trained on the task training set and its performance is evaluated on the validation set. The evaluation results include: defining a neural network model of the candidate architecture, defining a cross entropy loss function as an objective function for optimizing the neural network model, performing multiple rounds of iterative training on the training set, and selecting a candidate architecture based on a set validation error threshold.
4. The method according to claim 3, characterized in that The selection and prompt adjustment stage selects the candidate architecture with the highest fitness according to the evaluation results, and also includes: using the fitness function in combination with the verification error, model size and selection mechanism of the candidate architecture to identify the candidate architecture with the highest fitness.
5. The method according to claim 3, characterized in that: The cross entropy function uses early stopping to prevent overfitting.
6. The method according to claim 1, characterized in that The using of the candidate architecture with the highest applicability to fine-tune the prompt of the large language model includes using all candidate architectures not used in the training and evaluation phases as training data to update the prompt parameters of the large language model.
7. An automated neural architecture search system based on a large language model, characterized in that: Used to perform any of the methods described in claims 1 to 6, comprising: An initialization module, used to select a set of representative and diverse seed architecture sets and initialize the seed architecture sets; A hint evolution module that uses few-shot hints to perform crossover and mutation operations on an initialized set of seed architectures to generate new candidate architectures; A training and evaluation module, used to train the candidate architecture on the task training set and evaluate its performance on the validation set to obtain an evaluation result; The selection and prompt adjustment module is used to select the candidate architecture with the highest fitness according to the evaluation results, and use the candidate architecture with the highest fitness to fine-tune the prompt of the large language model.
8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.
9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store computer programs; The processor executes the method according to any one of claims 1 to 6 by calling a stored computer program.
10. A computer program product comprising a computer program and / or instructions, characterized in that: When the computer program and / or the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.