Black box neural network architecture search method

Abstract

The invention provides a black box neural network architecture search method, which is called UniSearch for short, a large language model LLM is used as an optimizer, architecture representation is simplified through a consistent prompt structure coding digital character string, and expert knowledge requirements are eliminated. The UniSearch adopts a sample-evaluation-update loop iterative optimization architecture, and focuses on a historical high reward architecture pair to improve the search efficiency. The method does not depend on specific knowledge, UniSearch fully mines and utilizes the reasoning ability of LLM, and discovers a new mode in a search space. Verification is carried out in a CIFAR-100, ImageNet image classification and diffusion Transform (DiT) architecture search task, and compared with a traditional method, the UniSearch provided by the invention has the advantage that better performance is realized on the basis of fewer samples.

Description

Technical Field

[0001] This invention relates to the application of Large Language Models (LLM) in Neural Architecture Search (NAS), and particularly to a black-box neural architecture search method. Background Technology

[0002] Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of language tasks, but their potential in reasoning tasks remains largely untapped. Neural architecture search (NAS) has traditionally relied on methods such as reinforcement learning (RL) and evolutionary algorithms, which face challenges in scalability and generalization due to hyperparameter sensitivity. Recent advances suggest that LLMs like GPT-4 may help address these challenges, although existing LLM-based NAS methods still require substantial expert knowledge and manual tweaking.

[0003] Specifically, Large Language Models (LLMs) have demonstrated impressive capabilities in various language tasks in recent years, enabling their application in advanced applications such as human chatbots. A major challenge facing LLMs is leveraging their reasoning skills to handle diverse downstream tasks. Early research has primarily focused on their mathematical abilities and problem-solving skills. However, the potential of LLMs in reasoning tasks has not yet been fully explored.

[0004] Finding the optimal neural network architecture for deep learning tasks is crucial, as it directly impacts model performance and efficiency. Neural Architecture Search (NAS) aims to automate this design process using specialized search algorithms. Traditional NAS methods primarily utilize reinforcement learning (RL) or evolutionary algorithms to explore a vast search space. While these methods have made significant progress, they often struggle with scalability and generalization due to their sensitivity to hyperparameters. Tuning these hyperparameters requires substantial expert knowledge and manual effort, making these methods impractical. Recent research suggests that LLMs may help address these challenges. For example, GENIUS (Zheng et al., 2023) demonstrates that competitive performance can be achieved with NAS hints for GPT-4. Leveraging the ability of LLMs to interact with users through natural language provides flexibility and expressiveness not found in traditional NAS methods. However, current LLM-based NAS methods still require significant expert knowledge and manual tuning to adapt to different search tasks, limiting their wider adoption.

[0005] Using LLM for model optimization

[0006] Transformer models, initially proposed by Vaswani et al. (2017) for sequence transduction, have demonstrated exceptional sequence modeling capabilities. These capabilities have been extended to tasks such as Bayesian inference and hyperparameter optimization, as noted by Müller et al. (2021) and Chen et al. (2022). In recent developments, Large Language Models (LLMs) have demonstrated significant in-context learning capabilities, coupled with their instruction-following abilities and extensive knowledge bases, enabling them to support model optimization tasks. Jawahar et al. (2023) evaluated the performance of different architectures using GPT-4, using these evaluations to guide the initialization of NAS. Furthermore, EvoPrompting (Chen et al., 2024) fine-tuned a PALM model, designing the neural network architecture directly at the code level. Further research, such as that by Song et al. (2024) and Liu et al. (2024b), has explored the potential of LLMs in performing black-box optimization.

[0007] A particularly relevant work is GENIUS (Zheng et al., 2023), which also adopts GPT-4 (OpenAI, 2023) to propose a model architecture during the search phase. However, their approach requires a detailed description of the target neural network and the problem, which is clearly complex and difficult to transfer. Furthermore, GENIUS does not strictly adhere to the black-box optimization framework because it exposes the block-level details of the model to GPT-4, potentially leading to biased recommendations based on the model's existing knowledge rather than genuine discoveries.

[0008] Neural Architecture Search (NAS) aims to define the structure of neural network architectures. Specifically, cell-based NAS divides the architecture into cell blocks, focusing on identifying the optimal combinations and permutations of various operations. Traditional methods typically use recurrent neural networks (RNNs) as controllers to sample architectures, driven by reward signals from experimental results (Zoph and Le, 2016; Zoph et al., 2018; Howard et al., 2019; Tan et al., 2019; Liu et al., 2021, 2022). Furthermore, reinforcement techniques such as proximal policy optimization (Schulman et al., 2017) are often employed to adapt the controller. However, this process of continuously updating the RNN controller weights in each search loop is not only time-consuming but also lacks interpretability.

[0009] Compared to traditional reinforcement learning-based search methods (Zoph and Le, 2016; Real et al., 2019), Neural Architecture Search (NAS) using Large Language Models (LLMs) (Zheng et al., 2023) has shown advantages in flexibility and efficiency. However, these works still require significant expert knowledge and manual tweaking, which limits their wider adoption.

[0010] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0011] The main objective of this invention is to solve the problems existing in the above-mentioned background technology and provide a black-box neural network architecture search method.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] The first aspect of this invention provides a black-box neural network architecture search method, comprising the following steps:

[0014] Encode the neural network architecture using numeric strings;

[0015] We employ a large language model LLM as a black-box optimizer and construct a sample-evaluation-update loop to iteratively search and optimize the architecture.

[0016] Construct a consistent prompt structure to present the LLM with a sequence of numbers to leverage its reasoning capabilities, enabling the LLM to discover patterns and relationships within the search space based on the sequence of numbers, rather than relying on prior architectural knowledge;

[0017] For each new architecture configuration proposed by LLM, an agent task is trained, and a reward score reflecting its performance is calculated.

[0018] Each architecture-reward pair is fed back to the LLM to update its contextual memory and provide an informational basis for the next architecture search.

[0019] The search algorithm searches and selects the top K architecture-reward pairs with the highest historical rewards to focus the search on more promising areas.

[0020] After the search phase is completed, the top architecture is retrained on the complete target task to obtain the final performance evaluation.

[0021] Based on the performance and computational complexity of the retrained architecture, the optimal architecture is selected as the search result.

[0022] In some embodiments, this black-box neural network architecture search method can be used to search for or optimize neural network architectures for image tasks (such as image classification, image generation, etc.).

[0023] A second aspect of the present invention provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the black-box neural network architecture search method.

[0024] A third aspect of the present invention is a computer program product, comprising a computer program, characterized in that the computer program, when executed by a processor, implements the black-box neural network architecture search method.

[0025] The present invention has the following beneficial effects:

[0026] To address the challenges of existing technologies for Neural Architecture Search (NAS) using Large Language Models (LLMs), this invention proposes a black-box neural architecture search method, denoted as UniSearch. UniSearch overcomes the limitations of traditional NAS methods by fully leveraging the reasoning capabilities of LLMs through a unified hint strategy. UniSearch treats the LLM as a black-box optimizer in a sample-evaluation-update loop, utilizing a consistent hint structure across various search tasks. Architectures are represented as numeric strings, and the LLM suggests new architectures based on historical data. Inspired by greedy search algorithms, this invention improves efficiency and reduces the need for extensive contextual information by focusing the LLM on the top K (e.g., the top 200) architecture-reward pairs. This method fully utilizes the potential of LLMs to optimize neural architectures, independent of specific knowledge or past experience. Instead, it leverages the LLM's ability to accurately absorb contextual information and infer patterns through logical reasoning, allowing it to discover patterns and relationships within the search space without bias from existing architectural knowledge. This invention validates UniSearch on traditional image classification tasks using CIFAR-100 and ImageNet, and further validates it on a novel task of searching for better Diffusion Transformer (DiT) architectures for image generation. The results demonstrate that UniSearch achieves superior performance on fewer samples compared to existing NAS methods.

[0027] In summary, this invention proposes an innovative black-box architecture search algorithm that leverages the inference capabilities of LLMs and employs a unified hint strategy. Inspired by greedy search, it accelerates the search process and improves efficiency. Extensive experiments on image classification and image generation demonstrate the effectiveness and generalization ability of UniSearch.

[0028] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0029] Figure 1 This is an illustrative example of the UniSearch framework according to an embodiment of the present invention. In the MobileNetV2 setting, the neural network architecture is divided into several stages, and the architecture configuration (i.e., kernel size, expansion ratio, filter size, and block repetition count) is represented by a numeric string consisting of option indices 0, 1, or 2.

[0030] Figure 2 The reward for the model sampled by GPT-3.5 on the CIFAR-100 dataset. The red line represents the reward of the original model.

[0031] Figure 3 To guide the LLM search for examples of MobileNetV2 architectures represented by numerical sequences based on the rewards of historical experiments.

[0032] Figure 4 The loss is for the DiT model sampled by GPT-4 on the ImageNet 64×64 dataset. The red line represents the loss of the standard DiT-S / 2 model. Detailed Implementation

[0033] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0034] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0035] This invention proposes a black-box neural network architecture search method, UniSearch, which overcomes the limitations of traditional methods using Large Language Models (LLMs) for Neural Network Architecture Search (NAS). UniSearch fully leverages the inference capabilities of LLMs through a unified hint strategy. UniSearch treats the LLM as a black-box optimizer in a sample-evaluation-update loop, utilizing a consistent hint structure across various search tasks. This invention also accelerates the search process and improves efficiency through a greedy search-inspired approach. Extensive experiments on image classification and image generation demonstrate the effectiveness and generalization ability of UniSearch.

[0036] See Figure 1This invention provides a black-box neural network architecture search method, comprising the following steps:

[0037] Initialize the architecture representation: Encode the neural network architecture using numeric strings to simplify the representation of the architecture and eliminate the need for expert knowledge;

[0038] The search algorithm is based on the following: a large language model (LLM) is used as a black-box optimizer, and a sample-evaluation-update loop is constructed to iteratively search and optimize the architecture.

[0039] Configure hint strategy: Build a consistent hint structure to present the LLM with numerical sequences to leverage its reasoning capabilities, enabling the LLM to discover patterns and relationships within the search space based on numerical sequences, rather than relying on prior architectural knowledge; thereby fully utilizing the reasoning capabilities of LLMs such as GPT-4, allowing them to discover patterns and relationships within the search space without being biased by existing architectural knowledge.

[0040] Execution of agent task evaluation: Train an agent task for each new architecture configuration proposed by LLM and calculate a reward score that reflects its performance;

[0041] Feedback rewards and update context memory: Feedback of each architecture-reward pair to the LLM to update its context memory and provide an informational basis for the next architecture search;

[0042] Application efficiency optimization techniques: Through search algorithms, the top K architecture-reward pairs with the highest historical rewards are selected using a heuristic greedy search algorithm to focus the search on more promising areas;

[0043] The strategy for retraining the architecture is as follows: After completing the search phase, the top architecture is retrained on the complete target task to obtain the final performance evaluation.

[0044] Select top-tier architecture: Based on the performance and computational complexity of the retrained architecture, select the most optimized architecture as the search result.

[0045] In some embodiments, a string of numbers is used to encode the neural network architecture, each architecture being represented by a series of indices corresponding to specific unit configuration options, such as kernel size, expansion ratio, filter size, and block repetition count. When providing hints to the LLM, it is made to see only the sequence of numbers, rather than the specific technical details of each index. By hiding the specific technical details, the LLM is allowed to reason and suggest new architecture configurations based on historical data and pattern recognition. In each iteration, the LLM proposes new architecture configurations based on historical experimental logs and reward feedback to optimize model performance.

[0046] In some embodiments, the LLM output architecture configuration is required to follow a set format by means of the numeric string, that is, using a prefix and a series of placeholders such as "?" to represent a specific configuration option in the architecture. The placeholders are used to be replaced by a specific number, which is limited to a set of numbers such as [0,1,2], representing different configuration options. Thus, each configuration option of the neural network architecture, such as the kernel size, expansion ratio, etc., is encoded by a number.

[0047] In some embodiments, the LLM employs GPT-4, wherein: GPT-4 is used as a black-box optimizer, specifically executing a sample-evaluation-update loop for architecture search; a prompting strategy for GPT-4 is configured, including generating a prompt template to guide GPT-4 in proposing new architectures; a proxy task is trained for each new architecture proposed by GPT-4, and a reward score reflecting its performance is calculated; rewards are fed back and GPT-4's context memory is updated, including feeding back architecture-reward pairs to GPT-4 in a specific format; efficiency optimization techniques are applied, selecting the top K architecture-reward pairs with the highest historical rewards for in-depth search; during search iterations, if GPT-4 proposes duplicate architectures, the search process is optimized by adjusting the prompts and requesting GPT-4 to provide alternative architectures. During search iterations, it is ensured that GPT-4's context memory contains the latest architecture-reward pairs to facilitate efficient subsequent iterations. It should be understood that the LLM in this invention is not limited to GPT-4 and can also be other LLMs.

[0048] In some embodiments, the greedy search algorithm includes:

[0049] A1. Initialize search iterations: Set the number of search iterations and prepare historical experiment logs;

[0050] A2. Select the top experiments: At the beginning of each iteration, select the top k experiments from the historical experiment log;

[0051] A3. Generate Hints: Generate hints for guiding the Language Model (LLM) based on the selected top experiment;

[0052] A4. Sample new model: In the loop, use LLM to generate a new model architecture based on the current prompts;

[0053] A5. Check for duplicates: If the generated model architecture is the same as the architecture in a historical experiment, update the prompt and repeat step A4;

[0054] A6. Break the loop: If the generated model architecture is new, then break the loop;

[0055] A7. Obtaining Rewards: Train and evaluate the newly sampled model on the agent task to obtain a reward score for its performance;

[0056] A8. Update historical experiment log: Add the new model and its reward score to the historical experiment log;

[0057] A9. End of iteration: Repeat steps A2 to A8 until the preset number of search iterations is reached.

[0058] In some embodiments, the retraining strategy includes employing a two-stage approach;

[0059] In the first phase, a proxy task is implemented to quickly evaluate and screen potential architecture configurations. This proxy task has similar performance to the target task, but significantly reduces the computational resource requirements. This can be achieved by using a reduced training set and shortening the training cycle.

[0060] Phase 2: After the search phase for the proxy task is completed, the top N architectures with the best performance are selected from the search history based on the performance of the architecture on the proxy task and the similarity with the preset reference model in floating-point operations (FLOPs). The selected top candidate architectures will be retrained from scratch on the complete target task in the second phase to obtain the final performance evaluation of each architecture.

[0061] In some embodiments, the method is applied to neural network architecture search for image classification tasks, specifically including:

[0062] The search space is defined as follows: The search space introduced by MNASNet is adopted and simplified to control the computational cost. The search space is designed for image classification tasks and consists of multiple configuration units. Each unit represents a building block of the network. Its configuration parameters include kernel size, expansion ratio, filter size and repetition count. Each parameter is selected from a predefined set of options to suit the needs of image classification tasks.

[0063] Configuration parameter options: The kernel size parameter is selected from a set of predefined options such as 3, 5, and 7 to determine the kernel size for convolution operations; the expansion ratio parameter is selected from another set of predefined options such as 4, 6, and 8 to represent the channel ratio between the network bottleneck input and intermediate layers; the filter size parameter searches within a preset range such as 0.75, 1.0, and 1.25 to control the relative number of convolutional filters; the repetition count parameter is selected from a set of predefined options such as 3, 4, and 5 to determine the number of times a module in the network structure is repeated.

[0064] The search phase involves training the sampled architecture on a selected dataset designed for image classification tasks, such as the CIFAR-100 dataset. The training period is fixed at multiple preset periods, such as 10 epochs.

[0065] Calculate the reward function: For each architecture A, define a reward function R(A), which... The number balances the model The reward function considers accuracy and computational complexity in image classification tasks. The accuracy S on the proxy task, the number of floating-point operations FLOPs of the sampling architecture, the preset target FLOPs F, and the trade-off between accuracy and computational complexity;

[0066] Architecture selection and retraining: After the search process is completed, the best-performing architecture is selected based on the architecture performance evaluated by the reward function, and then retrained on the corresponding full datasets such as CIFAR-100 and ImageNet datasets to obtain the final performance evaluation of each architecture in the image classification task.

[0067] In some embodiments, the method for architecture optimization of the deep learning model DiT for image generation specifically includes:

[0068] Constructing the search space: Based on the original DiT architecture, a search space is constructed, which is divided into multiple stages. Each stage represents a building block or layer in the model. Each stage allows customization in terms of conditional input, convolution in attention, LayerNorm position, number of channels, and number of repeating blocks.

[0069] Define customization options: For conditional input, choose adaptive LayerNorm, cross-attention mechanism, or connection; in the attention mechanism, decide whether to apply convolution; the LayerNorm position can be set to front LN, back LN, or sub-LN; the number of channels and the number of repeating blocks are selected according to the search space settings;

[0070] Implement the proxy task: Train a small DiT model using a proxy task, which is performed on a downsampled selected image dataset such as the ImageNet 64×64 dataset;

[0071] Initial data preprocessing: On the selected image dataset, first train a variational autoencoder (VAE) to encode the images;

[0072] Execute architecture search: Perform DiT architecture search multiple times, such as 2000 iterations, with the number of training steps set in each iteration, such as 25000 training steps;

[0073] Model performance was evaluated by computing the Fréchet Inception Distance (FID) on a selected image validation set and using a reward function R(A) to quantify model performance, which takes into account the model’s FID score and FLOPs, as well as the trade-off between accuracy and computational complexity.

[0074] Selection and retraining architectures: After the search phase is completed, select the top 5 architectures with the lowest FID and perform sufficient classifier-guided training, such as 400k steps, on the selected image dataset to achieve sufficient training depth and breadth.

[0075] Evaluation and scaling of the model: The trained model is evaluated using standard evaluation tools. The best-performing model is then selected and scaled up to a higher resolution image dataset, such as ImageNet 256×256, for training to evaluate its scalability and generation quality.

[0076] In some embodiments, the creation of the prompt includes the following process:

[0077] Define system instructions: Establish the main responsibilities of the LLM and provide a concise overview of instructions to guide the LLM in examining and analyzing the provided empirical results to identify patterns and insights;

[0078] Analysis of empirical results: Based on system instructions, LLM analyzes historical experimental data to search for potential model structures that are consistent with the analysis results and meet the preset search criteria;

[0079] Record experimental logs: Create a concise record of historical experimental results, but do not include detailed information directly related to the target model or task, to ensure that LLM is not unduly influenced by prior knowledge when searching for model architectures;

[0080] Implement format specifications: Establish clear output format specifications to ensure that the output generated by LLM is consistent and clear, in order to meet specific requirements and standards;

[0081] Minimize ambiguity: By establishing and implementing format specifications, reduce ambiguity in the output and improve the readability and usability of LLM output results;

[0082] Feedback and iteration: Based on the output of LLM, feedback and iterative optimization are performed to further improve the accuracy and efficiency of model search.

[0083] To address the challenges of existing technologies for Neural Architecture Search (NAS) using Large Language Models (LLMs), this invention proposes a black-box neural architecture search method, UniSearch, which maximizes the inference capabilities of LLMs such as GPT-4 through a unified prompting strategy. Traditional LLM-based NAS methods heavily rely on extensive expert knowledge and manual prompting adjustments across different search tasks. Compared to traditional LLM-based NAS methods, UniSearch fully leverages the inherent inference capabilities of LLMs, thereby improving its transferability and performance. Specifically, the unified prompting strategy places the LLM as a black-box optimizer within a sample-evaluation-update loop. A consistent prompting structure is applied to the LLM using numeric strings across different search tasks or datasets. In each search iteration, the LLM is required to propose a new architecture based on historically sampled architectures. The proposed architecture is trained on a proxy task to obtain a reward score reflecting its effectiveness. The architecture-reward pairs are then fed back to the LLM, updating its contextual memory for use in the next iteration. Inspired by greedy search algorithms, this invention further improves efficiency by providing the LLM with only the top K, such as the top 200, historical architecture-reward pairs, which have the highest rewards. This strategy encourages the LLM to focus on potentially rewarding regions in the search space, accelerating the process while reducing reliance on extensive contextual information. Since the search space in NAS problems is typically very large, this invention effectively reduces the total number of samples required for architecture search.

[0084] To verify the effectiveness of this invention, UniSearch has been experimentally applied to various architecture search problems. First, experiments were conducted on traditional image classification tasks on CIFAR-100 and ImageNet. Compared to existing NAS methods, UniSearch achieves better performance with fewer search samples. To further demonstrate the generalization ability of UniSearch, this invention applies UniSearch to a less-explored NAS task: searching for better Diffusion Transformer (DiT) architectures for image generation. Specifically, a new search space is constructed based on DiT architectures, and then UniSearch is applied to find new architectures with lower FID scores. Experimental results show that UniSearch still achieves better performance even without introducing any task-specific optimizations. These results demonstrate that UniSearch achieves superior performance with fewer samples compared to existing NAS methods.

[0085] The following describes specific embodiments of the present invention.

[0086] Large Language Models (LLMs) have been pre-trained on massive datasets, demonstrating robust performance in few-shot learning scenarios. Research focusing on LLMs has surged in recent years. These models excel at understanding broad contexts and generating responses based on user queries, making them versatile for various applications. Subsequent research has explored fine-tuning open-source Llama models on specific datasets, enhancing LLMs' capabilities in areas such as encoding, instruction following, and multimodal input processing. Leveraging the scalable properties of the Transformer architecture, LLMs are equipped with extensive knowledge bases, enabling them to solve complex challenges in mathematical and physical domains. The method proposed in this invention fully utilizes the potential of LLMs to optimize neural network architectures, relying not on specific knowledge or past experience, but rather on their ability to accurately absorb contextual information and infer patterns through logical reasoning.

[0087] In some embodiments, the method uses LLM for Neural Architecture Search (NAS), utilizing only experimental logs that provide numerical strings (or sequences of numbers) of operational configurations. By intentionally withholding specific information about neural network blocks, the method ensures that LLM generates novel architectures based solely on patterns identified in past experiments.

[0088] Unit-based neural network architecture search

[0089] In this embodiment, GPT-4 is used to explore novel neural network architectures. It demonstrates how LLM effectively masters and generates diverse and innovative model architectures. Unlike RL-based methods, LLM-based methods do not require tuning and provide a more intuitive architecture sampling mechanism.

[0090] method

[0091] UniSearch employs a black-box architecture search method using GPT-4 as the optimizer. Unlike methods that rely on expert knowledge or hints for tuning, UniSearch uses numeric strings to provide a consistent architecture representation of GPT-4 across different search settings. Furthermore, inspired by greedy search algorithms, it ranks historical search samples and only provides the top 200 samples with the highest rewards to GPT-4, significantly accelerating search efficiency.

[0092] UniSearch

[0093] Architecture representation

[0094] GPT-4 simplifies the representation of architectures, eliminating the need for network architecture expertise. Unlike previous methods (Zheng et al., 2023), this invention does not provide complex architectural details (i.e., code specifying the architecture), but instead encodes the architecture using simple numeric strings when prompted with GPT-4. Taking the MNAS search space (Tan et al., 2019) as an example, each architecture is compiled into a series of indices. Each index in this sequence represents a specific choice of cell configuration, such as... Figure 1 As shown. Importantly, this invention only exposes GPT-4 to these numerical sequences, rather than the technical details of each index. This strategy is adopted to fully leverage GPT-4's reasoning capabilities, enabling it to discover patterns and relationships within the search space without being biased by existing architectural knowledge.

[0095] Search Algorithm

[0096] Using GPT-4 as a black-box optimizer, architecture search is performed in a direct sample-evaluation-update loop. Specifically, in each search iteration, GPT-4 is prompted to propose new architecture configurations (as numeric strings) based on historically sampled architectures in its context memory. If GPT-4 proposes a duplicate architecture, the conversation continues, requesting an alternative architecture. The proposed architecture is trained on a proxy task to obtain a reward score reflecting its performance. The architecture-reward pair, such as "cell=20020120101010010102,reward=56.02", is then fed back to GPT-4 to update its context memory in preparation for the next iteration.

[0097] To further improve search efficiency, a method inspired by the greedy search algorithm is introduced. Instead of providing GPT-4 with all previous architecture-reward pairs, only the top 200 pairs with the highest rewards are provided. This strategy focuses GPT-4's attention on promising regions of the search space, accelerating the search process while requiring a smaller context window. Experiments show that this focused sampling significantly improves search efficiency and leads to better architecture performance. A complete exemplary search algorithm for UniSearch is outlined in the following pseudocode.

[0098]

[0099] Retraining strategy

[0100] To manage the computational cost of the search phase, a two-stage approach commonly used in NAS research (Tan et al., 2019) is employed. First, a surrogate task is used, which approximates the target task but requires significantly fewer computational resources. This surrogate task typically involves a reduced training set and fewer training epochs. Once the search phase using the surrogate task is complete, the top 5 performing architectures are selected from the search history. These architectures are chosen based on their performance on the surrogate task and their similarity to the reference model in terms of FLOPs. Finally, these top candidate architectures are retrained from scratch on the full target task to obtain their final performance evaluation. This two-stage strategy allows for a broader exploration of the architecture space within a manageable computational budget.

[0101] Application of UniSearch in image classification

[0102] To evaluate UniSearch for image classification, the search space introduced in MNASNet (Tan et al., 2019) was used, with some simplifications to manage computational costs. Specifically, the architecture in this search space consists of 5 units, each of which can be configured using the following parameters:

[0103] - Kernel size: Determines the kernel size for the convolution operation, select from 3, 5, and 7.

[0104] - Scalability ratio: Represents the ratio of the channels between the bottleneck input and the intermediate layer, selectable from 4, 6, and 8.

[0105] - Filter size: Controls the number of convolutional filters, searching within the range of 0.75, 1.0, and 1.25 of the reference size.

[0106] - Number of repetitions: Determines the number of times a block is repeated, choosing from 3, 4, or 5.

[0107] During the search phase, the sampled architectures are trained on the CIFAR-100 dataset for 10 epochs. The reward function R(A) for architecture A balances accuracy and computational complexity.

[0108]

[0109] Where S represents the accuracy on the proxy task, \(F'\) is the FLOPs of the sampled architecture, F is the target FLOPs, and λ is a penalty coefficient that controls the trade-off between accuracy and computational complexity. This penalty encourages UniSearch to discover architectures with competitive accuracy while maintaining a reasonable computational budget. After the search process is complete, the best-performing architecture is selected and retrained on the full CIFAR-100 and ImageNet datasets to obtain the final performance evaluation.

[0110] Application of UniSearch in DiT

[0111] To demonstrate UniSearch's generalization capabilities beyond image classification, it was applied to the task of optimizing the Diffusion Transformer (DiT) architecture for image generation (Peebles and Xie, 2023). A search space inspired by the original DiT architecture was constructed and divided into four stages. Each stage allows customization in the following five aspects:

[0112] -Conditional input: Adaptive LayerNorm, cross-attention mechanism, or connection.

[0113] - Convolution in attention: Whether to apply convolution within the attention linear mapping layer, following CvT (Wu et al., 2021).

[0114] -LayerNorm positions: front LN, back LN (Xiong et al., 2020) and child LN (Wang et al., 2022).

[0115] - Number of channels: The number of channels in each stage.

[0116] - Number of repeated blocks: The number of times a block is repeated within each stage.

[0117] A proxy task was employed to train a small DiT model on the downsampled ImageNet 64×64 dataset. Initially, a variational autoencoder (VAE) was trained on the 64×64 resolution dataset (Kingma and Welling, 2013). Then, a DiT architecture was searched for 2000 iterations. In each iteration, the model was trained for 25,000 steps. The model was evaluated by computing the Fréchet Inception Distance (FID) on the ImageNet 64×64 validation set (Heusel et al., 2017). The reward R(A) can be expressed as:

[0118]

[0119] Where \(F'\) and F are the FLOPs of the sampled model and the target model, respectively, S represents the FID score on the proxy task, and λ is a penalty coefficient that controls the tradeoff between accuracy and computational complexity. After the search phase, the top 5 architectures with the lowest FID were retrained. These sampled models were trained for 400k steps on the ImageNet 64×64 dataset without classifier guidance and evaluated under the MIT license using the ADM evaluation suite (Dhariwal and Nichol, 2021). The best-performing model was then selected and scaled up for training on the ImageNet 256×256 dataset to evaluate its scalability and generative quality.

[0120] experiment

[0121] Experiment setup and implementation

[0122] The MobileNetV2-based NAS sets up a search space based on MNASNet (Tan et al., 2019) and uses a classification task on the CIFAR-100 dataset as a proxy task to find the optimal architecture within the predefined search space. Given the relatively low resolution of the CIFAR-100 images, fewer downsampling operations are typically used during training. In this embodiment, downsampling is disabled in the second stage.

[0123] The GPT-4 sampled architectures were trained using the SGD optimizer for 10 epochs. The initial learning rate was set to 0.1, with cosine annealing scheduling, momentum of 0.9, and weight decay of 0.0005. The batch size was 128. The reward for each sampled architecture was calculated using function 1, where S represents the classification accuracy on the validation set. The target FLOPs F were set to 80, and the coefficient λ was set to 0.05. During the search phase, 500 unique model architectures were sampled for GPT-4. The top 5 models were completely retrained to determine the best-performing architecture, which was then transferred to the ImageNet classification task. For retraining, 200 epochs were trained on the CIFAR-100 training set, maintaining the same settings as the proxy task.

[0124] DiT NAS

[0125] When UniSearch is applied to the DiT architecture, a proxy task is used to train a small model with computational complexity close to DiT-S / 2 on the ImageNet 64×64 dataset. Initially, a KL-regularized autoencoder is trained on the 64×64 resolution dataset to encode images into 8×8 latent blocks. Utilizing the codebase of Rombach et al. (2022), a... The KL loss weight is 6×10^-6, the discrimination loss weight is 0.5, and the discrimination loss is included at step 50,001.A total of 2000 models were sampled during the search phase, each trained for 25k iterations on 8 V100 GPUs. The classifier-guided ratio was set to 1.5, and the global batch size was 256. The reward for each sampled model was calculated based on the model's FLOPs and the FID between the generated 50k images and images in the ImageNet 64×64 validation set. After the search phase, the top 5 architectures with the lowest FIDs were retrained. These sampled models were trained on the ImageNet 64×64 dataset for 400k steps without classifier guidance and evaluated using ADM's evaluation suite. The best-performing model was then selected and scaled up for training on the ImageNet 256×256 dataset.

[0126] Main results

[0127] Image classification

[0128] Table 1 shows the state-of-the-art performance of the sampled models fully trained on the CIFAR-100 dataset. UniSearch achieved a top-1 classification accuracy of 72.55%, outperforming the baseline MobileNetV2 model by 0.56%. Furthermore, 500 models were sampled using a random search algorithm and evaluated in the same settings described above. It can be observed that the random search algorithm is ineffective in the search space, which is approximately 3 × 10^9 in size. UniSearch outperformed the traditional random search algorithm by 0.56%, demonstrating its effectiveness.

[0129] Table 1: Image classification performance on the CIFAR100 dataset

[0130]

[0131] To further validate the generalization performance of the search model, it was transferred to the ImageNet dataset. As shown in Table 2, the transferred model achieved a 0.42% improvement in top-1 classification accuracy on the validation set compared to the original MobileNetV2.

[0132] Table 2: Transfer learning performance on the ImageNet dataset

[0133]

[0134] Image generation

[0135] Table 3 shows the performance of UniSearch in unconditional image generation after retraining on the ImageNet 64×64 dataset. The model has 26.3M parameters and 0.351G FLOPs. The FID score was reduced to 61.73. The model was then... After training at a resolution of 256×256The model's performance failed to maintain its previous superiority and was slightly inferior to the baseline model. This is attributed to the difference in model construction patterns between 64×64 and 256×256 resolutions. However, it was found that some construction principles of the best model at 64×64 resolution can be transferred to 256×256 resolution to improve performance. A variant of the top model sampled on the ImageNet 64×64 dataset was developed by searching. This variant, named UniSearch-B (as shown in Table 3), retains the channel number configuration and block repetition configuration derived from the search process while keeping all other aspects of the original model design unchanged. Despite having the same number of parameters and computational cost, UniSearch-B outperformed the baseline DiT-S / 2 model, achieving an FID score of 65.97 and an Inception score of 21.03.

[0136] Table 3: Performance of unconditional image generation on the ImageNet dataset

[0137]

[0138] NAS-Bench-201

[0139] To benchmark UniSearch's performance against other existing NAS algorithms, UniSearch was evaluated on the widely used NAS-Bench-201 dataset (Dong and Yang, 2020), which provides pre-computed results for 15,625 architectures pre-trained and evaluated on three different image classification datasets. Five experiments were performed on each dataset within the search space defined by NAS-Bench-201. In each run, GPT-4 was prompted to search for 50 unique model architectures. During these searches, GPT-4 sampled models based on the validation accuracy of historical experiments in the same round. The average of the highest validation accuracies achieved across the five runs, along with the corresponding average test accuracy, is reported. As shown in Table 4, UniSearch achieves competitive performance on all datasets while requiring fewer queries compared to most existing methods. Specifically, UniSearch achieved a validation accuracy of 91.31% and a test accuracy of 93.90% on CIFAR-10, 72.46% and 72.59% respectively on CIFAR-100, and 45.94% and 45.56% respectively on ImageNet16-120, using only 50 queries.

[0140] Table 4: Performance on NAS-Bench-201

[0141]

[0142] Ablation Research

[0143] Hint Strategy

[0144] In this section, GPT-3.5 is used to investigate the impact of providing only the top 200 architecture-reward pairs to LLMs.

[0145] like Figure 2 As shown, when only the first 200 architecture-reward pairs are provided to GPT-3.5, the rewards for the sampled model architectures continuously improve and eventually stabilize at a relatively high level. Conversely, when all architecture-reward pairs are provided to GPT-3.5, performance continuously declines. This indicates that the prompting strategy of this invention effectively guides the GPT model to focus its attention on promising regions in the search space. As shown in Table 5, providing only the first 200 architecture-reward pairs to GPT-3.5 leads to performance improvements.

[0146] Table 5: Performance when using different historical samples in context on CIFAR100

[0147]

[0148] Different GPT models

[0149] The performance of different versions of the GPT model on NAS-Bench-201 was evaluated. As shown in Table 6, the latest version, GPT-4o, performed best, while GPT-3.5 performed worst. This indicates that UniSearch's performance can be improved as LLMs acquire more general knowledge and stronger reasoning capabilities.

[0150] Table 6: Performance of different versions of the GPT model on NAS-Bench-201

[0151]

[0152] Example

[0153] A Tips Overview

[0154] like Figure 3As shown, the prompts are divided into three key components: system instructions, experimental log, and format specifications. The system instructions section concisely outlines the main responsibilities of the LLM. Specifically, the LLM is instructed to examine the provided empirical results to identify patterns and insights. Based on the analysis, the LLM searches for potential model architectures that are consistent with the identified patterns and meet specific search criteria. The experimental log section provides the LLM with a concise record of past experimental results. This section does not include any detailed information related to the target model or task, guiding the LLM to search for model architectures without revealing knowledge (e.g., the LLM may have seen results from NAS benchmarks in its training dataset). The format specifications section aims to ensure that the output generated by the LLM is consistent, clear, and meets requirements. By providing explicit format specifications, it aims to minimize ambiguity and improve the usability of the LLM output.

[0155] B-sampling DiT architecture rewards

[0156] Figure 4 The experimental loss distribution of the sampling model is shown, where the reward is the negative of the loss.

[0157] In summary, this invention proposes UniSearch, an innovative black-box architecture search method that utilizes Large Language Models (LLMs) for black-box structure search and effectively leverages the inference capabilities of GPT-4 through a unified prompting strategy. This method addresses the scalability and generalization challenges of traditional NAS methods by minimizing the need for extensive expert knowledge and manual prompting adjustments. Extensive experiments on image classification and image generation tasks demonstrate the effectiveness of UniSearch. Furthermore, UniSearch demonstrates its generalization ability by successfully identifying more efficient DiT architectures for image generation and achieving lower FID scores without task-specific optimization. Future work will further optimize the unified prompting strategy and expand the application of UniSearch to other challenging domains. In addition, plans are underway to integrate more advanced techniques to handle larger and more complex search spaces, enhancing the potential of LLMs in neural network architecture search.

[0158] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0159] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0160] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0161] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk drive or magnetic tape drive. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0162] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0163] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0164] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0165] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] Alternatively, if the integrated units of the present invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the present invention is implemented. The technical solution in the example is essentially, or rather, a modification of existing technology. The contribution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0167] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0168] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0169] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0170] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

[0171] References:

[0172] Rohan Asthana, Joschua Conrad, Youssef Dawoud, Maurits Ortmanns, and Vasileios Belagiannis. 2024. Multiconditional graph diffusion for neural architecture search. arXiv preprint arXiv:2403.06020.

[0173] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901.

[0174] Angelica Chen, David Dohan, and David So. 2024. Evoprompting: Language Models for Code-Level Neural Architecture Search. Advances in Neural Information Processing Systems, 36.

[0175] Yutian Chen, Xingyou Song, Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Marc'Aurelio Ranzano, et al. 2022. Learning a general hyperparameter optimizer using a transformer. Advances in Neural Information Processing Systems, 35:32053–32068.

[0176] Aakanksha Chowdhery Sharan Narang, Jacob Devlin, Maarten Bosma Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Modeling with Path-Extended Languages. Journal of Machine Learning Research, 24(240):1–113.

[0177] Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models outperform GANs in image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794.

[0178] Xuanyi Dong and Yi Yang. 2020. Nas-bench-201: Expanding the scope of the search for reproducible neural architectures. arXiv preprint arXiv:2001.00326.

[0179] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. GANs trained by dual-timescale update rules converge to local Nash equilibrium. Advances in Neural Information Processing Systems, 30.

[0180] Jonathan Ho and Tim Salimans. 2022. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598.

[0181] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. 2019. Search MobileNetV3 in IEEE / CVF International Conference on Computer Vision. Conference proceedings, pp. 1314–1324.

[0182] Sian-Yao Huang and Wei-Ta Chu. 2021. Search by Generation: Flexible and Efficient One-Off NAS with Architecture Generator. In the proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 983–992.

[0183] Ganesh Jawahar, Muhammad Abdul-Mageed, Laks vs. Lakshmanan, and Dujian Ding. 2023. The LLM performance predictor is a good initializer for architecture search. arXiv preprint arXiv:2310.16712.

[0184] Diederik P Kingma and Max Welling. 2013. Autoencoded Variational Bayes. arXiv preprint arXiv:1312.6114.

[0185] Hanxiao Liu, Karen Simonyan, and Yiming Yang. 2018. Darts: Search for Differentiable Architectures. arXiv preprint arXiv:1806.09055.

[0186] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024a. Visual instruction adjustment. Advances in Neural Information Processing Systems, 36.

[0187] Jihao Liu, Xin Huang, Guanglu Song, Hongsheng Li, and Yu Liu. 2022. Uninet: Unified Architecture Search, Including Convolutions, Transformers, and MLPs. European Conference on Computer Vision, pp. 33–49. Springer.

[0188] Jihao Liu, MingZhang, Yangting Sun, Boxiao Liu, GuangluSong, Yu Liu He Hongsheng Li. 2021. FNAS: Fast neural architecture search for perceiving uncertainty. arXiv preprint arXiv:2105.11694.

[0189] Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. 2024b. Augmented Bayesian Optimization for Large Language Models. arXiv preprint arXiv:2402.03921.

[0190] Jovita Lukasik, Steffen Jung, and Margret Keuper. 2022. Learning Focus - Generative NAS is Surprisingly Efficient. European Conference on Computer Vision, pp. 257–273. Springer.

[0191] Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. 2021. Transformers can be used for Bayesian inference. arXiv preprint arXiv:2112.10510.

[0192] OpenAI.2023.https: / / openai.com / index / gpt-4 / ., Accessed on: 2024-05-12.

[0193] William Peebles and Saining Xie. 2023. Scalable diffusion model with converter. IEEE / CVF International Conference on Computer Vision, pp. 4195–4205.

[0194] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. 2019. Regularized Evolution of Image Classifier Architecture Search. AAAI Conference on Artificial Intelligence, Vol. 33, pp. 4780–4789.

[0195] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Ommer.2022. High-resolution image synthesis and latent diffusion models. IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695.

[0196] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. Codellama: An Open Foundation Model for Code. arXiv preprint arXiv:2308.12950.

[0197] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved GANs training techniques. Advances in Neural Information Processing Systems, 29.

[0198] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. MobileNetV2: Inverted Residuals and Linear Bottlenecks. In the proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[0199] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithm. arXiv preprint arXiv:1707.06347.

[0200] Xingyou Song, Yingtao Tian, ​​Robert Tjarko Lange, Chansoo Lee, Yujin Tang, and Yutian Chen. 2024. Position Paper: Black-box Optimization Using a Baseline Model: Strengths, Challenges, and Future Directions. arXiv preprint arXiv:2405.03547.

[0201] Kenneth O. Stanley and Risto Miikkulainen. 2002. Evolutionary Neural Networks by Enhancing Topological Structure. Evolutionary Computation, 10(2):99–127.

[0202] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. 2019. Mnasnet: Platform-Aware Neural Architecture Search for Mobile Devices. In the proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 2820–2828.

[0203] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: A llama model that follows instructions. https: / / github.com / tatsu-lab / stanford_alpaca.

[0204] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: A family of high-energy multimodal models. arXiv preprint arXiv:2312.11805.

[0205] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: An Open and Efficient Foundational Language Model. arXiv preprint arXiv:2302.13971.

[0206] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open Foundations and Fine-tuning of Chat Models. arXiv preprint arXiv:2307.09288.

[0207] Ashish Vaswani,Noam Shazeer,Niki Parmar,Jakob Uszkoreit,Llion Jones,Aidan N Gomez, Kaiser and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems, 30.

[0208] Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj, Saksham Singhal, Alon Benhaim, et al. 2022. Fundamental Transformers. arXiv preprint arXiv:2210.06423.

[0209] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. 2021. CvT: Introducing Convolution into Visual Transformers. IEEE / CVF International Conference on Computer Vision, pp. 22–31.

[0210] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuii Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. 2020. Layer Normalization in Transformer Architectures. International Conference on Machine Learning, pp. 10524–10533. PMLR.

[0211] Mingkai Zheng, Xiu Su, Shan You, Fei Wang, Chen Qian, Chang Xu, and Samuel Albanie. 2023. Can GPT-4 perform neural architecture search? arXiv preprint arXiv:2304.10970.

[0212] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing Visual-Language Understanding with Advanced Large-Scale Language Models. arXiv preprint arXiv:2304.10592.

[0213] Baret Zoph and Quoc V Le. 2016. Neural architecture search using reinforcement learning. arXiv preprint arXiv:1611.01578.

[0214] Baret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. 2018. Learning a transferable architecture for scalable image recognition. In the proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697–8710.

Claims

1. A black-box neural network architecture search method, characterized in that, Comprise: Encoding neural network architectures using numerical strings; Using a large language model (LLM) as a black-box optimizer and building a sample-evaluate-update loop for iteratively searching and optimizing architectures; Building a consistent prompt structure to present numerical sequences to the LLM to leverage its reasoning capabilities, enabling the LLM to discover patterns and relationships within the search space based on numerical sequences rather than relying on pre-existing architectural knowledge; Training a proxy task for each new architecture configuration proposed by the LLM and calculating a reward score reflecting its performance; Feeding back each architecture-reward pair to the LLM to update its contextual memory and provide an information base for the next step of architecture search; Searching and selecting the top K architecture-reward pairs with the highest rewards in history through a search algorithm to concentrate the search in more promising areas; After completing the search phase, retraining the top architectures on the full target task to obtain the final performance evaluation; Selecting the optimal architecture as the search result based on the retrained architecture performance and computational complexity.

2. The black-box neural network architecture search method of claim 1, wherein: Numerical strings are used to encode neural network architectures, with each neural network architecture represented by a series of indices corresponding to specific unit configuration options; when providing prompts to the LLM, the LLM only sees numerical sequences rather than the specific technical details of each index, allowing the LLM to infer and suggest new architecture configurations based on historical data and pattern recognition by hiding specific technical details; In each iteration, the LLM proposes new architecture configurations based on historical experiment logs and reward feedback to optimize model performance.

3. The black-box neural network architecture search method of claim 2, wherein, The configuration options include kernel size, expansion ratio, filter size, and block repetition number.

4. The black-box neural network architecture search method of any one of claims 1 to 3, wherein, The architecture configuration output by the LLM through the numerical string follows a set format, using a prefix and a series of placeholders immediately following the prefix to represent a specific configuration option in the architecture, with the placeholders replaced by a specific number that defines a different configuration option within a set of numbers; thus, each configuration option of the neural network architecture is encoded by a number.

5. The black-box neural network architecture search method of any one of claims 1 to 4, wherein, The LLM uses GPT-4, wherein: GPT-4 is used as a black-box optimizer to specifically implement the sample-evaluate-update loop for architecture search; The prompt strategy of GPT-4 is configured, including generating a prompt template for guiding GPT-4 to propose new architectures; A proxy task is trained for each new architecture configuration proposed by GPT-4, and a reward score reflecting its performance is calculated; The reward is fed back and the context memory of GPT-4 is updated, including feeding back the architecture-reward pair to GPT-4 in a specific format; Efficiency optimization techniques are applied to select the top K architecture-reward pairs with the highest rewards in history for in-depth search; During the search iteration, if GPT-4 proposes a duplicate architecture, the search process is optimized by adjusting the prompt and requesting GPT-4 to provide alternative architectures. During the search iteration, ensure that the context memory of GPT-4 contains the latest architecture-reward pair to facilitate effective next iteration search.

6. The black-box neural network architecture search method of any one of claims 1 to 5, wherein, The architecture search is performed by a heuristic greedy search algorithm, which includes the following steps: A1. Initialize search iteration: set the number of search iterations and prepare the historical experiment log; A2. Select top experiments: at the beginning of each iteration, select the top k experiments from the historical experiment log; A3. Generate prompt: generate a prompt for the language model (LLM) based on the selected top experiments; A4. Sample new model: in a loop, use the LLM to generate a new model architecture based on the current prompt; A5. Check for duplicates: if the generated model architecture is duplicated in the historical experiments, update the prompt and repeat step A4; A6. Break the loop: if the generated model architecture is new, break out of the loop; A7. Get reward: train and evaluate the newly sampled model on the agent task to obtain its performance reward score; A8. Update historical experiment log: add the new model and its reward score to the historical experiment log; A9. End of iteration: repeat steps A2 to A8 until the preset number of search iterations is reached.

7. The black-box neural network architecture search method of any one of claims 1 to 6, wherein, The retraining strategy includes a two-stage approach: In the first stage, an agent task is implemented to quickly evaluate and screen potential architecture configurations; this agent task is similar in performance to the target task but significantly reduces the computational resource requirements, which can be achieved by using a reduced training set and shortening the training period; In the second stage, after the search phase of the agent task is completed, the top N architectures with the best performance are selected from the search history based on their performance on the agent task and their similarity to the preset reference model in terms of floating-point operation count (FLOPs); The top candidate architectures selected will be retrained from scratch in the second stage on the full target task to obtain the final performance evaluation of each architecture.

8. The black-box neural network architecture search method of any one of claims 1 to 7, wherein, The method is applied to neural network architecture search for image classification tasks, which includes the following steps: Define search space: adopt the search space introduced by MNASNet and simplify it to control the computational cost; this search space is designed for image classification tasks and consists of multiple configuration units, each representing a building block in the network, with configuration parameters including kernel size, expansion ratio, filter size, and repetition number, each parameter selected from a predetermined set of options to adapt to the needs of image classification tasks; Configure parameter options: the kernel size parameter is selected from a set of predefined options to determine the kernel size of the convolution operation; the expansion ratio parameter is selected from another set of predefined options to represent the channel ratio between the network bottleneck input and the intermediate layer; the filter size parameter is searched within a preset range to control the relative number of convolution filters; the repetition number parameter is selected from a set of predefined options to determine the number of repetitions of a certain module in the network structure; Perform search phase: train the sampled architecture on the selected dataset designed for image classification tasks, with a fixed training period of multiple preset epochs. Computing reward function: for each architecture A, define a reward function R(A) that balances the model's accuracy and computational complexity in image classification tasks; the reward function considers the accuracy S on the proxy task, the number of floating-point operations FLOPs of the sampled architecture, the preset target FLOPs F, and the trade-off between accuracy and computational complexity; Select and retrain architecture: after the search process is completed, select the best-performing architecture according to the architecture performance evaluated by the reward function, and retrain it on the corresponding complete dataset to obtain the final performance evaluation of each architecture in the image classification task.

9. The black-box neural network architecture search method of claim 8, wherein, The reward function R(A) of architecture A is: where S represents the accuracy on the proxy task, F' is the FLOPs of the sampled architecture, F is the target FLOPs, and λ is the penalty coefficient that controls the trade-off between accuracy and computational complexity.

10. The black-box neural network architecture search method of any one of claims 1 to 7, wherein, The method is applied to the architecture optimization task of the deep learning model diffusion Transformer DiT model for image generation, and specifically includes: Building search space: based on the original DiT architecture, a search space is constructed, which is divided into multiple stages, each stage representing a building block or layer in the model, and each stage allows customization in terms of conditional input, convolution in attention, LayerNorm position, channel number, and number of repeated blocks; Define customization options: in terms of conditional input, choose adaptive LayerNorm, cross-attention mechanism, or connection; in the attention mechanism, decide whether to apply convolution; LayerNorm position can be set to pre-LN, post-LN, or sub-LN; the channel number and the number of repeated blocks are selected according to the settings of the search space; Implement proxy task: train a small DiT model using a proxy task, which is performed on a down-sampled selected image dataset; Initialize data preprocessing: first train a variational autoencoder VAE on the selected image dataset to encode images; Perform architecture search: perform multiple iterations of DiT architecture search, with each iteration training the model for a set number of training steps; Evaluate model performance: evaluate the model by calculating the Fréchet Inception Distance (FID) on the selected image validation set, and use the reward function R(A) to quantify the model performance, which considers the FID score and FLOPs of the model, as well as the trade-off between accuracy and computational complexity; Select and retrain architecture: after the search phase is completed, select the top architectures with the lowest FID, and perform full-step classifier-free guidance training on the selected image dataset to achieve sufficient training depth and breadth; Evaluate and extend the model: evaluate the trained model using standard evaluation tools, then select the best-performing model and extend it to higher-resolution image datasets for training to evaluate its scalability and generation quality.

11. The black-box neural network architecture search method of claim 10, wherein The reward function R(A) is: where S denotes the FID score on the proxy task, F' and F are the FLOPs of the sampled model and the target model, respectively, and λ is a penalty coefficient that controls the trade-off between accuracy and computational complexity.

12. The black-box neural network architecture search method of any one of claims 1 to 11, wherein, The construction of the prompt includes the following processes: Formulate system instructions: Establish the primary responsibilities of the LLM and provide a concise instruction overview, guiding the LLM to examine and analyze the provided empirical results to identify patterns and insights; Analyze empirical results: Based on the system instructions, the LLM analyzes historical experimental data to search for potential model structures that align with the analysis results and meet the preset search criteria; Record experiment logs: Create a concise record of historical experimental results, excluding detailed information directly related to the target model or task, ensuring that the LLM does not be unduly influenced by prior knowledge when searching for model architectures; Implement format specifications: Develop clear output format specifications to ensure the LLM-generated output is consistent and clear, meeting specific requirements and standards; Minimize ambiguity: Reduce ambiguity in the output by developing and implementing format specifications, improving the readability and usability of the LLM's output results; Feedback and iteration: Based on the LLM's output, provide feedback and iterative optimization to further improve the accuracy and efficiency of model searching.

13. The black-box neural network architecture search method of any one of claims 1 to 12, wherein, Search or optimization of neural network architectures for image tasks.

14. A computer readable storage medium storing a computer program, wherein the computer program comprises program instructions configured to cause a processor to perform the method according to any one of claims 1 to 13. The computer program, when executed by a processor, implements the black-box neural network architecture search method according to any one of claims 1-13.

15. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the black-box neural network architecture search method according to any one of claims 1-13.

Images