Method, apparatus and electronic device for searching neural network ensemble model

Abstract

The application provides a method for searching a neural network architecture ensemble model, comprising: obtaining a data set, the data set comprising samples and labels in a classification task; searching using a neural network architecture distribution search algorithm, comprising: determining hyperparameters of a neural network architecture distribution; sampling a valid neural network architecture in the architecture distribution defined by the hyperparameters; training and evaluating the neural network architecture on the data set to obtain a performance indicator; determining a neural network architecture distribution sharing hyperparameters according to the performance indicator to obtain a candidate pool of base learners; the base learner is a neural network architecture meeting the requirements of the architecture distribution; determining a proxy model; the proxy model is used to predict the test performance of a neural network architecture that has not been evaluated; predicting the test performance of the base learners in the candidate pool through the proxy model to determine k diverse base learners meeting the requirements of a task scenario to form an ensemble model, and the size of the ensemble model is k.

Description

Technical Field

[0001] This application relates to the field of machine learning, and more particularly to a method, apparatus, and electronic device for searching ensemble models of neural networks. Background Technology

[0002] Deep neural network-based models have made significant progress in various tasks such as image recognition, speech recognition, and machine translation. However, the prediction probabilities (softmax probabilities) of individual deep models generally have large calibration errors and low confidence levels. Especially when faced with inconsistent distributions between test and training data (OOD data), individual deep neural networks have weak rejection capabilities, cannot accurately reflect uncertainty, and are prone to overconfidence in erroneous predictions. These problems significantly limit the reliability and robustness of deep neural networks in high-risk, high-uncertainty, or real-world applications involving OOD data.

[0003] In multi-class classification problems, OOD (out-of-distribution) data refers to test samples whose distribution differs from the training data. This difference could be due to different data generation environments, or the samples being damaged or disturbed. Taking autonomous driving as an example, if the training data is collected on sunny days while the test data is collected on rainy or snowy days, or if the training data is collected in the suburbs while the test data is collected in the city, then the test data is OOD compared to the training data. For this type of data, machine learning models must have the ability to reject it.

[0004] A base learner is a single model in an ensemble of models.

[0005] An ensemble model combines the predictions of multiple base learners to form a better model / prediction. Ensemble models not only achieve higher test accuracy but also possess better calibrated prediction probabilities, exhibiting more accurate quantification of uncertainty and greater robustness, especially for out-of-demand (OOD) data. These advantages are also present in deep neural network models.

[0006] For example, deep ensembles combine multiple neural networks with the same network architecture but different weights and initialization values, and then average the final predicted outputs (output logits), which can effectively improve test accuracy and model calibration accuracy.

[0007] Model calibration refers to ensuring that the model's predicted probability of an event's outcome aligns with the actual empirical probability of that event. For example, in a binary classification task, if we take 100 images for which the model predicts a probability of 0.7, and 70 of them actually have a true label of 1, then the model's predicted probability is consistent with the actual empirical probability. In other words, the model's prediction is accurate and reliable. In practical applications, especially high-risk applications, the predicted probabilities of machine learning models are often used to inform user judgments or decision-making, so the confidence level of their predictions is crucial.

[0008] The performance of ensemble models often depends on the diversity of their base learners: the greater the differences between the base models, the better the ensemble effect tends to be. Therefore, many ensemble methods attempt to promote diversity among base learners. For example... Figure 1 As shown, deep ensemble models increase diversity by changing the initial weights of the base learners; hyperparameter ensemble models further enhance diversity by altering the training hyperparameters in addition to changing the initial weights. However, since the base learners of these deep ensemble models all share the same neural network architecture, a very natural extension to increase diversity is to use neural networks with different architectures to form an ensemble model.

[0009] An ensemble model is an ensemble model composed of base learners from multiple deep neural networks with different network architectures.

[0010] However, searching for ensemble models composed of different network architectures is far more complex than searching for a single neural architecture (NAS), because searching for ensemble models requires not only searching for network architectures with good-performing base learners, but also exploring the performance of possible combinations of different base learners. Therefore, existing methods need to comprehensively evaluate a large number of neural network architectures to generate a candidate pool of base learners, and then use greedy algorithms to evaluate the test performance of ensemble models generated from different combinations. This results in ensemble model searching being computationally expensive and time-consuming. Summary of the Invention

[0011] To address the aforementioned problems, embodiments of this application provide a method, apparatus, and terminal device for searching neural network ensemble models.

[0012] In a first aspect, embodiments of this application provide a method for searching ensemble models of neural network architectures. The method includes: acquiring a dataset comprising samples and labels from a classification task; performing a search using a neural network architecture distribution search algorithm, including: determining hyperparameters of the neural network architecture distribution; sampling a neural network architecture from the architecture distribution defined by the hyperparameters; training and evaluating the neural network architecture based on the samples and labels from the classification task to obtain performance metrics; determining a neural network architecture distribution sharing the hyperparameters based on the performance metrics to obtain a candidate pool of base learners; the base learners being neural network architectures conforming to the architecture distribution requirements; the neural network architecture being composed of repeatedly stacked neural network architecture units; determining a surrogate model; the surrogate model being used to predict the test performance of unevaluated neural network architectures; and predicting the test performance of base learners in the candidate pool using the surrogate model to determine those conforming to the classification task requirements. An ensemble model is composed of several base learners, and the size of the ensemble model is [missing value]. This significantly reduces the number of evaluations required for individual neural network architectures and ensemble models, thereby significantly reducing the difficulty and cost of structural ensemble models without compromising search quality. Furthermore, the targeted ensemble models are better at rejecting out-of-demand (OOD) data compared to individual deep neural network models, making them more robust to distributional shifts.

[0013] As a feasible implementation, the search using the neural architecture distribution search algorithm includes: performing a neural architecture distribution search using the approximate neural architecture search via operation distribution (ANASOD) algorithm. This allows for traversal of a larger portion of the search space, significantly improving search efficiency.

[0014] As a feasible implementation, the hyperparameter for determining the distribution of the neural network architecture includes: the hyperparameter for determining the distribution of the neural network architecture is an ANASOD encoding; the ANASOD encoding is a vector indicating the probability distribution of various operators in the neural network architecture units, and the mapping between the ANASOD encoding and the neural network architecture units is one-to-many. In this way, the NAS problem can be approximated by the operator probability distribution, greatly compressing the search space.

[0015] As a feasible implementation, determining the hyperparameters of the neural network architecture distribution includes: optimizing the hyperparameters of the neural network architecture distribution using a search strategy, wherein the search strategy is Bayesian optimization, and the search strategy is used to sample neural network units in the next iteration that have performance metrics that better meet the requirements than the current neural network architecture units. In this way, each selection and evaluation focuses on the architecture distribution covered by the hyperparameter definition, thus allowing for the traversal of a larger portion of the search space and significantly improving search efficiency.

[0016] As a feasible implementation, sampling a neural network architecture from the architecture distribution defined by the hyperparameters includes: determining the specific number of each operator in the constituent units of the neural network architecture according to the operator probability distribution defined by the ANASOD encoding; and connecting different operators according to a set search space to obtain the neural network architecture. This yields an effective architecture that conforms to the hyperparameter definition, serving as a basis for all architectures sharing this ANASOD encoding. The performance proxy of the distributed neural network architecture.

[0017] As a feasible implementation, training and evaluating the neural network architecture units on the dataset to obtain performance metrics includes: training the neural network architecture on the training dataset; evaluating the neural network architecture on the validation dataset to obtain performance metrics; wherein the training set data and the validation set data both belong to the dataset. In this way, only one neural network architecture can be sampled and evaluated from each architecture distribution, and its performance metric y can be used as the basis for all architectures sharing this ANASOD encoding. The performance metrics of neural network architecture units can effectively avoid the high costs associated with repeatedly evaluating the performance of similar architecture units.

[0018] As a feasible implementation, the step of using the distributional neural architecture search (NAS) algorithm for searching further includes: determining a search strategy for the distribution of neural network architectures based on the performance metrics and the hyperparameters. This adjusts the search strategy to determine the search strategy for the distribution of neural network architectures in the next iteration, allowing the sampling of neural network units in the next iteration that have performance metrics that better meet the requirements than the current neural network architecture units.

[0019] As a feasible implementation, the search using the distributional neural architecture search (distributionalNAS) algorithm further includes: determining the performance prediction values ​​of the hyperparameters of other unknown distributions, including the mean and variance, based on the hyperparameters and performance metrics of the neural architecture distributions found in each search; and determining a performance prediction strategy for the neural architecture distribution based on the mean and variance, wherein the performance prediction strategy is used to predict the performance metrics of the neural architecture distribution. Thus, the performance prediction strategy can be updated based on the performance prediction values. (y_t, y_t), thus determining the next search strategy.

[0020] As a feasible implementation, determining the neural network architecture distribution sharing the hyperparameters based on the performance metrics and obtaining a candidate pool of basic learners includes: determining a search strategy for the neural network architecture distribution based on the performance metrics and the hyperparameters; determining a performance prediction strategy for the neural network architecture distribution based on the performance metrics and the neural network architecture units; and searching among the neural network architecture distributions sharing the hyperparameters based on the search strategy and the performance prediction strategy to determine a candidate pool of basic learners. In this way, an optimal neural network architecture distribution can be obtained, and a high-quality architecture distribution can generate high-quality neural network architectures with similar performance, providing a good candidate pool of basic learners.

[0021] As a feasible implementation, determining the distribution of neural network architectures sharing the hyperparameters based on the performance metrics to obtain a candidate pool for the base learner includes: outputting multiple neural network architectures sharing the hyperparameters based on multiple neural network architectures and corresponding performance metrics from historical searches; determining a suitable neural network architecture distribution based on the multiple neural network architectures sharing the hyperparameters; and generating multiple neural network architecture units based on the suitable neural network architecture distribution to obtain a generation distribution / candidate pool for the base learner. Thus, using a neural network architecture distribution search method to learn the candidate pool / architecture distribution is more efficient and significantly reduces the number of evaluations and the cost of evaluating individual network architectures.

[0022] As a feasible implementation, determining the surrogate model includes: obtaining the surrogate model by training it on the dataset based on the neural network architecture units and the performance metrics. In this way, the performance of other unevaluated architectures can be directly predicted using the surrogate model, thus avoiding huge evaluation costs.

[0023] As a feasible implementation, the step of predicting the test performance of the base learners in the candidate pool using a proxy model to determine those that meet the requirements of the task scenario is... An ensemble model is composed of q base learners, comprising: predicting the test performance of multiple base learners in the candidate pool using a surrogate model; performing a local search based on the prediction results to determine q estimated vertex architectures, wherein the estimated vertex architectures are neural network architectures whose performance metrics predicted by the surrogate model on the validation set are higher than those of their adjacent architectures; and combining k architectures from the q estimated vertex architectures that meet the performance metric requirements to obtain the ensemble model. In this way, the optimal combination can be selected from the candidate pool, reducing the difficulty of the extremely complex permutation and combination problem, requiring only the evaluation of a very small number of combinations of base learners to search for a high-quality ensemble model.

[0024] As a feasible implementation, the k architectures that meet the performance requirements from the q estimated vertex architectures are combined, including: sorting the performance indicators of the q estimated vertex architectures from best to worst, and selecting the k architectures with the highest performance indicators for combination. In this way, the optimal combination can be selected from the candidate pool, reducing the difficulty of the extremely complex permutation and combination problem. Only a very small number of combinations of basic learners need to be evaluated to search for a high-quality ensemble model.

[0025] As a feasible implementation, the k architectures that meet the performance requirements from the q estimated vertex architectures are combined. This involves using a greedy selection algorithm to iterate through the q estimated vertex architectures and select k architectures to add to the ensemble model. In this way, the optimal combination can be selected from the candidate pool, significantly reducing the probability and complexity of permutations and combinations (selecting k from all base learners in the candidate pool is reduced to selecting k from q base learners). A high-quality ensemble model can be searched by evaluating only a very small number of combinations of base learners.

[0026] Secondly, embodiments of this application provide an apparatus for searching neural network architecture ensemble models. The apparatus includes: a data acquisition module for acquiring a dataset, the dataset including samples and labels from a classification task; an architecture distribution search module for searching using a neural network architecture distribution search algorithm, including: determining hyperparameters of the neural network architecture distribution; sampling a neural network architecture from the architecture distribution defined by the hyperparameters; training and evaluating the neural network architecture based on the samples and labels from the classification task to obtain performance metrics; determining neural network architecture distributions sharing the hyperparameters based on the performance metrics to obtain a candidate pool of base learners; the base learners being neural network architectures conforming to the architecture distribution requirements; the neural network architecture being composed of repeatedly stacked neural network architecture units; determining a surrogate model; the surrogate model being used to predict the test performance of unevaluated neural network architectures; and an architecture ensemble model combination module for predicting the test performance of base learners in the candidate pool using the surrogate model to determine those conforming to the classification task requirements. An ensemble model is composed of several base learners, and the size of the ensemble model is [missing value]. .

[0027] Thirdly, embodiments of this application provide an electronic device including a processor and a memory; the processor is configured to execute computer execution instructions stored in the memory, and the processor executes the computer execution instructions to perform the method for searching neural network structures based on evolutionary learning as described in any embodiment of the first aspect.

[0028] Fourthly, embodiments of this application provide a storage medium, including a readable storage medium and a computer program stored in the readable storage medium, the computer program being used to implement the method for searching neural network structures based on evolutionary learning as described in any embodiment of the first aspect. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only a few embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] The accompanying drawings used in the description of the embodiments or prior art are briefly introduced below.

[0031] Figure 1 The flowchart shows the integrated model search method for the first scheme.

[0032] Figure 2 System architecture diagram provided for embodiments of this application;

[0033] Figure 3 A schematic diagram illustrating the application of the search neural network ensemble model method provided in this application to an image classification scenario;

[0034] Figure 4 A schematic diagram illustrating the application of the search neural network ensemble model method provided in this application to a target detection and recognition scenario;

[0035] Figure 5 A flowchart of the search neural network ensemble model provided in the embodiments of this application;

[0036] Figure 6 A comparison graph of test errors obtained on CIFAR10 for various benchmarks, including the DistriNAS-PM provided in this application;

[0037] Figure 7 A schematic diagram of 15 interference / noise patterns randomly selected and added to the CIFAR10 and CIFAR100 validation set images;

[0038] Figure 8 The effect of randomly selecting interference / noise to add to the images in the CIFAR10 and CIFAR100 validation sets;

[0039] Figure 9 This diagram illustrates the OOD verification comparison between DistriNAS-PM provided in this application and other search methods on the NAS-Bench-201 space.

[0040] Figure 10 This is a schematic diagram of an electronic device. Detailed Implementation

[0041] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0042] In the following description, the terms “first, second, third, etc.” or module A, module B, module C, etc. are used only to distinguish similar objects and do not represent a specific ordering of objects. It is understood that a specific order or sequence may be interchanged where permitted so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0043] In the following description, the labels of the steps, such as S110, S120, etc., do not necessarily mean that the steps will be executed in this way. The order of the steps can be interchanged or executed simultaneously if permitted.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0045] The technical solutions related to the embodiments of this application will now be described with reference to the accompanying drawings.

[0046] The first option is as follows: Figure 1 As shown, random search (NES-RS) or evolutionary algorithm (NES-RE) is used to search for architectures suitable as base learners, thereby building a sufficiently large candidate pool of base learners. Then, a greedy selection algorithm (GSA) is used to traverse the base learners in the candidate pool and select members to form the final set one by one.

[0047] Among them, the random search algorithm is based on the objective function and the size of the candidate pool of the basic learners. As input, random sampling is performed in the NAS search space. Each architecture unit is trained and its performance is evaluated to obtain its metrics, and the output is a candidate pool of basic learners.

[0048] Evolutionary algorithms are based on an objective function and a candidate pool size n for the basic learners. pool As input, randomly sample n from the NAS search space. init Each architecture unit is assigned a set of architectural units, and its performance metrics are obtained through complete training and performance evaluation. The n units with the best performance metrics are then selected. parent Each set of architecture units serves as a parent unit. Before reaching the termination criterion, the process iteratively executes the following: randomly sample B architecture units from the parent units, and randomly mutate them to obtain B child architecture units; fully train and evaluate the performance of these B child architecture units; iterate through these B child architecture units, selecting those that can form an ensemble model with the maximum optimization objective function, adding them to the parent architecture unit pool, and removing the oldest parent architecture unit to ensure that the size of the parent architecture unit pool remains unchanged at n. parent The output is the candidate pool for the basic learners.

[0049] The greedy algorithm takes a pool of candidate base learners and an ensemble architecture size *k* as input. It initializes the ensemble architecture by setting it to the neural network architecture with the lowest test error in the candidate pool, and then removes this architecture from the pool. Before the ensemble size becomes smaller than *k*, it iterates through the remaining base learners in the candidate pool, adding each one to the existing ensemble and evaluating the performance of the new ensemble. Finally, it selects the architecture that results in the greatest performance improvement, adds it to the existing ensemble, and removes this architecture from the candidate pool. The output is the final ensemble model.

[0050] This method adopts the traditional NAS (Neural Network Architecture) approach, treating each neural network architecture in the architecture search space as a single entity. Therefore, each neural network architecture / basic learner selected into the candidate pool needs to undergo complete training and evaluation. Such methods typically require a candidate pool containing hundreds of neural network architectures, resulting in extremely high evaluation costs for the candidate pool construction itself.

[0051] When searching for the optimal combination of base learners / architecture integration from the candidate pool, the greedy algorithm used can greatly simplify the originally highly complex permutation and combination problem, but it still needs to evaluate a large number of possible architecture combinations / architecture integrations, which increases the evaluation cost in the integration selection process.

[0052] Therefore, such search methods require significant graphics processing unit (GPU) time and computational resources to find high-quality ensemble models. The enormous demands on GPU time and hardware resources often limit their application in real-world scenarios.

[0053] GPU time is a common unit for measuring the computational load of an algorithm. It is the time a single GPU needs to run to complete a task, and is specifically expressed as GPU-days, GPU-seconds, etc.

[0054] Existing methods for automatically searching ensemble models suffer from enormous search costs and low efficiency. The fundamental reason is that they require first constructing a sufficiently large pool of candidate base learners, and then trying various combinations of these base learners to find the optimal ensemble model. Constructing a sufficiently large pool of candidate base learners necessitates training and evaluating a large number of neural network architectures. Because traditional NAS methods use a single neural network architecture as the search target, many very similar neural network architectures also need to be trained and evaluated individually. Finding the optimal ensemble model from various combinations of base learners in the candidate pool requires evaluating numerous possible architecture combinations. Although greedy algorithms can greatly simplify the NP-hard permutation and combination problem, the number of combinations that need to be evaluated is still proportional to the number of base learners in the candidate pool and the size of the ensemble model.

[0055] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings.

[0056] Within the existing neural network architecture search space, ensemble models exhibit the following characteristics: The performance of an ensemble model depends on the average performance of the base learners within the ensemble; provided the average performance is excellent, the greater the difference between the base learners (i.e., the greater the diversity of the ensemble), the better the performance of the ensemble model; the predicted outputs of vertex architectures in the neural network architecture search space vary relatively greatly, resulting in high diversity in ensemble models composed of vertex architectures. Here, a vertex architecture refers to an architecture whose test accuracy is higher than that of its directly adjacent architectures.

[0057] Based on the characteristics of the above-mentioned ensemble model, this application provides a method for searching neural network ensemble models that uses distributional neural network architecture search (NAS) to determine the architecture distribution or candidate pool for generating base learners. According to the architecture distribution or candidate pool for generating base learners, a surrogate model is used to predict the optimal neural network vertex architecture in the candidate pool. These vertex architectures are then combined to obtain the ensemble model.

[0058] It's important to understand that neural network architecture distribution search integrates neural networks with similar architectures or belonging to the same distribution for evaluation, thus avoiding repeated evaluation of individual neural network architectures and significantly improving search efficiency. Furthermore, the base learners generated from the final found architecture distribution all exhibit similar and excellent average performance, thereby satisfying the requirement that the performance of an ensemble model depends on the average performance of the base learners within the ensemble model.

[0059] The method for searching neural network ensemble models provided in this application adopts a more efficient candidate pool generation scheme for basic learners and a basic learner combination scheme, which can reduce the search cost of ensemble models. While ensuring performance, it can greatly improve the search cost of ensemble models, making the cost of searching ensemble models in a larger search space controllable and acceptable, and applicable to more practical production scenarios.

[0060] Proxy models are typically simple models used to simulate overly complex or black-box real-world problems. They can also be used to quickly predict the output of black-box or overly complex problems. Proxy models can quickly and relatively accurately predict the test performance of neural network architectures, thus avoiding extensive training and evaluation to obtain their true performance. The predicted outputs of vertex architectures in the neural network architecture search space vary relatively greatly, while ensemble models composed of vertex architectures have high diversity. Combining these vertex architectures ensures the diversity of ensemble models and avoids the evaluation cost of repeatedly trying different architecture combinations to generate ensemble models.

[0061] Since the performance of an ensemble model depends primarily on the average performance and diversity of its base learners, the method for searching neural network ensemble models provided in this application reduces the number of evaluations while specifically ensuring the aforementioned characteristics of the ensemble model, thereby enabling users to efficiently search for high-performance ensemble models.

[0062] Figure 2 This is a system architecture diagram provided for embodiments of this application. The method for searching neural network ensemble models provided in this application can be widely applied to various system architectures and scenarios that require the use of convolutional neural networks. For example... Figure 3 As shown, the data collection device 10 obtains the required data or samples through various means and provides the data to the computing device 11. The method for searching neural network ensemble models provided in this application embodiment will run on the computing device 11 to search for neural network architectures and train the finally searched architectures, and deploy the trained ensemble model to devices in various application scenarios, such as personal computers 12, servers 13, and mobile devices 14.

[0063] Figure 3 This diagram illustrates the application of the search neural network ensemble model method provided in this embodiment of the application to an image classification scenario. Figure 3As shown, users often store a large number of pictures in their photo albums on smartphones or other multimedia storage devices. Categorizing these pictures based on information within them can help users manage and find them. The method for searching neural network ensemble models provided in this application can quickly pre-search and train the most suitable convolutional neural network ensemble model for similar image classification tasks and deploy it on a smartphone, replacing the commonly used network models manually designed by human experts. This achieves higher classification accuracy and improves the user experience.

[0064] Figure 4 This diagram illustrates the application of the search neural network ensemble model method provided in this embodiment of the application to a target detection and recognition scenario. Figure 4 As shown, the ensemble model searched using the method for searching neural network ensemble models provided in this application can identify objects in the original image and output a labeled image after object detection. Object detection and recognition in images or videos are widely used in tasks such as smart cities and autonomous driving. Figure 3 Similar to other application scenarios, the method for searching neural network ensemble models provided in this application embodiment can be used to find the most suitable neural network backbone for object detection and recognition tasks in various scenarios, such as hardware limitations of mobile devices, and deploy it on relevant devices to improve recognition accuracy.

[0065] The method for searching neural network ensemble models provided in this application embodiment is also applicable to image classification scenarios in medical imaging. Machine learning systems are now used in medical imaging to help medical personnel make diagnoses using imaging data. After training with existing data, the convolutional neural network ensemble model searched using the method provided in this application embodiment can not only perform efficient and high-precision classification and diagnosis based on image features, but also output confidence levels for its diagnoses due to the ensemble model's good calibration for prediction uncertainty, helping doctors screen cases requiring manual confirmation.

[0066] The method for searching neural network ensemble models provided in this application can be deployed on computing nodes of relevant devices. Its data and code can be stored on various common storage devices in various computers. For the execution of instructions and functional modules, steps other than performance evaluation can generally be performed by the central processing unit (CPU). Performance evaluation involves training the neural network architecture, which is generally performed by the graphics processing unit (GPU). The ensemble model obtained by the method for searching neural network ensemble models provided in this application can be deployed on various computers and mobile computing devices after training, and applied to various tasks requiring convolutional neural network prediction, such as image classification, object detection and recognition, and medical image classification.

[0067] This application provides a method for searching neural network ensemble models, comprising: acquiring a dataset including samples and labels from a classification task; performing a search using a distributional neural network architecture search (NAS) algorithm, including: determining hyperparameters of the neural network architecture distribution; sampling a neural network architecture from the architecture distribution defined by the hyperparameters; training and evaluating the neural network architecture based on the samples and labels from the classification task to obtain performance metrics; determining a neural network architecture distribution sharing hyperparameters based on the performance metrics to obtain a candidate pool of base learners; the base learners are neural network architectures that meet the architecture distribution requirements; the neural network architecture is composed of repeatedly stacked neural network architecture units; determining a surrogate model; the surrogate model is used to predict the test performance of unevaluated neural network architectures; and predicting the test performance of the base learners in the candidate pool using the surrogate model to determine models that meet the requirements of the classification task scenario. The ensemble model consists of 1 base learners, and the size of the ensemble model is 1. .

[0068] This application's embodiments involve two aspects of evaluation: 1) Evaluation of a single neural network architecture, i.e., the base learner, includes training the neural network architecture from scratch on the training dataset and, after training, evaluating its performance, such as test accuracy, on the validation dataset. 2) Evaluation of a single ensemble model involves evaluating the performance of the entire ensemble model on the validation dataset. The number of evaluations is a common metric for measuring the computational cost of an algorithm.

[0069] Figure 5 A flowchart illustrating a method for searching neural network ensemble models provided in this application embodiment. The following will be combined with... Figure 5This application provides a detailed explanation of each step in a method for searching an ensemble neural network model, as illustrated in the embodiments of this application. Figure 5 As shown, this application embodiment provides a method for searching neural network ensemble models, which involves performing the following steps 1-5 to perform a neural network architecture distribution search to obtain a candidate pool of basic learners.

[0070] Step 1, obtain the dataset.

[0071] In one feasible implementation, existing data and data corresponding to the target task are acquired.

[0072] For example, a certain number of images and their correct annotations for an image classification task can be obtained from existing datasets or other manually labeled data as samples (ground truth). This data can be used as training datasets and / or test set data.

[0073] Step 2: Determine the architecture search space.

[0074] In one feasible implementation, the morphology of candidate neural network architectures and the target of neural network architecture search are determined by defining the search space and the objective function.

[0075] One common search space is defined based on neural architecture cells, which includes defining the number of operators in the search neural architecture cell, the types of operators available, the maximum number of connections between operators, and the number of times the neural architecture cells are stacked in the final neural architecture.

[0076] For example, a NAS search space can be defined as: a neural network architecture unit. The number of operators is 10, and there are 3 types of operators to choose from: A, B, and C; the maximum number of connections between each operator is i, and the number of times the neural network architecture units are stacked is j.

[0077] The objective function is defined according to the target task. A common objective function is to maximize task accuracy. For example, the accuracy on the validation set in an image classification problem can be defined as the objective function. The objective function can also include other constraints and objectives. For example, the objective function can be defined as minimizing the floating-point operations per second (FLOPs) of the neural network while maximizing image classification accuracy, to be applied to mobile devices with limited computing power.

[0078] The method for searching neural network ensemble models provided in this application uses the distributional neural network architecture search (NAS) algorithm to determine the architecture search space as the search space of distributional neural network architecture search (NAS). This distributional neural network architecture search space defines hyperparameters of the architecture distribution while defining the NAS search space, to determine the probability distribution of each operator appearing in the neural network architecture unit.

[0079] In one feasible implementation, the distributional neural architecture search (NAS) algorithm can employ the approximate neural architecture search via operation distribution (ANASOD) algorithm to perform the neural architecture distribution search. This involves defining the NAS search space and simultaneously defining the corresponding ANASOD encoding. For the hyperparameters of the architecture distribution.

[0080] In one feasible implementation, the search space of ANASOD can be defined as: for each neural network architecture unit Corresponding encoding It is located in A vector in a simplex space, where each value represents a different operator within a neural network architecture unit. The probability of it appearing in the middle.

[0081] For example, a typical neural network architecture unit has 10 operators, and there are three types of operators to choose from: A, B, and C. A appears 5 times in the neural network architecture unit, B appears 2 times, and C appears 3 times. Then the encoding corresponding to the neural network architecture unit... The values ​​are: [0.5, 0.3, 0.2].

[0082] coding The mapping between neural network architecture units is one-to-many, meaning multiple similar neural network architecture units share the same encoding. This greatly compresses the search space. For example, the following neural network architecture units share the same encoding. .

[0083] Neural Network Architecture Unit 1: AAAAABBBCC

[0084] Neural Network Architecture Unit 2: AAAABABBCC

[0085] Neural network architecture unit 3: AAABBAABCC

[0086] The above three neural network architecture units only share encoding. This is a subset of multiple similar neural network architecture units in the range [0.5, 0.3, 0.2]. Other similar neural network architecture units obtained through permutations and combinations are not listed here.

[0087] Meanwhile, in the NAS search space, there are other sites with the same ANASOD code. The test accuracy of different neural network architecture units on the test set is often very similar, indicating that the approximation of the NAS problem by the operator distribution probability alone is relatively accurate.

[0088] Understandably, analysis of the existing NAS search space reveals that searching for an exact solution where the operator type, number, and topology are all fully determined is unnecessary. Conversely, a set of neural network architectures with the same operator type and number, the same operator probability distribution, but different topologies exhibit very similar performance. Therefore, ANASOD encoding... Defined as a vector of probability distributions of various operators in a neural network architecture unit, the sum of the probability distributions of various operators in a neural network architecture unit is 1.

[0089] Unlike the vast permutation and combination search space in NAS, ANASOD encoding... It resides in a smaller vector space that is easier to optimize. Based on this, it is possible to encode... The ANASOD algorithm, which uses a series of approximate NAS algorithms in the corresponding search space, significantly reduces the search difficulty and improves search efficiency while maintaining search accuracy. This is due to a low-dimensional encoding. It can correspond to a set of multiple approximate neural network architecture units. The ANASOD algorithm can directly search for multiple approximate neural network architecture units and apply them to the ensemble model.

[0090] The neural architecture ensemble model search method provided in this application embodiment can also employ other neural architecture distribution search methods.

[0091] Step 3: Use a search strategy to recommend new hyperparameters for the neural network architecture distribution, in order to determine the hyperparameters for the neural network architecture distribution.

[0092] In one feasible implementation, the search strategy can be Bayesian optimization, using Bayesian optimization to recommend hyperparameters for new architecture distributions. For example, Bayesian optimization can be used to recommend the encoding of new ANASODs. Furthermore, evolutionary algorithms can be used to recommend hyperparameters for new neural network architecture distributions.

[0093] For example, a search strategy can be represented as: Search strategy ( ,y); where The encoding / hyperparameters of ANASOD recommended for the Bayesian optimization model, where y is the performance metric for the predicted neural network architecture distribution. Its meaning can be interpreted as: searching for the encoding of ANASOD. The distribution of neural network architectures under coverage, with predicted performance metric y.

[0094] Performance prediction models used to predict performance metrics for neural network architecture distributions can include Gaussian processes, Bayesian neural networks, and random forests.

[0095] The hyperparameter-based ensemble model search method provided in this embodiment is completely different from the first approach, which uses a search strategy to recommend new neural network architecture distributions. The first approach uses NAS to search for candidate architectures suitable as the base learners of the ensemble model, and each search strategy directly recommends a new neural network architecture unit. However, in step 3 provided in this embodiment, each selection and evaluation is of the architecture distribution covered by the hyperparameter definition, thus allowing for a larger traversal of the search space and greatly improving search efficiency.

[0096] Step 4: Randomly sample a neural network architecture from the architecture distribution defined by the hyperparameters, perform performance evaluation, and obtain performance metrics.

[0097] In one feasible implementation, an ANASOD-optimized / search architecture distribution is adopted ( , , When, from Searching for and training a neural network architecture within a defined architecture distribution. To evaluate its performance and its performance metrics As having the same ANASOD code The neural network architectures share common performance metrics.

[0098] In one feasible implementation, step 4 includes:

[0099] Step 41: Randomly sample a neural network architecture from the distribution defined by the hyperparameters.

[0100] In one feasible implementation, within the ANASOD encoding space, according to the ANASOD encoding... The probability distribution of operators in the defined neural network architecture unit determines the specific number of each operator in the unit. Based on the search space constraints, different operators are randomly connected to obtain the neural network architecture unit. After determining the neural network architecture unit, it is stacked several times according to the definition of the search space to obtain the neural network architecture. In determining the neural network architecture The probability distribution of the middle operator conforms to ANASOD coding. Then, a neural network architecture was used. As all share this ANASOD code The performance proxy of the distributed neural network architecture.

[0101] Step 42, neural network architecture The dataset is used for training and evaluation to obtain performance metrics.

[0102] In one feasible implementation, the neural network architecture can be trained on the training dataset obtained in step 1, using conventional neural network optimization methods based on the samples and labels in the classification task. And evaluate and obtain its performance index on the validation set according to the objective function defined in step 2. This process can be represented as performance evaluation ( , The training and validation sets belong to the same distribution and are based on the same dataset.

[0103] In one feasible implementation, performance evaluation can be used as a basis ( , The performance of the distributed neural network architecture is evaluated, and its performance metric y is used as the parameter for all architectures sharing this hyperparameter. Performance metrics for the architecture distribution.

[0104] Since different neural architectures generated from the same architecture distribution ultimately exhibit extremely similar performance, when optimizing / searching architecture distributions, only one neural network architecture is sampled and evaluated from each architecture distribution, and its performance metric y is used as the encoding for all architectures sharing this ANASOD. The performance metrics of neural network architecture units can effectively avoid the high costs associated with repeatedly evaluating the performance of similar architecture units.

[0105] Step 5, based on performance indicators Update the search strategy; while updating the search strategy, output the current optimal neural network architecture distribution based on the search history; update the neural network architecture performance proxy model. The current optimal neural network architecture distribution is the neural network architecture distribution that meets the requirements.

[0106] In one feasible implementation, a search strategy for the distribution of neural network architectures can be determined based on performance metrics and hyperparameters, enabling it to search for neural network architecture units with higher performance metrics in the next iteration. This includes the following steps:

[0107] Step 51, update the search strategy based on hyperparameters and performance metrics. Let _t and y_t be the number of iterations, and determine the search strategy.

[0108] For example, based on the hyperparameters of the distribution of the neural network architecture found in each search... The search strategy is adjusted using _t and the performance metric y_t to determine the search strategy for the neural network architecture distribution search in the next iteration.

[0109] Step 52: Determine the performance prediction strategy for the neural network architecture distribution based on the performance metrics and the hyperparameters of the neural network architecture distribution.

[0110] In one feasible implementation, the hyperparameters of the neural network architecture distribution obtained each time can be used as a basis. The performance prediction strategy is updated using _t and the performance metric y_t. (_t, y_t), thereby determining the performance prediction strategy for the neural network architecture distribution.

[0111] For example, the hyperparameters of the neural network architectures searched each time are distributed. _t and its performance metrics ( The model takes (y_t, y_t) as input and outputs performance prediction values ​​for hyperparameters with other unknown distributions, including the mean (m) and variance (v).

[0112] The performance prediction value (m, v) for the tth iteration = performance prediction model ( | { _i, y_i}^t_{i=1, 2,...t}).

[0113] in, These are unknown hyperparameter values, and the {} part refers to the history of hyperparameters that have been searched and evaluated.

[0114] Step 53: Update the performance prediction strategy based on the performance prediction values ​​to determine the next evaluation target. _(t+1), y_t+1).

[0115] Step 54: Output the current optimal neural network architecture distribution based on the search history.

[0116] In one feasible implementation, a suitable neural network architecture distribution is determined based on multiple neural network architecture distributions and corresponding performance metrics from historical searches.

[0117] For example, the search history has three architecture distributions of hyperparameters and their performance: ( _1, y_1), ( _2,y_2)( _3, y_3), where ( If the performance index y_2 of (y_2, y_2) is optimal, then the shared hyperparameters are... The architecture distribution of _2 is a neural network architecture distribution that meets the requirements.

[0118] Step 55: Based on the neural network architecture distribution that meets the requirements, generate multiple neural network architecture units to obtain the generation distribution / candidate pool of the basic learner.

[0119] In one feasible implementation, numerous specific neural network architectures can be randomly generated based on the optimal neural network architecture distribution output by S54 to obtain the generation distribution / candidate pool of the basic learners. The optimal neural network architecture distribution is the neural network architecture distribution that meets the requirements.

[0120] Step 56: Determine the surrogate model. The surrogate model is used to predict the test performance of the unevaluated neural network architecture to help predict and assist in the search for the optimal vertex architecture, thereby quickly generating a high-quality ensemble model.

[0121] In one feasible implementation, the surrogate model can be trained and updated based on the neural network architecture units and performance metrics evaluated during t search cycles.

[0122] In one feasible implementation, the surrogate model can be a Gaussian process with a WL graph kernel (GPWL), for example, y = GPWL model(a| {a_i, y_i}^t_{i=1,2,…t}), where a is an unknown neural network architecture, y is the architecture performance predicted by the surrogate model, and the {} part refers to the history of searched and evaluated neural network architectures.

[0123] Based on existing work on distributional neural network architecture search (NAS) and experimental verification, a high-quality architecture distribution often generates high-quality neural network architectures with similar performance. Therefore, for ensemble model search, a good architecture distribution naturally provides a good pool of candidate learners. Compared to existing techniques that use traditional NAS to iteratively evaluate individual neural network architectures and then add them to the candidate pool, the method proposed in this application, which uses distributional neural network architecture search to learn the candidate pool / architecture distribution, is more efficient and significantly reduces the number of evaluations and the cost of individual network architectures.

[0124] The effectiveness of an ensemble model depends not only on the performance of its base learners but also on the diversity of the base learners in its combination. Therefore, directly combining the best-performing base learners... Combining basic learners may not necessarily generate the optimal ensemble model. Instead, selecting a model of size [missing information] from the candidate pool... Finding the optimal combination is an extremely complex permutation and combination problem.

[0125] The first approach primarily uses a greedy algorithm to select the base learners in the ensemble one by one. However, this method requires the number of ensemble models to be evaluated and the size of the candidate pool to be proportional to the size of the final ensemble model.

[0126] Based on previous research on ensemble models and further experimental analysis, it can be found that, assuming the average performance of the base learners is guaranteed, the greater the prediction difference among the learners constituting the ensemble model, the better the performance of the ensemble model. The prediction differences among vertex architectures in the search space are often quite large. Therefore, combining the k best vertex architectures naturally ensures the diversity of the ensemble model, generating a high-performance architecture ensemble. Finding the true vertex architecture requires traversing and evaluating all neural network architectures in the search space.

[0127] The method for searching neural network ensemble models in this embodiment performs the following step 6: architecture sampling and architecture ensemble search. A high-quality ensemble model can be found by evaluating combinations of base learners only a very small number of times (even just once). Step 6 is described in detail below.

[0128] Step 6: Predict the test performance of the base learners in the candidate pool using the surrogate model to determine those that meet the requirements of the classification task scenario. The ensemble model consists of 1 base learners, and the size of the ensemble model is 1. .

[0129] Based on the generation distribution / candidate pool of the base learner, the most suitable candidate needs to be searched from it. If 1 base learner is used to form an ensemble model, then the size of the ensemble model is 1 / 2 * 1 / 3 ... .

[0130] In one feasible implementation, step 6 obtains the integrated model through the following steps S61-S63.

[0131] Step 61: Use the proxy model determined in Step 5 to directly predict the performance of other unevaluated architectures to avoid huge evaluation costs.

[0132] In one feasible implementation, random sampling can be performed first from the optimal architecture distribution, and then a local search can be performed starting from multiple sampled architectures based on the predicted performance metrics.

[0133] Step 62: Based on the predicted performance metrics output by the surrogate model, determine q estimated vertex architectures. The estimated vertex architectures are neural network architectures whose predicted performance metrics on the validation set are higher than those of their neighboring architectures.

[0134] In this context, adjacent architectures refer to network architectures whose operator arrangements differ by only one bit. For example, consider the following three neural network architectures: Architecture 1: AABC, Architecture 2: ABBC, and Architecture 3: AACC. Architecture 1 and Architecture 2 are adjacent architectures, and Architecture 1 and Architecture 3 are adjacent architectures.

[0135] Step 63: Combine the k best-performing vertices from the q estimated vertex architectures to obtain the ensemble model.

[0136] In one feasible implementation, the performance metrics of the q predicted vertex architectures can be ranked from best to worst, and the top k architectures with the best performance metrics can be combined to obtain an ensemble model. Alternatively, the k architectures among the q predicted vertex architectures whose performance metrics meet the requirements of the classification task can be combined to obtain an ensemble model. The classification task requirements could be highest accuracy, lowest error rate, or minimum loss function, etc.

[0137] In one feasible implementation, a greedy algorithm can be used to find k of the q predicted vertex architectures that can bring the greatest performance improvement to the existing ensemble model and combine them to obtain the final ensemble model.

[0138] Finally, execute step S7 to output the integrated model.

[0139] In one feasible implementation, a maximum number of searches or a maximum search time can be defined as the termination criterion. If the termination criterion is not reached, the iterations of steps 3 to 6 continue. Upon reaching the termination criterion, the algorithm iteration of this embodiment terminates. Downstream applications can directly apply the output ensemble model to the various scenarios mentioned above.

[0140] Example 1

[0141] The method for searching neural network ensemble models provided in this application, applied to image classification tasks, aims to search for high-performance neural network ensemble models on common search spaces such as DARTS and NAS-Bench-201 and datasets such as CIFAR10, CIFAR100, and ImageNet16-120, assuming that the training and validation sets belong to the same distribution. The specific steps of the method for searching neural network ensemble models provided in this application in the image classification task scenario of Embodiment 1 are described in detail below.

[0142] S701, Data Acquisition.

[0143] For example, data can be obtained from common image classification datasets. For instance, CIFAR-10 and CIFAR-100 have 50,000 and 10,000 images respectively in their training and validation sets, while ImageNet16-120 has over 154,700 and 3,000 training / validation images respectively. Training and validation data can be obtained from these datasets.

[0144] In one feasible implementation, a neural network architecture can be searched using images from the training set and their manually labeled images, and then validated on a validation set.

[0145] S702 defines the search space and objective function.

[0146] In one feasible implementation, the search space can be defined as the search space of NAS-Bench-201, which has 6 operator positions. and 5 operator type options The five operator types are: 3x3 convolution, 3x3 flat pooling, 1x1 convolution, residual connection, and zero connection. The search space contains a total of 15,625 neural network architecture units; after removing isomorphic architecture units, there are 6,466 unique neural network architecture units. Within this search space, the objective function of this application is to output the classification error rate of the ensemble model on the CIFAR-10 validation set.

[0147] Within the NAS-Bench-201 search space, the desired ensemble model is one with a low classification error rate on the validation set.

[0148] In one feasible implementation, the search space can be defined as a DARTS-based search space with 8 operator positions. and 7 type options The specific types of operators are: 3x3 or 5x5 separable convolution, 3x3 or 5x5 dilated convolution, 3x3 max pooling, 3x3 average pooling, and skip connection. Within this search space, the objective function of this application is also to output the classification error rate of the ensemble model on the CIFAR-10 validation set.

[0149] The following steps detail the search for a neural network ensemble model based on ANASOD and Gaussian processes:

[0150] Using the objective function and the ensemble model size k as input, the hyperparameters of the initial architecture distribution search (ANASOD) and the initial architecture distribution are initialized, also known as ANASOD encoding. The following steps are iterated before reaching the termination criterion:

[0151] S703 is a hyperparameter of the architecture distribution recommended by the search strategy, namely ANASOD encoding.

[0152] S704 randomly samples a specific neural network architecture from the operator distribution defined by the ANASOD encoding, performs performance evaluation, and uses the evaluation result as a performance proxy for all neural network architectures sharing this ANASOD encoding.

[0153] S705 updates the distribution search strategy and surrogate model using the evaluation results of the S704 neural architecture. The surrogate model is a Gaussian process with a WL graph kernel (GPWL) based on the Weisfeiler-Lehman graph kernel.

[0154] S706: Take q samples from the current optimal architecture distribution / ANASOD encoding recommended by the search strategy, evaluate the test performance of each sample using a surrogate model, and then, starting from each sample, use local search to find the predicted vertex architectures in its vicinity. Select the k architectures with the best prediction performance from the q predicted vertex architectures to form an ensemble model.

[0155] Output integrated model.

[0156] In the first embodiment provided in this application, the ANASOD neural network architecture distribution search method is used in the first stage of architecture integration search, as shown in steps S703-S705. In the second stage, a Gaussian process with WL graph kernel (GPWL) based on Weisfeiler-Lehman graph kernel is used as a surrogate model, as shown in the surrogate model updated in step S705, to assist in the prediction of vertex models and the final integration search, as shown in step S706.

[0157] The method for searching neural network ensemble models provided in this application embodiment is compared with existing architecture ensemble search benchmarks NES-RS and NES-RE on the NAS-Bench-201 search space. Table 1 shows the comparison data of the method for searching neural network ensemble models provided in this application embodiment (denoted as DistriNAS-PM) and other methods on the NAS-Bench-201 search space. The results are the average validation set error rate (%) in 10 trials. (1 standard error). The method for searching neural network ensemble models provided in the embodiments of this application in Table 1 is denoted as DistriNAS-PM. Other methods include NES-RS, NES-RE, and Deep Ensemble. The number of evaluations, test errors, and confidence scores are compared on the datasets CIFAR10, CIFAR100, and ImageNet16-120, respectively.

[0158] As shown in Table 1, the DistriNAS-PM method proposed in this application evaluates 30 network architectures (N). Compared to NES-RS and NES-RE, the DistriNAS-PM method requires less than 1 / 3 of the search cost to find an ensemble model with comparable or even lower test errors. Furthermore, the architecture ensembles found in this application not only have low test errors, but their calibration level and confidence are also comparable to the optimal ensemble model found by NES-RE. The confidence level is measured by negative log likelihood (NLL); the lower the NLL, the higher the model calibration level.

[0159] Table 1

[0160]

[0161] Furthermore, the last row of Table 1 includes a deep ensemble benchmark: the base learners in the architecture ensembles found in this application's embodiments are each trained using k different initialization weights to evaluate their corresponding deep ensemble performance. The results in Table 1 further demonstrate that ensemble models composed of different architectures are superior to ensemble models composed of different initialization weights.

[0162] Figure 6 This is a comparison graph of test errors obtained on CIFAR10 for various benchmarks, including the DistriNAS-PM provided in this application. Figure 6 As shown, the test errors obtained on CIFAR-10 for each benchmark were compared as the search progressed and the number of architecture evaluations gradually increased. This included ensemble models searched by different methods in the NAS-Bench-201 search space, such as DistriNAS-PM, NES-RS, and NES-RE, as well as the deep ensemble models corresponding to each architecture found by DistriNAS in this application, and the optimal base learner. The validation set error rate (%) on the CIFAR-10 dataset was considered to be as low as possible. Figure 8 The upper x-axis represents the number of architecture evaluations.

[0163] Depend on Figure 6 It is evident that the architecture ensemble search method (distriNAS-pm) proposed in this application embodiment can find architecture ensembles with lower verification errors faster than NES-RS and NES-RE, greatly reducing search costs; compared with architecture ensemble and deep ensemble, architecture ensemble achieves lower verification errors; all ensemble models are significantly better than the optimal single neural network architecture / base learner.

[0164] To verify the effectiveness of the search method in this application embodiment on a larger and more realistic search space, a further comparison can be made on the DARTS search space with NES-RS using different search costs. Table 2 shows the comparison data of the proposed method (DistriNAS-PM) and NES-RS using different search costs on the CIFAR10 task in the DARTS search space. The results are the average validation set error rate (%) across three trials.

[0165] Table 2

[0166]

[0167] The results in Table 2 show that, in the DARTS search space, the architecture integration search method proposed in this application requires only 1 / 8 of the search cost to find a better architecture integration than that found by the existing benchmark NES-RS.

[0168] In the application scenario of this embodiment, compared with the existing methods, the process of this embodiment can search for neural network ensemble models with similar performance in less time and computational cost, which not only speeds up the search efficiency of ensemble models but also improves the accuracy of architecture ensemble search.

[0169] Compared to existing methods, the embodiments of this application can search for neural network architecture ensembles with higher accuracy using the same time and cost. In practical production, using the embodiments of this application allows for the search of higher-accuracy convolutional neural network architecture ensembles for image classification tasks in less time and with less computation. This beneficial effect is mainly because the embodiments of this application employ architecture distribution search to find a candidate pool of base learners (first stage) and use a combination of surrogate models and vertex architectures to find high-quality ensemble models (second stage), greatly reducing the need for extensive training and evaluation of individual neural network architectures (single base learners).

[0170] Example 2

[0171] Example 2 examines the robustness of the embodiments of this application to OOD data through an image classification application. OOD data is very common in many real-world applications, such as autonomous driving and medical image diagnosis. The embodiments of this application search for high-performance neural network ensemble models on common search spaces and datasets, including DARTS and NAS-Bench-201, and CIFAR10 and CIFAR100 datasets. Validation samples perturbed to varying degrees by different noise levels are used to evaluate the test error, calibration level, and rejection capability of the architectures searched by different methods on OOD data. Validation samples perturbed by various noise levels include CIFAR10-C and CIFAR100-C.

[0172] CIFAR10-C and CIFAR100-C are generated by adding one of 15 random perturbations / noises to the validation set images of CIFAR10 and CIFAR100, respectively. The perturbation / noise intensity (shift severity) has 5 levels from low to high. The higher the intensity level, the greater the shift in the image generation distribution, that is, the greater the difference in distribution between the perturbed image and the original image.

[0173] Figure 7 Fifteen types of interference / noise were randomly selected and added to the CIFAR10 and CIFAR100 validation set images.

[0174] Figure 8 The effect of randomly selecting interference / noise added to images in the CIFAR10 and CIFAR100 validation sets; Figure 8 The higher the level of interference / noise, the greater the difference in (distribution) between the generated interfered image and the original image.

[0175] First, the method DistriNAS-PM proposed in this application embodiment is compared with the existing architecture integration search methods NES-RS and NES-RE on the CIFAR100-C OOD task in the NAS-Bench-201 search space. Similar to the comparison results in Embodiment 1, the architecture integration of NES-RS and NES-RE is found after evaluating 100 neural network architectures, while the DistriNAS-PM proposed in this application embodiment only needs to evaluate 30 neural network architectures.

[0176] Figure 9 This diagram illustrates the OOD verification comparison between DistriNAS-PM provided in this application and other search methods on the NAS-Bench-201 space. Figure 9 As shown in the NAS-Bench-201 space, compared to NES-RS and NES-RE, the DistriNAS-PM embodiment of this application can find an architecture integration with lower error and better model calibration (lower NLL) on the OOD validation set at less than 1 / 3 of the cost.

[0177] like Figure 9 As shown, the architecture ensemble found by DistriNAS-PM not only outperforms NES-RS and is comparable to NES-RE in test error on the original image (severity = 0), but also, in OOD scenarios with varying degrees of data perturbation (severity = 2, 4), the architecture ensemble found by DistriNAS-PM even slightly outperforms the architecture ensembles found by NES-RS and NES-RE in both test error and model calibration level (NLL). This demonstrates that the search method proposed in this application can not only find good ensembles faster, but also find ensembles with higher robustness to OOD data.

[0178] It is worth mentioning that Figure 9 It also demonstrates the performance of the optimal base learner in the ensemble architectures found by different methods (highlighted modules). It can be found that the test error and NLL achieved by all ensemble models on OOD data are significantly lower than those of the optimal single neural network architecture / base learner, proving that the ensemble model mentioned in this application has better rejection capability and robustness when the data is disturbed or when the distribution of test data is inconsistent with the distribution of training data.

[0179] Similarly, the search was further performed on a larger DARTS search space and validated on the CIFAR10-C dataset. Table 3 shows the comparison data of the proposed method (DistriNAS-PM) and NES-RS with different search costs on the CIFAR10-C task within the DARTS search space. The results are the average validation set error rate (%) across three trials, as shown in Table 3.

[0180] Table 3

[0181]

[0182] The results in Table 3 show that, in the DARTS space, the method proposed in this application can also efficiently search for architecture integrations with smaller test errors on OOD data, better model uncertainty calibration (lower NLL), and better robustness.

[0183] In the application scenario of this embodiment 2, compared with existing methods, the method for searching neural network ensemble models provided in this application embodiment can search for neural network ensemble models with similar performance in less time and computational cost. This not only accelerates the search efficiency of ensemble models but also improves the robustness of architecture ensemble search to OOD data, achieving lower verification errors, higher model calibration, and more accurate uncertainty values. Applied to actual production, using this application embodiment can search for high-quality architecture ensembles suitable for high-risk or high-uncertainty use cases in less time and with less computation.

[0184] This application provides an efficient method for searching multi-neural network ensemble models. Based on a two-stage search framework that is more efficient and suitable for ensemble model search, it uses neural architecture distribution search instead of traditional NAS to quickly find a candidate pool of base learners, thereby avoiding repeated evaluation of similar network architectures and improving search efficiency. A surrogate model is used to quickly select the optimal and diverse vertex models in the candidate pool by predicting (rather than actual) performance to form the target set model.

[0185] Compared with existing methods, the method of this application can perform ensemble model search more efficiently, thereby greatly reducing search costs and significantly enhancing the feasibility of ensemble model search in more application scenarios. Existing methods rely on traditional NAS, requiring the search and evaluation of each basic learner to build a sufficiently large candidate pool. In contrast, this method utilizes distributed architecture search to quickly build a candidate pool of basic learners, simplifying the search space and reducing search difficulty.

[0186] When searching for and building the optimal ensemble model from the candidate pool of basic learners, this embodiment of the application uses a surrogate model to search for vertex models in the candidate pool. This ensures both high efficiency and the diversity of vertex models, and that the ensemble model can achieve a significant performance improvement compared to a single basic learner.

[0187] The embodiments of this application can be combined with various types of architecture distributed search, and are applicable to different search spaces. They are highly versatile and can be applied to different scenarios and tasks.

[0188] In addition to being applied to architecture search of convolutional neural networks, the embodiments of this application can also be potentially applied to other types of neural architecture search tasks with similar architecture search unit structures, as well as other tasks where using ensemble models can bring further benefits, such as recurrent neural network (RNN) architectures commonly used in natural language processing and deep transformer self-attention networks (represented by transformers) commonly used in natural language processing and vision tasks, to obtain a better measure of uncertainty.

[0189] This application provides an apparatus for searching neural network architecture ensemble models. The apparatus includes: a data acquisition module for acquiring a dataset, the dataset including samples and labels from a classification task; and an architecture distribution search module for searching using a neural network architecture distribution search algorithm, including: determining hyperparameters of the neural network architecture distribution; sampling a neural network architecture from the architecture distribution defined by the hyperparameters; training and evaluating the neural network architecture based on the samples and labels from the classification task to obtain performance metrics; determining neural network architecture distributions sharing the hyperparameters based on the performance metrics to obtain a candidate pool of base learners; the base learners being neural network architectures that conform to the architecture distribution requirements; the neural network architectures being composed of repeatedly stacked neural network architecture units; determining a surrogate model; the surrogate model being used to predict the test performance of unevaluated neural network architectures; and combining architecture ensemble models to predict the test performance of base learners in the candidate pool through the surrogate model, thereby determining those conforming to the classification task requirements. An ensemble model is composed of several base learners, and the size of the ensemble model is [value missing]. .

[0190] This application provides an electronic device 1000, such as... Figure 10As shown, it includes a processor 1001 and a memory 1002; the processor 1001 is used to execute computer execution instructions stored in the memory 1002, and the processor 1001 executes the computer execution instructions to perform the method of neural network structure search based on evolutionary learning described in any of the above embodiments.

[0191] This application provides a storage medium, including a readable storage medium and a computer program stored in the readable storage medium, the computer program being used to implement the method for searching neural network structures based on evolutionary learning as described in any of the above embodiments.

[0192] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0193] Furthermore, various aspects or features of the embodiments of this application can be implemented as methods, apparatus, or articles of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" as used in this application encompasses a computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disks, floppy disks, or magnetic tapes), optical discs (e.g., compact discs (CDs), digital versatile discs (DVDs), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROMs), cards, sticks, or key drives, etc.). Additionally, the various storage media described herein may represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0194] It should be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0195] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0197] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0198] If the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application embodiment, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or an access network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0199] The above description is merely a specific implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.

Claims

1. A method for searching ensemble models of neural network architectures, characterized in that, The method includes: Obtain a dataset, which includes samples and labels from a classification task, including an image classification task; The algorithm uses a neural network architecture distribution search to search for a candidate pool of basic learners, including: determining the hyperparameters of the neural network architecture distribution; sampling a neural network architecture from the architecture distribution defined by the hyperparameters; training and evaluating the neural network architecture based on samples and labels in the classification task to obtain performance metrics; determining the neural network architecture distribution sharing the hyperparameters based on the performance metrics to obtain a candidate pool of basic learners; the basic learner is a neural network architecture that meets the requirements of the architecture distribution; the neural network architecture is composed of repeated stacking of neural network architecture units; A surrogate model is determined using each sampled neural network architecture unit and the performance metric; the surrogate model is used to predict the test performance of unevaluated neural network architectures. The test performance of the base learners in the candidate pool is predicted using a surrogate model to determine those that meet the requirements of the classification task. An ensemble model is composed of several base learners, and the size of the ensemble model is [missing value]. ,include: The proxy model is used to predict the test performance of multiple base learners in the candidate pool. Based on the prediction results, a region search is performed to determine q estimated vertex architectures. The estimated vertex architectures are neural network architectures whose performance metrics predicted by the proxy model on the validation set are higher than those of the adjacent architectures. The k architectures that meet the performance requirements from the q predicted vertex architectures are combined to obtain the ensemble model.

2. The method for searching an integrated model of a neural network architecture according to claim 1, characterized in that, The process of using a distributed search algorithm based on a neural network architecture to obtain a candidate pool of basic learners includes: The ANASOD algorithm, based on the probability distribution of learning operators, is used to search for the distribution of neural network architectures and obtain a candidate pool of basic learners.

3. The method for searching an integrated model of a neural network architecture according to claim 1 or 2, characterized in that, The hyperparameters for determining the distribution of the neural network architecture include: The hyperparameter for determining the distribution of the neural network architecture is the ANASOD code; the ANASOD code is a vector indicating the probability distribution of various operators in the neural network architecture unit, and the mapping between the ANASOD code and the neural network architecture unit is one-to-many.

4. The method for searching an integrated model of a neural network architecture according to claim 1 or 2, characterized in that, The hyperparameters for determining the distribution of the neural network architecture include: A search strategy is employed to optimize the hyperparameters of the neural network architecture distribution, wherein the search strategy is Bayesian optimization.

5. The method for searching an integrated model of a neural network architecture according to claim 3, characterized in that, Sampling a neural network architecture within the architecture distribution defined by the hyperparameters includes: Based on the operator probability distribution defined by the ANASOD encoding, determine the specific number of each operator in the constituent units of the neural network architecture; An effective neural network architecture is obtained by connecting different operators according to the defined search space.

6. The method for searching an integrated model of a neural network architecture according to claim 1 or 2, characterized in that, The process of training and evaluating the neural network architecture based on samples and labels in the classification task to obtain performance metrics includes: Train the neural network architecture on the training dataset; The neural network architecture is evaluated on a validation dataset to obtain performance metrics; the training dataset and the validation dataset belong to the same dataset.

7. The method for searching an integrated model of a neural network architecture according to claim 1 or 2, characterized in that, The method of using a distributed search algorithm based on a neural network architecture to obtain a candidate pool of basic learners also includes: The search strategy for the neural network architecture distribution is determined based on the performance metrics and hyperparameters of the neural network architecture distribution.

8. The method for searching an integrated model of a neural network architecture according to claim 1 or 2, characterized in that, The method of using a distributed search algorithm based on a neural network architecture to obtain a candidate pool of basic learners also includes: Based on the hyperparameters and performance metrics of the neural network architecture distributions found in each search, determine the performance prediction values ​​of hyperparameters for other unknown distributions, including mean and variance; The performance prediction strategy for the neural network architecture distribution is determined based on the mean and variance.

9. The method for searching an integrated model of a neural network architecture according to claim 1 or 2, characterized in that, The step of determining the distribution of neural network architectures sharing the hyperparameters based on the performance metrics and obtaining a candidate pool for the basic learners includes: The search strategy for the neural network architecture distribution is determined based on the performance metrics and the hyperparameters. The performance prediction strategy for the neural network architecture distribution is determined based on the performance metrics and the neural network architecture units. Based on the search strategy and performance prediction strategy, a candidate pool of basic learners is determined by searching within the neural network architecture distribution that shares the hyperparameters.

10. The method for searching an integrated model of a neural network architecture according to claim 1 or 2, characterized in that, The step of determining the distribution of neural network architectures sharing the hyperparameters based on the performance metrics and obtaining a candidate pool for the basic learners includes: Based on multiple neural network architectures and corresponding performance metrics from historical searches, output multiple neural network architectures that share the hyperparameters; The distribution of neural network architectures that meet the requirements is determined based on the multiple neural network architectures that share the hyperparameters; Based on the required neural network architecture distribution, multiple neural network architecture units are generated to obtain the generation distribution / candidate pool of the basic learner.

11. The method for searching an integrated model of a neural network architecture according to claim 1 or 2, characterized in that, The determination of the agent model includes: The surrogate model is obtained by training on the dataset based on the neural network architecture units and the performance metrics.

12. The method for searching an integrated model of a neural network architecture according to claim 1 or 2, characterized in that, The k architectures that meet the performance requirements from the q estimated vertex architectures are combined, including: Sort the performance metrics of the q predicted vertex architectures from best to worst, and then combine the k architectures with the best performance metrics.

13. The method for searching an integrated model of a neural network architecture according to claim 1 or 2, characterized in that, The k architectures that meet the performance requirements from the q estimated vertex architectures are combined, including: Using a greedy algorithm, we iterate through q estimated vertex architectures and add k architectures one by one to form an ensemble model.

14. An apparatus for searching ensemble models of neural network architectures, characterized in that, The device includes: The data acquisition module is used to acquire a dataset, which includes samples and labels from a classification task; the classification task includes an image classification task. An architecture distribution search module is used to search using a neural network architecture distribution search algorithm to obtain a candidate pool of basic learners. This includes: determining hyperparameters for the neural network architecture distribution; sampling a neural network architecture from the architecture distribution defined by the hyperparameters; training and evaluating the neural network architecture based on samples and labels in the classification task to obtain performance metrics; determining neural network architecture distributions sharing the hyperparameters based on the performance metrics to obtain a candidate pool of basic learners; the basic learner is a neural network architecture that meets the requirements of the architecture distribution; the neural network architecture is composed of repeatedly stacked neural network architecture units; determining a surrogate model using each sampled neural network architecture unit and the performance metrics; and the surrogate model is used to predict the test performance of unevaluated neural network architectures. The architecture integrates a model combination module, which is used to predict the test performance of the base learners in the candidate pool through a surrogate model, and determine those that meet the requirements of the classification task. An ensemble model is composed of several base learners, and the size of the ensemble model is [missing value]. include: The proxy model is used to predict the test performance of multiple base learners in the candidate pool. Based on the prediction results, a region search is performed to determine q estimated vertex architectures. The estimated vertex architectures are neural network architectures whose performance metrics predicted by the proxy model on the validation set are higher than those of the adjacent architectures. The k architectures that meet the performance requirements from the q predicted vertex architectures are combined to obtain the ensemble model.

15. An electronic device, characterized in that, include: Processor and memory; The memory is used to store computer execution instructions, and the processor is used to run the computer execution instructions in the memory to perform the method according to any one of claims 1-13.

16. A storage medium, characterized in that, It includes a readable storage medium and a computer program stored in the readable storage medium, the computer program being used to implement the method of any one of claims 1-13.

Images