Neural architecture search system using training based on weight-related metrics

Abstract

Systems and methods for performing a neural architecture search are provided. In one aspect, a system includes a processor configured to: select a plurality of candidate neural networks within a search space; performance of each of the plurality of candidate neural networks is evaluated by training each candidate neural network on a training dataset to perform a predetermined task and determining a ranking indicator for each candidate neural network based on an objective function. The ranking indicators include weight-related indicators that are determined based on weights of the prediction layer of each respective candidate neural network before and after the respective candidate neural network is trained. The processor is configured to rank the plurality of candidate neural networks based on the determined ranking indicator.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 17 / 816,167, filed on July 29, 2022, entitled “NEURALURALARCHITECTURE SEARCHSYSTEM USING TRAINING BASED ON A WEIGHT-RELATEDMETRIC,” the entire contents of which are incorporated herein by reference. Background Art

[0003] Machine learning is a developing field of research in which models are trained to perform inference tasks based on training data using training algorithms. One type of machine learning model is a neural network. Neural networks have been applied to perform various inference tasks in fields such as computer vision and natural language processing. A neural network includes a layer of nodes connected by weighted connections. The layer can be configured to perform different functions, such as convolution, pooling, deep learning, classification, and regression. The layer is configured by adjusting the connection topology and activation function of the node. In this way, the node layer in the neural network can be formed into functional building blocks, and those functional building blocks can be arranged into different neural network architectures to complete different training and inference tasks. These architectures can be grouped into different network types and for each network type, various specific architectures can be generated. Common neural network types include convolutional neural networks (CNNs), recurrent neural networks (RNNs), bidirectional long short-term memory (LSTM) RNNs, encoder-decoder transformers, encoder-only transformers, twin networks, etc. The neural network is trained at training time based on a training data set, and the weights of the connections are adjusted using algorithms such as back propagation. At inference time, the trained neural network is configured to receive inference time inputs and generate inference time outputs in response. A trained neural network will generally perform more accurate predictions at inference time if the inference time inputs are similar to the training time inputs.

[0004] It will be appreciated that the performance of neural networks depends heavily on their architectural design, which can require a high level of expertise and a lot of time and effort. Instead of time-consuming manual design, neural architecture search (NAS) has been proposed as a technique to automate the design of neural networks by searching for network structures for a given prediction task to find an efficient and effective architecture for the model. Early NAS methods used reinforcement learning or evolutionary algorithms to search for networks. These methods use controllers to generate several networks and the performance of these networks is used as feedback information to update the controller in an iterative loop. However, due to the large number of candidate networks being fully trained and evaluated during the iterative search process, traditional NAS algorithms are often slow and expensive, reaching costs of thousands of GPU days in some NAS algorithms.

[0005] There are different methods to speed up the NAS algorithm. One category of such methods includes using weight sharing in a hypernetwork. In an implementation called Efficient Neural Architecture Search (ENAS) via parameter sharing, weights are shared between candidate networks so that they can be trained simultaneously. Another method includes Differentiable Architecture Search (DARTS), which cascades all candidate operations into a hypernetwork, and each operation is assigned an architecture parameter representing its importance. During training, architecture parameters and weight parameters are optimized alternately. Another class of weight sharing methods is called one-shot NAS, in which the hypernetwork is trained using a subnetwork randomly sampled in each iteration.

[0006] Another class of solutions that aims to speed up the search process includes training-free NAS methods. These methods propose to predict network performance without training. One such method evaluates the effectiveness of different initial pruning criteria for NAS. Another method, called training-free neural architecture search (NASWOT), uses the number of linear regions (LRs) to rank different networks. In another framework, called training-free neural architecture search (TE-NAS), the number of LRs can be further combined with the neural tangent kernel (NTK) to rank the network according to its expressiveness and trainability. Compared to previous NAS methods, these training-free metrics have incrementally reduced the search cost to a few minutes while still obtaining competitive results. Summary of the invention

[0007] A system for performing a neural architecture search is described in the present disclosure. In one aspect, the system includes at least a processor and a memory, wherein the memory includes a neural architecture search program, which, when executed by the processor, causes the processor to perform a neural architecture search. The processor is configured to receive search space data, the search space data indicating a search space of a neural network configured to perform a predetermined task. A number of candidate neural networks are selected from the search space data. The candidate neural networks are then evaluated based on an objective function with the aid of a training process, the training process including training on a training data set to perform the predetermined task. Based on the training process, at least one ranking metric is calculated. The ranking metric can be a weight-related metric determined based on weights of a prediction layer of the candidate neural network. The candidate neural networks are then ranked based on their ranking metrics.

[0008] This summary is provided to introduce various concepts that will be further described in the following detailed description in a simplified form. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that address any or all of the disadvantages mentioned in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The present description will be more fully understood with reference to the following drawings, which present exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention.

[0010] Figure 1 A schematic diagram of an example computing system configured to perform a neural architecture search using a ranking metric that has a weak or no correlation with the number of parameters (#Param) in a model architecture is shown in accordance with an implementation of the present disclosure.

[0011] Figure 2 is a flowchart conceptually illustrating a neural architecture search (NAS) method including ranking and utilizing a neural network according to an implementation of the present disclosure. The neural architecture search (NAS) method may use Figure 1 system or other suitable hardware and software to implement.

[0012] Figure 3 It shows the use of Figure 1 Table of comparison of Kendall's Tau for two angle indicator variables on randomly sampled networks with different and same #Param for systems sampled from .

[0013] Figure 4 is a flowchart conceptually illustrating a NAS method according to an implementation of the present disclosure, the NAS method utilizing an angle index and a training loss index to rank candidate neural networks, the NAS method being Figure 2 A variation of the method.

[0014] Figure 5 is a flow chart conceptually illustrating an iterative network training process having a predetermined number of iterations, which may be used to implement Figure 2 Steps of the method.

[0015] Figure 6 is a flowchart conceptually illustrating a process for generating a proxy dataset from a target image dataset separated into different categories, which process can be used to implement Figure 2 Steps of the method.

[0016] Figure 7A-7C Shows that it can be Figure 1 An example of sampling a training dataset from a target image dataset used in conjunction with a system.

[0017] Figure 8 is a table showing the comparison of the test accuracy using the #Param metric and the no-training metric LR1, LR2, and NTK, which illustrates the Figure 1 The results obtained by the system.

[0018] Fig. 9 is a table showing Kendall's Tau comparison between different untrained metrics and #Param on NAS-Bench-201, showing the Figure 1 The results obtained by the system.

[0019] Fig.10 is a graph showing the test accuracy (%) of different ranking metrics evaluated on neural networks with the same number of parameters, the graph illustrates the Figure 1 The results obtained by the system.

[0020] Fig.11 is a table showing the Kendall's Tau comparison between angle, loss and angle loss and #Param, which illustrates the Figure 1 The results obtained by the system.

[0021] Fig.12 is a table showing the comparison of the test accuracy of different ranking metrics based on random search on NAS-Bench-201, which illustrates the Figure 1 The results obtained by the system.

[0022] Fig.13 is a table showing the comparison of test accuracy and search cost on NAS-Bench-201 based on pruning-based search for different methods. The table illustrates the Figure 1 The results obtained by the system.

[0023] Fig.14 is a table comparing the angle loss metric with other methods on the DARTS space CIFAR-10 dataset. The table illustrates the Figure 1 The results obtained by the system.

[0024] Fig.15 is a table comparing the angle loss metric with other methods on the DARTS space ImageNet-1K dataset. The table illustrates the Figure 1 The results obtained by the system.

[0025] Fig.16 is a table showing an ablation study of different training hyperparameters on the CIFAR-100 dataset, which illustrates the Figure 1 The results obtained by the system.

[0026] Fig.17 The diagram can be used to implement Figure 1 An example computer environment for a system of FIG. DETAILED DESCRIPTION

[0027] As briefly discussed above, different approaches have been proposed to alleviate the problem that traditional NAS methods consume a lot of time and resources. One such approach involves using differentiable search on candidate networks sampled from a hypernetwork. During training, network parameters and architecture parameters are optimized alternately. In order to further reduce the search cost, training-free metrics have been proposed to rank candidate networks without any training process. These metrics are mainly inspired by pruning methods and theoretical findings in deep neural networks. They aim to rank networks from different aspects of network properties such as trainability and expressibility. These metrics can achieve competitive results with much smaller search costs using previous NAS methods.

[0028] While initial evaluations indicate that the aforementioned untrainable metrics are able to estimate network performance, further exploration of these methods revealed that their network performance either has a poor correlation with their actual performance or their ranking ability comes primarily from a high correlation with an ignored metric, the number of parameters present in the model architecture (#Param). #Param is an untrainable metric that intuitively contributes to the ranking ability of the corresponding methods of those metrics due to its high correlation with previously proposed untrainable metrics (e.g., linear area, neural tangent kernel).

[0029] Motivated by the above observations, systems and methods for neural architecture search according to various embodiments of the present invention are presented. In many embodiments, neural architecture search includes the use of a ranking metric that is an effective indicator of neural network performance while having a weak or no correlation with #Param. The performance of neural architecture search utilizing such a ranking metric becomes more pronounced in search spaces where #Param information is not available, as the performance of traditional untrained metrics such as those described above drops dramatically in such cases. In some embodiments, the ranking metric used is a lightweight metric that is computationally inexpensive, allowing for efficient and fast training-based neural architecture search.

[0030] Figure 1 A schematic diagram of a computing system 100 having a computing device 102 configured to perform a neural architecture search using a ranking metric that has a weak correlation or no correlation with the number of parameters (#Param) in a neural network model is shown in accordance with an implementation of the present disclosure. As shown, the computing device 102 includes a processor 104 (e.g., a central processing unit or "CPU") and a memory 106 (e.g., volatile and non-volatile memory) operably coupled to each other. The memory 106 stores a NAS program 108 that contains instructions for various software modules described herein that are executed by the processor 104. The memory 106 also stores data 110 used by the NAS program 108 and its software modules.

[0031] When executed by processor 104, the instructions stored in NAS program 108 cause processor 104 to retrieve initialization data 112 from data 110 stored in memory 106 for use by NAS program 108. Figure 1In the depicted system, the initialization data 112 includes search space data 114 that is used as input to a network selection module 116. The search space data 114 for a NAS application defines or describes the neural network architectures in which the algorithm searches to find the desired architecture for the specific needs of the application. For example, the search space data 114 may define possible neural network architectures configured to perform a predetermined task. Different network architectures may also be implemented, such as, but not limited to, convolutional neural networks, recurrent neural networks, transformers, subnetworks. The network selection module 116 generates a set of candidate neural networks 118 by selecting neural networks from the search space data 114. Depending on the specific application, different selection methods may be used. For example, a random or random sampling of only a subset of neural architectures in the search space may be performed. Since only a subset of architectures are evaluated, an approximation of the best efficiency neural network is generated among all candidate architectures. Such sampling methods are typically used in an initialization phase where there is no additional information. In other applications, the set of candidate neural networks 118 includes every neural network architecture in the search space, and the selection method may be to select all neural network architectures in the search space data 114. Such an approach can improve the estimation quality of the most efficient neural network architectures while still remaining computationally feasible due to the efficient evaluation method described in this paper.

[0032] exist Figure 1 In the depicted system, the retrieved initialization data 112 includes a target dataset 120. The type of target dataset 120 used depends on the specific application and the neural network architecture implemented. For example, a NAS program for finding an effective neural network architecture for image classification can utilize a target dataset including images. Alternatively, the NAS program can search for different prediction tasks among neural architectures, such as natural language processing tasks (translation, next word in a sequence or sentiment classification, etc.), graph-based prediction (social network search, route optimization, etc.), etc. The training dataset selection module 122 generates a training dataset 124 of appropriate form by selecting from the target dataset 120. Depending on the specific application, different selection methods can be used. For example, arbitrary or random sampling can be performed when the evaluation of the entire search space may be too costly or time-consuming or simply unnecessary. In such a case, the training dataset can be generated from several sampling iterations. In other applications, the entire target dataset 120 is used as the training dataset 124. The training module 126 receives the candidate neural network set 118 and the training data set 124, and trains the candidate neural network 118 to perform a predetermined task (e.g., image classification, natural language processing, etc.) by using the training data set 124, and evaluates the performance of each candidate neural network in the candidate neural network 118 based on the objective function.

[0033] The neural network training process can be performed in many different ways. For example, the training module 126 can train each candidate neural network 118 in an iterative training process. In a further configuration, the candidate neural network 118 is trained for a predetermined number of iterations. From the neural network training process, at least one ranking metric 128 is derived for each candidate neural network 118. As described above, ranking metrics that have a weak correlation or no correlation with #Param can be used to provide meaningful and effective information when estimating the performance of the neural network. For example, the ranking metric can include a weight-related metric that is determined based on the weights of the prediction layer of the corresponding candidate neural network before and after the corresponding candidate neural network is trained. Further details of these and other metrics are discussed in the following sections. Using the derived ranking metric 128, the ranking module 130 can generate a ranked list 132 of the candidate neural networks 118 according to certain predetermined criteria.

[0034] although Figure 1 A specific computing device for performing a neural architecture search is illustrated, but other devices and configurations may be implemented depending on the specific requirements of a given application. For example, the target data set and the search space data may be stored on a memory module (e.g., a non-volatile mass storage device) separate from the NAS algorithm program. In addition to different system configurations, the NAS program may be implemented in different ways. For example, the NAS method may include steps for utilizing and deploying a ranked list of candidate neural networks for a particular application.

[0035] Figure 2is a flow chart conceptually illustrating an efficient NAS method 200 including ranking and utilizing neural networks according to an implementation of the present disclosure. At step 202, the method includes receiving search space data and a target data set. At step 204, the method selects a set of N candidate neural networks from the search space data. The method also selects a training data set that is a subset of the target data set. The method then performs an iterative training loop including steps 206 and 208. At step 206, the method trains an i-th candidate neural network from the set of N candidate neural networks on the training data set. At step 208, the method calculates at least one ranking metric for the i-th candidate neural network. Different ranking metrics may be calculated depending on the specific implementation. In method 200, step 208 calculates at least one ranking metric, the at least one ranking metric including a weight-related metric for the i-th candidate neural network. After N training loops, the ranking metric is calculated for each candidate neural network in the set. At step 210, the method ranks the candidate neural networks using their associated calculated ranking metrics. The ranking method may be based on the type of ranking metric or the metric calculated and / or the intended implementation of the neural network. At step 212, the method selects the highest ranked candidate neural network. At step 214, the method deploys the highest ranked candidate neural network. Deployment of the highest ranked candidate neural network may include implementing the neural network in an external system for use in another application.

[0036] Effective NAS methods such as those described above can be implemented in different variants, with different variants utilizing different datasets and ranking metrics to determine the ideal neural network architecture based on a given objective function. In applications where fast and efficient training is desired, NAS methods based on shortened training can be implemented. These and other implementations are discussed in more detail below.

[0037] The ranking metric or metrics used in the NAS method are an important part of determining the effectiveness of the corresponding method in estimating the performance of the neural network. Intuitively, the performance of the network after training is indicated by the structure (e.g., #Param, number of layers), weight initialization, and dynamics during training (e.g., loss, gradient). Metrics generated by training dynamics are usually weakly correlated with #Param. For example, certain metrics derived from the weights of the network structure are highly correlated with the accuracy of the neural network performance, but are poorly correlated with #Param. This indicates that such metrics can provide additional information about the estimated network performance that #Param does not have.

[0038] A weight-related, lightweight (i.e., fast to compute and small in storage) metric is the angle metric, which is a training dynamic that indicates the generalization ability of a network and can be calculated differently depending on the specific implementation. In one example, the angle metric is defined as the angle between the one-dimensional vectorized network weights before and after training. Consider W0 to represent the weights of the network N at initialization, and W t represents the weight after training. The angle index can then be defined as:

[0039]

[0040] Where W0·W t is W0· and W t It can be easily understood that similar or other methods can be used to describe the angle index.

[0041] Typically, the angular metric is positively correlated with the final performance of the network. However, the angular metric may behave differently at different network stages (i.e., when examining different layers or groups of layers within the network). Specifically, in some cases, the angular metric calculated using the weights from the feature extraction layer of the neural network is positively correlated with the final accuracy of the network. In most NAS search spaces, the feature extraction stage consists primarily of a stack of network modules that typically include convolutional layers. On the other hand, the angular metric calculated using the weights of the prediction layer (e.g., the final fully connected layer of multiple hidden layers downstream of the feature extraction layer) is typically negatively correlated with the performance of the network. The angular metric for the feature extraction layer can be denoted as θ feat , and the angle index of the prediction layer can be expressed as θ pred . Figure 3 is a graph showing the two angle indicator variables θ on randomly sampled networks with different and same #Param feat and θ pred Table comparing the Kendall rank correlation coefficient (Kendall's Tau) of . Tested on 50 randomly sampled networks from NAS-Bench-201, a commonly used benchmark for neural architecture search with a fixed search space. The sampled networks are fully trained on CIFAR-10, an image dataset commonly used for computer vision training. As shown in the figure, for the sampled networks with different #Param, θ feat is positively correlated with the accuracy of the network, while θ pred Negatively correlated with the accuracy of the network. θ pred With a ratio of θ feat Higher correlation. For sampled networks with the same #Param, θ feat The Kendall's Tau of θ drops significantly to about zero. Therefore, θ featmay be a poor indicator for ranking neural networks without #Param information. On the other hand, θ pred Kendall's Tau shows a correlation level indicating its ability to effectively predict network performance without #Param metrics.

[0042] In addition to angular metrics, training loss can also be used as an effective, lightweight metric to achieve good performance without any or more additional computation. For example, comparing training loss to θ pred The combination gives richer information about the model performance without increasing computational cost. pred Unlike the scale of the training loss, directly increasing their values ​​during the ranking process can cause one to significantly outperform the other. To avoid this problem, the network can be ranked using both metrics separately. Their ranking indices can then be used to provide the final ranking. pred Both θ and training loss are negatively correlated with model accuracy. Thus, for some calculations, negative values ​​of both metrics can be made positive.

[0043] Figure 4 FIG. 1 is a flowchart conceptually illustrating a NAS method for ranking candidate neural networks using an angle metric and a training loss metric according to an implementation of the present disclosure. As shown in the figure, the method is similar in some aspects to Figure 2 , but with a different configuration for implementing steps 206-214. At step 208A, the method calculates a ranking metric, the ranking metric comprising an angle metric and a training loss metric for the current i-th candidate neural network being trained. At step 210A, the method ranks the candidate neural networks. Since more than one ranking metric is calculated, the ranking process may include specific steps for providing a final ranking. For example, the ranking step 210A includes performing a first ranking of the candidate neural networks based on the calculated angle metric, and performing a second ranking of the candidate neural networks based on the calculated training loss metric. At step 212A, the method selects the highest ranked candidate neural network based on the first ranking and the second ranking. At step 214A, the method deploys the selected highest ranked candidate neural network.

[0044] although Figure 4 Methods utilizing a particular ranking metric are described, but any of a number of other ranking metrics may be appropriately utilized depending on the specific requirements of a given application. For example, other weight-related metrics may be used. In some configurations, the ranking metric is a distance metric calculated from the weights of the prediction layer of the neural network. Distance metrics such as Minkowski distance, cosine distance, Mahalanobis distance, Hamming distance may be appropriately used.

[0045] Neural network training can be performed in a number of different ways. For example, training of the candidate neural network can be implemented in an iterative process. The candidate neural network can be trained using a reinforcement learning loop. In some applications, the candidate neural network can be trained to convergence. In other applications, it is highly desirable that the training process is fast and efficient. In such cases, the training process can be configured to perform a short training process in which the candidate neural network is trained for a predetermined number of iterations that is less than the number of iterations that will result in convergence, thereby resulting in a partially trained candidate neural network not being trained to convergence. Figure 5 FIG. 1 is a flow chart conceptually illustrating an iterative network training process with a predetermined number of iterations according to an implementation of the present disclosure. As shown in the figure, the process may be Figure 2 Specific implementation of step 206 of method 200 described in . At step 206A1, the process receives a set of N candidate neural networks and a training data set. The set of N candidate neural networks and the training data set can be generated using various methods. For example, similar to the methods described in the above sections, the candidate neural network can be generated by sampling from the search space of the network architecture, and the training data set can be generated by sampling from the target data set. The process includes an iterative training loop. At step 206A2, the i-th candidate neural network in the set of N candidate neural networks is trained for the j-th iteration of a predetermined number of M iterations. At step 206A3, the i-th candidate neural network is adjusted in the j-th iteration based on its performance during the j-th training iteration. Steps 206A2 and 206A3 are performed in two nested loops, where the inner loop describes an iterative training process for a single candidate neural network, and the outer loop describes an iterative training process for the set of N candidate neural networks. In this way, steps 206A2 and 206A3 are performed M×N times. It can be easily understood that Figure 5 The method described in can be extended to include steps of other processes described herein. For example, the process can include determining a training loss metric based on the performance of the corresponding candidate neural network on the training data set in the last iteration of a predetermined number of iterations.

[0046] Another process for achieving efficient training includes training the candidate neural network on a proxy dataset, which can reduce computational requirements. In some cases, such a training scheme can be thousands of times faster than traditional training, and can even be faster than a scheme that utilizes no training indicators. The proxy dataset can be generated by sampling the target dataset. Different sampling methods can be used, including random sampling. The specific method used can be related to the target dataset. For example, common machine learning and computer vision algorithms use target image datasets that are separated into different categories. The sampling method can take into account the classification of images in such a dataset. Figure 6FIG. 1 is a flowchart conceptually illustrating a process for generating a proxy dataset from a target image dataset separated into different categories according to an implementation of the present disclosure. As shown in the figure, the process can be implemented as follows: Figure 2 Different configurations of steps 202 and 204 of method 200 are described. At step 202A, the process receives a target data set including a set of categories, each category including a set of images. The process includes a sampling step 204A as an implementation of the selection step 204 of method 200. At step 204A1, the process samples a subset of categories from the target data set. At step 204A2, the process samples a subset of images within each sampled category. Figure 7A-7C An example of sampling a training dataset from a target image dataset according to an implementation of the present disclosure is shown. Fig. 7A A training dataset 700 comprising images 702 separated into categories 704 is shown. Figure 7B A training dataset 710 is shown that is sampled from the original training dataset 700. As shown, the sampled training dataset 710 includes images 702 from categories B and C. Images from category A are not sampled. Figure 7C Another sampled training data set 720 is shown, where another sampling is performed within each sampled class, thereby reducing the number of images in each sampled class.

[0047] It can be readily appreciated that the above methods can be used in different combinations depending on the specific requirements of a given application. For example, an efficient NAS method may include training a candidate neural network on a proxy dataset for a predetermined number of iterations. When a method includes the use of a proxy dataset and a limited number of training iterations, the effectiveness of certain ranking metrics may be different than when used in more traditional training methods. Therefore, it is important to understand how ranking metrics perform in such a setting. For example, θ pred The metrics and training loss metrics are effective for ranking candidate neural networks in a compact setting that utilizes a proxy dataset and a limited number of training iterations.

[0048] There are many types of ranking metrics for implementations using NAS methods. The effectiveness of certain ranking metrics can vary depending on the method being implemented. Selecting an appropriate ranking metric that is an effective indicator for ranking neural networks can be very difficult. For example, previous methods that relied on untrained metrics assumed that such metrics were effective in estimating the performance of the network. However, a closer look at such methods showed that the utilized metric was strongly correlated with #Param. Without further investigation, it is difficult to know whether the metric used or #Param contributes to the ranking ability of the method.

[0049] TE-NAS and NASWOT are two untrained methods that rely on the number of linear regions as a ranking metric. Linear regions are a well-studied theoretical criterion for indicating the learning ability of a network. The linear region metric describes how many regions the network can split the input space into. A larger number of LRs generally indicates that the network has higher performance. The number of LRs is estimated differently in TE-NAS and NASWOT (referred to as LR1 and LR2, respectively). While NASWOT feeds batches of samples to the network and calculates the Hamming distance between different activation patterns, TE-NAS calculates the linear region by forwarding batches of samples to the network and counting how many samples have different activation patterns. The Hamming distance between these activation patterns is used to define the kernel matrix K. The ranking metric is defined as the determinant of K. TE-NAS also utilizes the Neural Tangent Kernel (NTK) to score the network. The convergence speed of the network can be determined by the number of conditions of NTK. Intuitively, a faster convergence speed indicates that the network has higher performance. In this way, the number of conditions of NTK is also used to rank the network.

[0050] The above NAS methods and untrained metrics produce effective results. However, given that these untrained metrics are highly correlated with #Param, it is important to understand which metric contributes to the ranking ability of their methods. Figure 8 is a table showing the comparison of the test accuracy using the #Param metric and the no-training metric LR1, LR2, and NTK. The tests were performed on the benchmarks NAS-Bench-101 and NAS-Bench-201 on the datasets CIFAR-10, CIFAR-100, and ImageNet16-120. Figure 8 Results are shown for the performance of the top ranked neural networks based on the relevant metrics. The neural networks were selected from one hundred sampled networks from the appropriate search space. Test accuracy is presented as mean and standard deviation (n=5). As shown, all tested metrics yield high performance, with #Param performing best on NAS-Bench-201 and CIFAR-10CIFAR-100. Fig. 9 is a table showing Kendall's Tau between different untrained metrics and #Param on NAS-Bench-201. As shown, the untrained metrics have high correlation with #Param, especially the two linear region metrics. This makes intuitive sense because the number of linear regions is determined by 2 #activation An upper bound is set and the number of activation units is highly correlated with #Param. High correlation indicates that these untrained metrics or most of their ranking power may come from #Param.

[0051] To further validate the above observations, the different untrained metrics mentioned above can be evaluated on a reduced search space of neural networks with the same number of parameters. This prevents the metrics from using parameter information to evaluate the neural network. Fig.10 is a graph showing the test accuracy (%) of different ranking metrics evaluated on neural networks with the same number of parameters. The search space was made based on NAS-Bench-201. NAS-Bench-201 defines a cell-based search space, where each cell is represented as a densely connected directed acyclic graph (DAG). Each cell has four nodes and six edges, where each edge represents an operation. There are 5 candidate operations, including zeroing, skip connection, 1×1 convolution, 3×3 convolution, and 3×3 average pooling. Different models can have the same number of parameters but different structures and performances. Eight model groups were selected, and the models in the same group have the same number of parameters. For Fig.10 , the parameter M for this number of the eight selected groups is {0.37, 0.40, 0.59, 0.62, 0.64, 0.83, 0.86, 1.05}. The number of neural networks in each group is {1602, 540, 1602, 810, 180, 540, 180, 135} respectively. The neural networks are evaluated on the datasets CIFAR-10, CIFAR-100, and ImageNet16-120. Ten categories are randomly sampled, and ten images are sampled from each category. One hundred neural networks are randomly sampled from each network group, and the best performing network is selected based on the corresponding ranking metric. Each neural network is trained for 50 iterations at a fixed learning rate of 0.2. Other settings follow the settings in NAS-Bench-201. Unless otherwise stated, similar settings are used for other tests described in this disclosure.

[0052] exist Fig.10 In , the ranking metrics evaluated include LR1, NTK, LR2, loss, angle, and angle loss. A baseline of randomly selecting a neural network from the candidate neural networks is also included. The angle is represented using θ pred The search is performed using the training loss, and the angle loss indicates that the search is performed using a combination of the two metrics. For N = 5, the average accuracy and standard deviation are reported. Fig.10 As shown, with #Param (such as Fig. 9 ) has the highest correlation and has high performance on NAS-Bench-101 and CIFAR-10 (as shown in Figure 8LR2 (shown in Figure 2) performs the worst in this scenario, even worse than the random baseline. From these results, it can be inferred that the tested untrained metrics rely heavily on parameter information to rank the networks. On the other hand, the angular loss metric (a ranking metric with weak or no correlation with #Param) consistently outperforms the untrained metric on all neural network groups on the three datasets. In most cases, angular loss outperforms the untrained metric by more than 1%. Fig.11 is a table showing the comparison of Kendall's Tau between angle, loss and angle loss and #Param. As shown in the figure, the correlation is much lower than Fig. 9 Correlation of untrained metrics in .

[0053] NAS-Bench-201 is a cell-based search space with 15625 architectures. Following a unified training protocol, the performance of each network is evaluated on CIFAR-10, CIFAR-100, and ImageNet16-120. This benchmark makes it convenient for researchers to focus on search algorithms without network evaluation. Fig.12 It is a table showing the comparison of the test accuracy of different ranking indicators on NAS-Bench-201 based on random search. One hundred neural networks are randomly sampled from the search space, and different ranking indicators are used to select the one with the best performance. The average accuracy and standard deviation are reported (n=5). The search cost is measured on a single GTX-1080Ti GPU. #Param is added as a baseline indicator. As shown in the figure, #Param performs well on CIFAR-10 and CIFAR-100, even slightly better than the untrained indicators LR1, NTK, and LR2. Additionally, #Param is easy to calculate, and the search cost on 100 networks is only 3 seconds. The indicators LR1 and LR2 based on the linear region perform better and more stable than NTK. The performance of NTK is very low and has a lot of changes. Although LR1 and LR2 are both based on the linear region, LR2 is slightly better and more stable. As mentioned above, the effectiveness of the untrained indicators can be attributed to their high correlation with #Param.

[0054] By visualizing the searched network structures, the angular metric may get stuck in some trivial structures where most of the connections are zeroed, skipped or average pooled. In these trivial structures, the feature extraction layer does not learn anything meaningful and the prediction layer is optimized towards random directions in each training iteration. In this way, the weight vector of the prediction layer does not change much after training, which means that the angular metric and the extended angular loss metric will give high scores to these structures. By combining the angular loss metric with #Param, structures with a small number of parameters can be avoided. They can also be avoided when pruning-based search methods are implemented. As Fig.12As shown, when combined with #Param, the performance of the angle loss metric is significantly improved by about 10%, achieving higher performance than the untrained metric.

[0055] exist Fig.12 In , the results of combining the untrained metrics LR1, NTK, and LR2 with #Param are also shown. These untrained metrics benefit slightly from #Param, but the improvement is minor. Taking #Param as the baseline, its combination with the untrained metrics reduces its performance on CIFAR-10 and CIFAR-100. On the other hand, the angular loss metric combined with #Param achieves consistent improvements compared to only #Param on the three datasets. Similarly, angular loss + LR2 is an improvement over LR2 on all three datasets. From these results, it can be inferred that the angular loss metric provides orthogonal information to #Param and the untrained metrics. Combining them can improve performance.

[0056] TE-NAS combines LR1 and NTK into a pruning-based search method. For further comparison, the angular loss metric can be applied to the same pruning-based search method. The search starts with a supernetwork containing all operations on each edge. Then, it prunes the operations and uses the metrics (LR1 and NTK) to score the subnetwork. The operation that results in the lowest score will be finally pruned. This process is iteratively performed until only one operation is left for each edge. Due to longer convergence, the supernetwork is targeted for one hundred iterations.

[0057] Fig.13 is a table showing the comparison of test accuracy and search cost on NAS-Bench-201 based on pruning-based search for different methods. Means and standard deviations are reported (n=5). The performance of other NAS methods is included for reference. Two results are shown for TE-NAS, one is the result reported in the literature and the other is Reproduced using the official code. The reproduction performance is lower, and the search cost is also cheaper. The search cost is evaluated on a 1080Ti GPU. Fig.13 As shown in Figure 2, the above short training method using the angle loss metric is faster than TE-NAS. TE-NAS needs to calculate two metrics (LR1 and NTK), and for each metric, it repeats three times and takes the average to get better and stable performance. The short training method calculates the metric once.

[0058] Under pruning-based search, the angular loss metric does not show the same stuck problem as in random search. This is because pruning-based methods start from a hypernetwork, which is non-trivial. With a finite number of pruning steps, the neural network almost never reaches a trivial structure with a large number of empty operations. Fig.13As shown, the original results of TE-NAS outperform the angular loss metric on CIFAR-10 and CIFAR-100, but the search cost is 3 times more. At lower search cost, the angular loss metric performance is comparable to the reproduced results of TE-NAS. On ImageNet16-120, the angular loss metric outperforms TE-NAS in both cases. The angular loss metric is also combined with #Param with negligible additional search cost, which improves the performance on all three datasets by 0.2%-0.7%.

[0059] DARTS is a cell-based search space. The networks in the search space consist of stacks of normal and reduced cells whose architectures need to be searched. There are 8 candidate operations in the search space, including zeroing, skip connections, 3×3 and 5×5 separable convolutions, 3×3 and 5×5 dilated separable convolutions, 3×3 max pooling, and 3×3 average pooling. The angular loss metric is applied to the pruning-based search method used in TE-NAS for the following tests. The data sampling and short training scheme are similar to the above settings, except that the learning rate is 0.5.

[0060] Fig.14 is a table comparing the angular loss metric with other methods on the DARTS spatial CIFAR-10 dataset. After TE-NAS, in the evaluation phase, the network is stacked with 20 units and the initial number of channels is 36. The reduction units are located at 1 / 3 and 2 / 3 of the network, and each unit contains six nodes. Fig.14 As shown, the angular loss metric completes the search process in 0.09 days (i.e., 2.16 hours) on a single 1080Ti GPU. Unlike the results on NAS-Bench-201, in this case, the search cost is higher than TE-NAS because TE-NAS uses a smaller batch size to calculate NTK on DARTSCIFAR-10, resulting in less calculations. However, the search cost for angular loss is still much lower than other NAS methods. The angular loss metric also achieves comparable performance to TE-NAS, but the network size searched is much smaller. When combined with #Param, the angular loss metric achieves a lower test error of 2.56%, which is competitive with prior art methods.

[0061] Fig.15 is a table comparing the angular loss metric with other methods on the DARTS spatial ImageNet-1K dataset. The short training setting is the same as for CIFAR-10. For evaluation, the network is stacked with 14 units and the initial number of channels is 48. Fig.15In the upper half of the figure, the network is searched on CIFAR-10 and then evaluated on ImageNet-1K. As shown, the angle loss is competitive with state-of-the-art NAS methods while having a much lower search cost. Compared to TE-NAS, the performance is significantly better and the network size is much smaller. Fig.15 The lower half shows the results of different methods searching directly on ImageNet-1K. The pruning-based search with the angular loss metric is completed in only 0.11 GPU days (i.e., 2.64 GPU hours), which is even faster than TE-NAS. This is because TE-NAS uses a larger batch size on ImageNet-1K than on CIFAR-10. The angular loss metric is 30 times faster than other NAS methods. The performance of the angular loss metric alone is slightly lower than other methods with smaller model sizes. When combined with #Param, the performance of the angular loss metric is greatly improved and reaches a competitive top-1 / top-5 error rate of 24.1% / 7.1%, outperforming the listed differentiable and untrained methods.

[0062] Fig.16 is a table showing ablation studies of different training hyperparameters on the CIFAR-100 dataset. The different hyperparameters in the above short training scheme include the number of training iterations, sampling categories, images per category, and weight initialization methods. The above random search method is used as a baseline. As shown in the figure, longer training iterations tend to achieve better performance because they allow the network to converge better, which produces more informative angle indicators and training losses. However, ten training iterations can still achieve acceptable performance. Increasing the number of categories does not always improve performance. Although more categories can provide more information about the target dataset, it also makes the proxy dataset more complex, which makes it more difficult for the neural network to converge in limited iterations, producing less informative angle indicators and training losses. Similarly, increasing the number of images does not guarantee better performance. In order to achieve the best accuracy-efficiency trade-off, the training hyperparameters can be tuned. Due to the efficient nature of the short training method, it is feasible to tune the hyperparameters.

[0063] In some embodiments, the methods and processes described herein may be bound to a computing system of one or more computing devices. Specifically, such methods and processes may be implemented as a computer application or service, an application programming interface (API), a library, and / or other computer program products.

[0064] Fig.17 A non-limiting embodiment of a computing system 1700 that can perform one or more of the above methods and processes is schematically shown. The computing system 1700 is shown in a simplified form. The computing system 1700 can embody the above description and Figure 210. The computing system 1700 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), and / or other computing devices, as well as wearable computing devices such as smart watches and head-mounted augmented reality devices.

[0065] The computing system 1700 includes a logic processor 1702, a volatile memory 1704, and a non-volatile storage device 1706. The computing system 1700 may optionally include a display subsystem 1708, an input subsystem 1710, a communication subsystem 1712, and / or Fig.17 Other components not shown.

[0066] Logical processor 1702 includes one or more physical devices configured to execute instructions. For example, a logical processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform tasks, implement data types, transform the state of one or more components, implement technical effects, or otherwise achieve desired results.

[0067] The logical processor may include one or more physical processors (hardware) configured to execute software instructions. Additionally or alternatively, the logical processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of the logical processor 1702 may be single-core or multi-core and the instructions executed thereon may be configured for sequential, parallel and / or distributed processing. The various components of the logical processor may optionally be distributed in two or more separate devices, which may be remotely located and / or configured for coordinated processing. Various aspects of the logical processor may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration. In such a case, it will be understood that these virtualized aspects are run on different physical logical processors of various different machines.

[0068] The non-volatile storage device 1706 includes one or more physical devices that are configured to store instructions that can be executed by a logical processor to implement the methods and processes described herein. When such methods and processes are implemented, the state of the non-volatile storage device 1706 can be transformed to, for example, store different data.

[0069] The non-volatile storage device 1706 may include a removable and / or built-in physical device. The non-volatile storage device 1706 may include an optical memory (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), a semiconductor memory (e.g., ROM, EPROM, EEPROM, flash memory, etc.), and / or a magnetic memory (e.g., a hard disk drive, a floppy disk drive, a tape drive, MRAM, etc.) or other mass storage device technology. The non-volatile storage device 1706 may include a non-volatile, dynamic, static, read / write, read-only, sequential access, location addressable, file addressable, and / or content addressable device. It will be understood that the non-volatile storage device 1706 is configured to retain instructions even when power to the non-volatile storage device 1706 is cut off.

[0070] The volatile memory 1704 may include a physical device having random access memory. The volatile memory 1704 is typically used by the logical processor 1702 to temporarily store information during the processing of software instructions. It should be understood that when the power to the volatile memory 1704 is cut off, the volatile memory 1704 typically does not continue to store instructions.

[0071] Aspects of the logic processor 1702, volatile memory 1704, and non-volatile storage device 1706 may be integrated together into one or more hardware-logic components. Such hardware-logic components may include, for example, field programmable gate arrays (FPGAs), program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOCs), and complex programmable logic devices (CPLDs).

[0072] The terms "module", "program", and "engine" may be used to describe an aspect of a computing system 1700, which is typically implemented in software by a processor to perform a specific function using portions of volatile memory, which involves a transformation process that specifically configures the processor to perform the function. Thus, a module, program, or engine may be instantiated via a logical processor 1702 executing instructions stored by a non-volatile storage device 1706 using portions of volatile memory 1704. It should be understood that different modules, programs, and / or engines may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same module, program, and / or engine may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module", "program", and "engine" may cover individual or groups of executable files, data files, libraries, drivers, scripts, database records, and the like.

[0073] When included, the display subsystem 1708 can be used to present a visual representation of the data stored by the non-volatile storage device 1706. The visual representation can take the form of a graphical user interface (GUI). As the methods and processes described herein change the data stored by the non-volatile storage device, and thus transform the state of the non-volatile storage device, the state of the display subsystem 1708 can also be transformed to visually represent the change of the underlying data. The display subsystem 1708 can include one or more display devices that utilize virtually any type of technology. Such a display device can be combined with the logical processor 1702, the volatile memory 1704 and / or the non-volatile storage device 1706 in a shared shell, or such a display device can be a peripheral display device.

[0074] When included, input subsystem 1710 may include or interface with one or more user input devices, such as a keyboard, mouse, trackpad, touch-sensitive display, camera, or microphone.

[0075] When included, the communication subsystem 1712 can be configured to communicatively couple the various computing devices described herein to each other, as well as to communicatively couple to other devices. The communication subsystem 1712 can include wired and / or wireless communication devices compatible with one or more different communication protocols. As non-limiting examples, the communication subsystem can be configured to communicate via a wireless telephone network or a wired or wireless local area network or wide area network. In some embodiments, the communication subsystem can allow the computing system 1700 to send messages to other devices and / or receive messages from other devices via a network such as the Internet.

[0076] The following paragraphs provide additional support for the claims of the present application. One aspect provides a computer system for performing neural architecture search, the computer system comprising a processor coupled to a storage medium storing instructions, the instructions, when executed by the processor, causing the processor to: receive search space data indicating a search space of a neural network configured to perform a predetermined task; select a plurality of candidate neural networks within the search space, wherein each candidate neural network comprises a prediction layer; evaluate the performance of each of the plurality of candidate neural networks based on an objective function by: training each candidate neural network on a training data set to perform the predetermined task; and determining at least one ranking metric for each candidate neural network based on the objective function, wherein the at least one ranking metric comprises a weight-related metric determined based on a weight of a prediction layer of the corresponding candidate neural network before and after the corresponding candidate neural network is trained; and rank the plurality of candidate neural networks based on the determined ranking metric. In this aspect, additionally or alternatively, the weight-related metric is an angle metric indicating an angle between one-dimensional vectors calculated from weights of a prediction layer of the corresponding candidate neural network before and after the corresponding candidate neural network is trained. In this aspect, additionally or alternatively, the weight-related indicator is a distance indicator selected from the group consisting of: Minkowski distance, cosine distance, Mahalanobis distance, and Hamming distance. In this aspect, additionally or alternatively, at least one ranking indicator further comprises a training loss indicator, the training loss indicator being determined based on the performance of the corresponding candidate neural network on the training data set during training; each candidate neural network is trained for a predetermined number of iterations; and the training loss indicator is determined based on the performance of the corresponding candidate neural network on the training data set in the last iteration of the predetermined number of iterations. In this aspect, additionally or alternatively, the instructions, when executed by the processor, cause the processor to rank a plurality of candidate neural networks by: assigning a first ranking number to each candidate neural network based on the weight-related indicator of the candidate neural network; assigning a second ranking number to each candidate neural network based on the training loss indicator of the candidate neural network; and ranking the plurality of candidate neural networks based on the first and second ranking numbers. In this aspect, additionally or alternatively, the instructions, when executed by the processor, cause the processor to deploy the highest-ranked candidate neural network to the computing device. In this aspect, additionally or alternatively, the plurality of candidate neural networks comprises each neural network in the search space. In this aspect, additionally or alternatively, the at least one ranking metric further comprises a parameter metric indicating the number of parameters of the corresponding candidate neural network.In this aspect, additionally or alternatively, the training data set is generated by: receiving a target data set, the target data set comprising a plurality of input data elements classified into a plurality of categories; sampling a subset of the plurality of categories; and for each sampled category, sampling a subset of the plurality of input data elements within the sampled category. In this aspect, additionally or alternatively, the search space of the neural network comprises a plurality of convolutional neural networks, recurrent neural networks, transformers, subnetworks, and / or combinations thereof.

[0077] In another aspect, a method for evaluating the performance of a neural network architecture is provided, the method comprising: receiving search space data indicating a search space of a neural network configured to perform a predetermined task; sampling a plurality of candidate neural networks within the search space, wherein each candidate neural network comprises a prediction layer; evaluating the performance of each of the plurality of candidate neural networks based on an objective function by: training each candidate neural network on a training data set to perform the predetermined task; and determining at least one ranking metric for each candidate neural network based on the objective function, wherein the at least one ranking metric comprises a weight-related metric determined based on the weights of the prediction layer of the corresponding candidate neural network before and after the corresponding candidate neural network is trained; and ranking the plurality of candidate neural networks based on the determined ranking metric. In this aspect, additionally or alternatively, the weight-related metric is an angle metric indicating an angle between one-dimensional vectors calculated from the weights of the prediction layer of the corresponding candidate neural network before and after the corresponding candidate neural network is trained. In this aspect, additionally or alternatively, the weight-related metric is a distance metric selected from the group consisting of: Minkowski distance, cosine distance, Mahalanobis distance, and Hamming distance. In this aspect, additionally or alternatively, at least one ranking metric further comprises a training loss metric, the training loss metric being determined based on the performance of the corresponding candidate neural network on the target data set during training; each candidate neural network being trained for a predetermined number of iterations; and the training loss metric being determined based on the performance of the corresponding candidate neural network on the training data set in the last iteration of the predetermined number of iterations. In this aspect, additionally or alternatively, ranking a plurality of candidate neural networks comprises: assigning a first ranking number to each candidate neural network based on a weight-related metric of the candidate neural network; assigning a second ranking number to each candidate neural network based on the training loss metric of the candidate neural network; and ranking the plurality of candidate neural networks based on the first and second ranking numbers. In this aspect, additionally or alternatively, the method comprises deploying the highest ranked candidate neural network to a computing device. In this aspect, additionally or alternatively, the plurality of candidate neural networks comprises each neural network in the search space. In this aspect, additionally or alternatively, at least one ranking metric further comprises a parameter metric indicating the number of parameters of the corresponding candidate neural network. In this aspect, additionally or alternatively, the training data set is generated by: receiving a target data set, the target data set comprising a plurality of input data elements classified into a plurality of categories; sampling a subset of the plurality of categories; and for each sampled category, sampling a subset of the plurality of input data elements within the sampled category. In this aspect, additionally or alternatively, the search space of the neural network comprises a plurality of convolutional neural networks, recurrent neural networks, transformers, subnetworks, and / or combinations thereof.

[0078] It should be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered as restrictive, because many variations are possible. The specific routine or method described herein can represent one or more strategies in any number of processing strategies. Therefore, the various actions illustrated and / or described can be performed in the illustrated and / or described sequence, in other sequences, in parallel, or omitted. Equally, the order of the above process can be changed.

[0079] The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A computer system for use in performing a neural architecture search, the computer system comprising: A processor coupled to a storage medium storing instructions, the instructions, when executed by the processor, causing the processor to: receiving search space data indicating a search space of a neural network configured to perform a predetermined task; selecting a plurality of candidate neural networks within the search space, wherein each candidate neural network comprises a prediction layer; The performance of each candidate neural network in the plurality of candidate neural networks is evaluated based on the objective function in the following manner: Training each candidate neural network on the training data set to perform the predetermined task; as well as Based on the objective function, determining at least one ranking metric for each candidate neural network, wherein the at least one ranking metric comprises a weight-related metric determined based on weights of the prediction layer of the corresponding candidate neural network before and after the corresponding candidate neural network is trained; as well as The plurality of candidate neural networks are ranked based on the determined ranking metric.

2. The computer system of claim 1 , wherein the weight-related indicator is an angle indicator indicating an angle between one-dimensional vectors calculated from the weights of the prediction layer of the corresponding candidate neural network before and after the corresponding candidate neural network is trained.

3. The computer method of claim 1, wherein the weight-related indicator is a distance indicator selected from the group consisting of: Minkowski distance, cosine distance, Mahalanobis distance, and Hamming distance.

4. The computer system of claim 1, wherein: The at least one ranking metric further comprises a training loss metric, the training loss metric being determined based on the performance of the corresponding candidate neural network on the training dataset during training; Each candidate neural network is trained for a predetermined number of iterations; and The training loss indicator is determined based on the performance of the corresponding candidate neural network on the training data set in a last iteration of the predetermined number of iterations.

5. The computer system of claim 4, wherein the instructions, when executed by the processor, cause the processor to rank the plurality of candidate neural networks by: assigning a first ranking number to each candidate neural network based on the weight-related indicator of the candidate neural network; Based on the training loss indicator of the candidate neural network, assigning a second ranking number to each of the candidate neural networks; as well as The plurality of candidate neural networks are ranked based on the first ranking number and the second ranking number.

6. The computer system of claim 4, wherein the instructions, when executed by the processor, cause the processor to deploy the highest ranked candidate neural network to a computing device.

7. The computer system of claim 1, wherein the plurality of candidate neural networks comprises every neural network in the search space.

8. The computer system of claim 1 , wherein the at least one ranking metric further comprises a parameter metric indicating the number of parameters of the corresponding candidate neural network.

9. The computer system according to claim 1, wherein the training data set is generated by: receiving a target data set, the target data set comprising a plurality of input data elements classified into a plurality of categories; sampling a subset of the plurality of categories; as well as For each sampling category, a subset of the plurality of input data elements within the sampling category is sampled.

10. The computer system of claim 1, wherein the search space of the neural network comprises a plurality of convolutional neural networks, recurrent neural networks, transformers, sub-networks, and / or combinations thereof.

11. A method for evaluating the performance of a neural network architecture, the method comprising: receiving search space data indicating a search space of a neural network configured to perform a predetermined task; Sampling a plurality of candidate neural networks within the search space, wherein each candidate neural network comprises a prediction layer; The performance of each candidate neural network in the plurality of candidate neural networks is evaluated based on the objective function in the following manner: Training each candidate neural network on the training data set to perform the predetermined task; as well as Based on the objective function, determining at least one ranking metric for each candidate neural network, wherein the at least one ranking metric comprises a weight-related metric determined based on weights of the prediction layer of the corresponding candidate neural network before and after the corresponding candidate neural network is trained; as well as The plurality of candidate neural networks are ranked based on the determined ranking metric.

12. The method of claim 11, wherein the weight-related indicator is an angle indicator indicating an angle between one-dimensional vectors calculated from the weights of the prediction layer of the corresponding candidate neural network before and after the corresponding candidate neural network is trained.

13. The method of claim 11, wherein the weight-related indicator is a distance indicator selected from the group consisting of: Minkowski distance, cosine distance, Mahalanobis distance, and Hamming distance.

14. The method according to claim 11, wherein: The at least one ranking metric further comprises a training loss metric, the training loss metric being determined based on the performance of the corresponding candidate neural network on the training dataset during training; Each candidate neural network is trained for a predetermined number of iterations; and The training loss indicator is determined based on the performance of the corresponding candidate neural network on the training data set in a last iteration of the predetermined number of iterations.

15. The method of claim 14, wherein ranking the plurality of candidate neural networks comprises: assigning a first ranking number to each candidate neural network based on the weight-related indicator of the candidate neural network; Based on the training loss indicator of the candidate neural network, assigning a second ranking number to each candidate neural network; as well as The plurality of candidate neural networks are ranked based on the first ranking number and the second ranking number.

16. The method of claim 11, further comprising deploying the highest ranked candidate neural network to a computing device.

17. The method of claim 11, wherein the plurality of candidate neural networks comprises every neural network in the search space.

18. The method of claim 11, wherein the at least one ranking metric further comprises a parameter metric indicating the number of parameters of the corresponding candidate neural network.

19. The method according to claim 11, wherein the training data set is generated by: receiving a target data set, the target data set comprising a plurality of input data elements classified into a plurality of categories; sampling a subset of the plurality of categories; as well as For each sampling category, a subset of the plurality of input data elements within the sampling category is sampled.

20. The method of claim 11, wherein the search space of the neural network comprises a plurality of convolutional neural networks, recurrent neural networks, transformers, sub-networks, and / or combinations thereof.

Images