Selecting neural network architecture for supervised machine learning problems

Abstract

Systems and methods for selecting neural networks for machine learning problems are disclosed. A method includes accessing an input matrix. The method includes accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem. The method includes calculating, for each untrained candidate neural network, at least one expressivity metric that captures an expressivity of the candidate neural network with respect to a machine learning problem. The method includes calculating, for each untrained candidate neural network, at least one trainability metric that captures trainability of the candidate neural network with respect to a machine learning problem. The method includes selecting at least one candidate neural network for solving a machine learning problem based on at least one expressive metric and at least one trainability metric.

Description

Technical Field

[0001] The present disclosure relates to the field of machine learning. Background Art

[0002] A variety of different types of neural network architectures are known (e.g., convolutional neural networks, feed-forward neural networks, etc.). Selecting a neural network architecture (and sub-architectures within a given architecture type) for solving a given machine learning problem can be challenging. Summary of the Invention

[0003] The present disclosure generally relates to machines configured to select neural network architectures for solving machine learning problems, including computerized variants of such specialized machines and improvements to such variants, and to techniques that improve such specialized machines as compared to other specialized machines that provide neural network technology. In particular, the present disclosure is directed to systems and methods for selecting neural network architectures for solving machine learning problems within a given machine learning problem space.

[0004] According to some aspects of the techniques described herein, a system includes processing hardware and a memory. The memory stores instructions that, when executed by the processing hardware, cause the processing hardware to perform operations. The operations include accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem. The operations include, for each untrained candidate neural network, computing at least one expressivity measure that captures the expressivity of the candidate neural network with respect to the machine learning problem. The operations include, for each untrained candidate neural network, computing at least one trainability measure that captures the trainability of the candidate neural network with respect to the machine learning problem. The operations include selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressivity measure and the at least one trainability measure. The operations include providing an output representing the selected at least one candidate neural network.

[0005] In accordance with some aspects of the techniques described herein, a machine-readable medium stores instructions that, when executed by one or more machines, cause the one or more machines to perform operations. The operations include accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem. The operations include, for each untrained candidate neural network, computing at least one expressiveness metric that captures the expressiveness of the candidate neural network with respect to the machine learning problem. The operations include, for each untrained candidate neural network, computing at least one trainability metric that captures the trainability of the candidate neural network with respect to the machine learning problem. The operations include selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressiveness metric and the at least one trainability metric. The operations include providing an output representing the selected at least one candidate neural network.

[0006] In accordance with some aspects of the techniques described herein, a method includes accessing an input matrix. The method includes accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem. The method includes, for each untrained candidate neural network, computing at least one expressiveness metric that captures the expressiveness of the candidate neural network with respect to the machine learning problem. The method includes, for each untrained candidate neural network, computing at least one trainability metric that captures the trainability of the candidate neural network with respect to the machine learning problem. The method includes selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressiveness metric and the at least one trainability metric. The method includes providing an output representing the selected at least one candidate neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In the figures of the drawings, some embodiments of the present techniques are shown by way of example and not limitation.

[0008] Figure 1 An example system is shown in which a neural network architecture for solving a machine learning problem can be implemented in accordance with some embodiments.

[0009] Figure 2 A flowchart of an example method for selecting a neural network architecture for solving a machine learning problem is shown in accordance with some embodiments.

[0010] Figure 3 A flowchart of an example method for reducing an error rate is shown in accordance with some embodiments.

[0011] Figures 4A to 4B A flowchart of an example method for selecting a cell from a block is shown in accordance with some embodiments.

[0012] Figure 5 is a block diagram showing components of a machine capable of reading instructions from a machine-readable medium and performing any of the methods discussed herein. Detailed Description

[0013] Overview

[0014] The present disclosure particularly describes methods, systems, and computer program products that individually provide various functions. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of various aspects of the different embodiments of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without all of the specific details.

[0015] As mentioned above, there are known many different types of neural network architectures (e.g., feedforward neural networks, convolutional neural networks, recurrent neural networks, etc.). Selecting a high-performance neural network architecture to solve a given machine learning problem (e.g., a regression problem, a ranking problem, or a classification problem, such as classifying data in a given space, such as classifying images of birds according to the type of bird in the image) can be challenging.

[0016] Some aspects of the techniques described herein are directed to solving the following technical problem: selecting a neural network architecture from a set of neural network architectures for solving a given machine learning problem before the neural network architecture is trained. Advantageously, as a result of certain aspects, a high-performance neural network architecture is trained to solve the given machine learning problem, and less desirable architectures are not trained. This can save computational time and improve efficiency without resulting in the use of a non-high-performance neural network.

[0017] In certain cases, the solution to the problem is implemented at a server. The server accesses, via a data repository, a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem. The server computes, for each untrained candidate neural network, at least one expressiveness metric that captures the expressiveness of the candidate neural network with respect to the machine learning problem. The server computes, for each untrained candidate neural network, at least one trainability metric that captures the trainability of the candidate neural network with respect to the machine learning problem. The server selects, based on the at least one expressiveness metric, the at least one trainability metric, and the architecture of the candidate neural network, at least one candidate neural network for solving the machine learning problem. The server provides an output representing the selected at least one candidate neural network.

[0018] In some cases, at least one selected candidate neural network is partially or fully trained to solve a machine learning problem. As used herein, a neural network that is "partially or fully trained" can include being trained for several epochs or being trained until a certain convergence indicator has been met. The at least one trained candidate neural network operates on a machine learning problem space to solve the machine learning problem. The server then provides a solution to the machine learning problem generated by the at least one trained candidate neural network.

[0019] According to some examples, at least one expressivity metric represents a measure (e.g., magnitude or angle) of the separation of samples from a machine learning problem space by an untrained candidate neural network. According to some examples, given a sample from a machine learning problem space, at least one trainability metric represents a function of the gradients at the last and first layers. According to some examples, the expressivity and trainability metrics can include quantities, measures, or statistics of a neural network that capture different properties of the architecture, such as expressivity and trainability.

[0020] Figure 1 An example system 100 is shown in which a neural network architecture selected for solving a machine learning problem can be implemented. As shown, system 100 includes a server 110, a data repository 120, and a client device 130 that are connected to each other via a network 140. Network 140 includes one or more of the Internet, an intranet, a local area network, a wide area network, a wired network, a wireless network, a cellular network, a WiFi network, etc.

[0021] Client device 130 can be a laptop computer, a desktop computer, a mobile phone, a tablet computer, a smart TV having a processor and a memory, a smart watch, etc. Client device 130 can be used to display output to a user or receive input from a user.

[0022] Data repository 120 can be implemented as a database or any other data storage structure. As shown, data repository stores a machine learning problem space 122. Machine learning problem space 122 includes data to be classified by a neural network. For example, machine learning problem space 122 can include photos of birds to be classified according to the type of bird or email messages to be classified as "important email", "unimportant email", or "junk email".

[0023] Server 110 may include one or more servers. Server 110 may be implemented as a server farm including multiple servers. As shown, server 110 stores untrained candidate neural networks 112.1 - n (where n is a positive integer greater than or equal to 2), a selection module 114, and a training module 116. The untrained candidate neural networks 112.1 - n are neural networks that can be used for various classification tasks. For example, the untrained candidate neural networks 112.1 - n may include untrained versions of convolutional neural networks or feed - forward neural networks.

[0024] The selection module 114 selects at least one of the untrained candidate neural networks 112.1 - n for training to solve a machine - learning problem associated with the machine - learning problem space 122. More details of example operations of the selection module are provided in conjunction with Figure 2 The training module 116 trains the (multiple) neural networks selected (by the selection module 114) (from the untrained candidate neural networks 112.1 - n) to solve the machine - learning problem. After training, the trained neural network can be used to solve the machine - learning problem by classifying data in the machine - learning problem space 122 (or another problem space).

[0025] Figure 2 A flowchart of an example method 200 for selecting a neural network architecture for solving a machine - learning problem according to some embodiments is shown. As described below, method 200 is implemented using the selection module 114 of Figure 1 server 110. However, method 200 is not limited by the architecture of system 100 and can be implemented in other architectures or other systems.

[0026] In operation 210, the selection module 114 accesses (e.g., via network 140) the machine - learning problem space 122 associated with the machine - learning problem to be solved. The selection module 114 accesses multiple untrained candidate neural networks 112.1 - n for solving the machine - learning problem.

[0027] In operation 220, the selection module 114 calculates (multiple) expressivity measures related to the expressivity of each untrained candidate neural network 112.1 - n with respect to the machine - learning problem. The (multiple) expressivity measures represent a measure of the separation of samples from the machine - learning problem space 122 by the untrained candidate neural network. The measure of separation can be size or angle.

[0028] In operation 230, the selection module 114 calculates (multiple) trainability measures related to the trainability of each untrained candidate neural network 112.1 - n with respect to the machine - learning problem.

[0029] At operation 240, selection module 114 selects (a) candidate neural network(s) for solving a machine learning problem based on (a) measure(s) of expressiveness and (a) measure(s) of trainability. The selection is also based on the (a) architecture(s) of the (a) candidate neural network(s). The (a) candidate neural network(s) are selected from a plurality of untrained candidate neural networks 112.1 - n. In some cases, selection module 114 selects (a) candidate neural network(s) for which the (a) measure(s) of expressiveness exceed a threshold and the (a) measure(s) of trainability are within a range. The range is defined by a range minimum and a range maximum.

[0030] At operation 250, selection module 114 provides an output representing the selected (a) candidate neural network(s). In some cases, training module 116 trains the selected (a) candidate neural network(s) to solve the machine learning problem. Training module 116 runs the trained (a) candidate neural network(s) on machine learning problem space 122 to solve the machine learning problem. Server 110 provides (e.g., provides to client device 130 for display thereon, or provides to data repository 120 for storage thereon) a solution to the machine learning problem generated by the trained (a) candidate neural network(s).

[0031] One goal of the techniques described herein is to automatically select and configure a neural network architecture for a given task. More specifically, given a data set, certain aspects automatically identify layer types (e.g., convolutional, max pooling, fully connected, etc.), their hyperparameters (e.g., stride size, convolution size), their connections to all other layers, and the total number of layers. Some aspects also identify the training algorithm to use (e.g., stochastic gradient descent, RMSProp, Adam, etc.) and how to initialize weights (e.g., Glorot, Normal, Laplace, Uniform, etc.).

[0032] Some approaches can be grouped based on: (1) how they define the search space architecturally (e.g., unrestricted or restricted); (2) how they explore the space (e.g., reinforcement learning, Monte Carlo tree search); (3) what prediction model they use to guide the search (e.g., sequential model - based optimization, recurrent neural network (RNN), genetic algorithm); and (4) whether they use a cheap surrogate function to more effectively guide the search. In the case of genetic algorithms and reinforcement learning, points 2 and 3 above are combined into one, as they jointly learn a prediction model and explore the space.

[0033] In some embodiments, the search space is explored by means of sequential model-based optimization (SMBO). An example model in SMBO can be a Bayesian recurrent neural network. A set of characteristics of the stochastic neural network is defined to act as a cheap surrogate (or statistic) of the true performance of the neural network. In some cases, other models that output both predictions and uncertainties about the predictions (e.g., Gaussian processes) are used. In some cases, the cheap surrogate (or statistic) is not given as an input to the model but is considered as an output. The cheap surrogate can include the trainability and expressivity measures discussed above.

[0034] In some extended embodiments, the search space is unrestricted. Reinforcement learning or Monte Carlo tree search is used to explore the search space. In some cases, the representation of the architecture itself can also be additionally used to assist in making predictions.

[0035] Some aspects relate to two main measures for predicting the performance of an initial model on a dataset after training before training. Both of these measures are statistics of the model collected on random batches from the dataset. In some examples, a batch consists of n points Some aspects ignore the label y i . The model typically includes two components: a deep neural network that embeds the input space into a latent space, and a fully connected linear layer followed by a softmax that transforms the embedding into a probability distribution over a set of possible labels. Let f be the former, i.e., the embedding, and assume it has L layers.

[0036] The measure of expressivity is defined according to Expression 1, which approximates Expression 2.

[0037] In Expression 1, {x1,..., x n} is a batch of inputs.

[0038]

[0039] Expression 1

[0040]

[0041] Expression 2

[0042] In Expression 2, E represents expectation and P is the data distribution of multiple x's. Intuitively, this measure represents the tendency of f to expand the input space and pull points apart. A larger measure of expressivity should be related to better performance after training. Variations of this measure include sampling (x, x') pairs rather than choosing every pair from the same batch, and choosing other powers than 2 (i.e., for some p > 0, ), and other methods for testing the "scalability" (tendency to expand the input space) of neural networks.

[0043] In addition to measuring expressiveness characteristics, gradient distortion also includes sampling the stochastic gradient vector with respect to the last layer, and performing backpropagation after a forward pass for each sample x i to obtain the gradient vector for each previous layer. Assuming such a fixed input x i and a fixed last-layer gradient vector, the gradient vector of the parameter w at layer l is denoted as Then, the gradient distortion of the parameter w is defined in Expression 3.

[0044]

[0045] Expression 3

[0046] In other words, gradient distortion is a measure of how much gradient explosion or vanishing occurs on typical data points. The larger this is, the worse the performance after training is expected to be, because training (via stochastic gradient descent (SGD)) is expected to be difficult. Variations of this measure include sampling a new last-layer gradient vector for each x i replacing the summation with Expression 4, and other methods for measuring gradient explosion / vanishing. The above measure predicts the remaining network performance for a fixed architecture and random initialization.

[0047]

[0048] Expression 4

[0049] Given the same settings as in the case of measuring expressiveness, angular expressiveness is defined in Expression 5.

[0050]

[0051] Expression 5

[0052] Angular expressiveness measures how much f "decorrelates" input vectors in the sense of pulling apart their angles. Thus, large angular expressiveness can be expected to be associated with better performance. In some scenarios, the predicted quantity is actually the deviation from its asymptotic limit when the depth of f goes to infinity. The representation of this asymptotic deviation is the Cauchy error C l -C (l+1) , where C l is the angular expressiveness of the network up to layer l.

[0053] In some cases, statistical data can be learned automatically. Although the data-based statistical data described in this article are highly correlated with the final performance of the model, they may not be the most predictive statistical data that one can extract to predict the generalization ability of a given model. The most predictive statistical data may be a complex non-linear function of the raw statistical data (i.e., the embedding and gradient metrics used to compute the statistical data in the previous section).

[0054] However, like any other function approximation problem in machine learning, if there is sufficient training data, these complex functions can be learned. This will motivate another version of the framework, where server 110 also learns predictive statistical data from the raw data. In particular, some aspects can use a neural network, whose inputs are the data representations of the last layer (i.e., f(x) above), the gradients of the last layer and the first layer, etc.

[0055] In a general version of some aspects, the process of Algorithm 1 is repeated until a predetermined desired error rate is reached or a predetermined total computational cost is reached.

[0056] Algorithm 1

[0057] 1. Propose multiple deep neural network (DNN) architectures. The DNN architectures can be sampled from a space of predefined architectures or can be constructed using general building blocks.

[0058] 2. Initialize the weights of the architecture using Glorot Normal initialization. In alternative embodiments, independent normal distributions and Laplace distributions can be used.

[0059] 3. Compute the measure of expressiveness and gradient distortion for each in the architecture.

[0060] 4. Include the measure of expressiveness and gradient distortion of each architecture as inputs to a Bayesian recurrent neural network (Bayesian RNN) to predict architecture performance. Gaussian processes can also be used for this task. In principle, any probabilistic machine learning algorithm that outputs a distribution over its predictions can be used.

[0061] 5. Use the posterior mean and variance computed for each in the architecture using the Bayesian RNN as inputs to an acquisition function, such as expected improvement, upper confidence bound, or Thompson sampling.

[0062] 6. Sample the (multiple) architectures with (multiple) values for the acquisition function that exceed a threshold.

[0063] 7. Train the (multiple) architectures with (multiple) values for the acquisition function that exceed a threshold, and evaluate their performance on a held-out set. Update the model used in operation 4 with the observed performance. Return to operation 1.

[0064] Figure 3 Algorithm 1 is summarized in. Figure 3 FIG. 300 is a flow chart of an example method for reducing error rate according to some embodiments.

[0065] In operation 310, a server (e.g., server 110) proposes multiple DNN architectures. The DNN architectures can be sampled from a space of predefined architectures or can be constructed using general building blocks.

[0066] In operation 320, the server initializes the weights of the architectures using Glorot Normal initialization. In alternative embodiments, independent normal and Laplace distributions can be used.

[0067] In operation 330, the server computes the measure of expressiveness and gradient deformation for each architecture in the architectures.

[0068] In operation 340, the server includes the measure of expressiveness and gradient deformation of each architecture as inputs to a Bayesian RNN to predict architecture performance. A Gaussian process can also be used for this task. In principle, any probabilistic machine learning algorithm with an output that is a distribution over its predictions can be used.

[0069] In operation 350, the server uses the posterior mean and variance computed for each architecture using the Bayesian RNN as inputs to an acquisition function, such as expected improvement, upper confidence bound, or Thompson sampling.

[0070] In operation 360, the server samples the (multiple) architectures with (multiple) values for the acquisition function that exceed a threshold.

[0071] In operation 370, the server trains the (multiple) architectures with (multiple) values for the acquisition function that exceed a threshold, and evaluates their performance on a held-out set. Update the Bayesian RNN model of operation 340 based on the observed performance. After operation 370, if the value for the acquisition function (of operation 360) is sufficient, method 300 ends. Otherwise, method 300 returns to operation 310.

[0072] Specific implementations are described herein. Given a fixed deep neural network architecture where a recursive basic unit (cell) is repeated multiple times, some aspects relate to reasoning about the cell with the highest accuracy. First, a cell is defined as a directed acyclic graph of a certain number of blocks. Each block takes two inputs (I1, I2), performs operations (O1, O2) on each input, and then returns the sum of the outputs from these two operations. The set of possible inputs for a block is the outputs of all previous blocks in a cell and the outputs of the previous two cells. The operations on these inputs are standard neural network layers such as convolution, max pooling, etc. It is very important to automatically and accurately search for such basic units in a deep neural network because their performance typically generalizes when applied to larger datasets. To find the best cell with B blocks, Algorithm 2 is used.

[0073] In some cases, the search in Algorithm 2 may be exhaustive because the number of cells in a block (C) may be small. For example, for 8 possible operations, the search space is 8 * 8 * 2 * 2 = 256, which can be exhaustively studied, for example, by using 256 graphics processing units (GPUs) in parallel.

[0074] Algorithm 2

[0075] 1. Consider a set of cells in a block (C).

[0076] 2. For each cell in C:

[0077] · Initialize the weights of the architecture using Glorot Normal initialization. In alternative embodiments, an iid normal distribution and a Laplace distribution can be used.

[0078] · Compute the measure of expressiveness and gradient distortion. This can be done over multiple initializations,

[0079] and the mean and variance can be reported.

[0080] · Train the neural network for a fixed number of epochs.

[0081] · Compute the accuracy on the test set.

[0082] 3. If the number of blocks in the cell in C is B, then stop and return the cell with the highest test accuracy.

[0083] 4. Train a Bayesian Recurrent Neural Network (BRNN) that takes the cell architecture and random statistics as input and predicts the accuracy of the test set.

[0084] 5. Consider the set D, including the cells obtained by expanding each cell in C and adding new blocks.

[0085] 6. Compute the measure expressiveness and gradient deformation for the cells in D. If this computation is too expensive (e.g., due to the large number of cells in D), then one can (i) train an additional BRNN that only takes the cell structure and predict the test set accuracy, and (ii) use it to filter the set D based on the predicted test accuracy.

[0086] 7. Use the BRNN in operation 4 to determine the set of cells to consider in the next iteration This decision is made by trading off exploitation and exploration within a Bayesian optimization framework.

[0087] 8. ε → C and then go to step 2.

[0088] In Figures 4A-4B Algorithm 2 is summarized. Figures 4A-4B FIG. 4 shows a flowchart of an example method 400 for selecting cells from a block according to some embodiments.

[0089] As Figure 4A shown, operation 410 includes sub-operations 412-418. In operation 410, for each cell in block C, the server (e.g., server 110) performs sub-operations 412-418.

[0090] In sub-operation 412, the server initializes the weights of the architecture using Glorot Normal initialization. In alternative embodiments, the same-distribution normal distribution and Laplace distribution can be used.

[0091] In sub-operation 414, the server computes the measure expressiveness and gradient deformation. This can be done over multiple initializations and the mean and variance can be reported.

[0092] In sub-operation 416, the server trains the neural network for a fixed number of epochs.

[0093] In sub-operation 418, the server computes the accuracy on the test set. After operation 410 is completed for each cell in C, method 400 proceeds to operation 420.

[0094] In operation 420, the server determines whether the number of blocks in C is within a predetermined range. If so, method 400 proceeds to operation 430. If not, method 400 proceeds to operation 440.

[0095] In operation 430, if the number of blocks in C is within a predetermined range, the server returns the cell in C with the highest test accuracy. After operation 430, method 400 ends.

[0096] In operation 440, if the number of blocks of C is not within a predetermined range, the server trains a BRNN that takes the unit architecture and random statistical data as input and predicts the test set accuracy. After operation 440, method 400 continues to Figure 4B operation 450 shown.

[0097] As Figure 4B shown, in operation 450, the server generates a set D, where set D includes units obtained by expanding each unit in C and adding new blocks.

[0098] In operation 460, the server calculates the measure expressiveness and gradient deformation for the units in D. If this calculation is too expensive (e.g., due to a large number of units in D), the server can (i) train an additional BRNN that only takes the unit structure and predicts the test set accuracy, and (ii) use the additional BRNN to filter set D based on the predicted test accuracy.

[0099] In operation 470, the server uses the BRNN trained in operation 440 to determine the set of units to be considered in the next iteration This decision is made by weighing exploitation and exploration within a Bayesian optimization framework.

[0100] In operation 480, the server stores ε as C. After operation 480, method 400 returns to Figure 4A operation 420.

[0101] Numbered Examples

[0102] Certain embodiments are described herein as numbered examples 1, 2, 3, etc. These numbered examples are provided only as examples and do not limit the claimed subject matter.

[0103] Example 1 is a system that includes: processing hardware; and a memory storing instructions that cause the processing hardware to perform operations including: accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem; for each untrained candidate neural network, calculating at least one expressiveness metric that captures the expressiveness of the candidate neural network with respect to the machine learning problem; for each untrained candidate neural network, calculating at least one trainability metric that captures the trainability of the candidate neural network with respect to the machine learning problem; selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressiveness metric and the at least one trainability metric; and providing an output representing the selected at least one candidate neural network.

[0104] In Example 2, the subject matter according to Example 1 includes: wherein the at least one expressivity metric represents a measure of the separation by the untrained candidate neural network of samples from the machine learning problem space.

[0105] In Example 3, the subject matter according to Example 2 includes, wherein the measure of the separation is magnitude.

[0106] In Example 4, the subject matter according to Examples 2-3 includes, wherein the measure of the separation is angle.

[0107] In Example 5, the subject matter according to Examples 1-4 includes, wherein the at least one trainability metric represents the stochastic gradient descent of the weights in the candidate neural network during a first stage of training.

[0108] In Example 6, the subject matter according to Examples 1-5 includes, wherein selecting the at least one candidate neural network for solving the machine learning problem includes: selecting the at least one candidate neural network having the at least one expressivity metric exceeding a threshold and the at least one trainability metric within a range, wherein the range is defined by a range minimum and a range maximum.

[0109] In Example 7, the subject matter according to Examples 1-6 includes, the operation further includes: training the at least one candidate neural network to solve the machine learning problem.

[0110] In Example 8, the subject matter according to Example 7 includes, the operation further includes: running the at least one trained candidate neural network on the machine learning problem space to solve the machine learning problem; and providing a solution to the machine learning problem generated by the at least one trained candidate neural network.

[0111] Example 9 is a non-transitory machine-readable medium storing instructions that cause one or more machines to perform operations, the operations including: accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem; for each untrained candidate neural network, calculating at least one expressivity metric capturing the expressivity of the candidate neural network with respect to the machine learning problem; for each untrained candidate neural network, calculating at least one trainability metric capturing the trainability of the candidate neural network with respect to the machine learning problem; selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressivity metric and the at least one trainability metric; and providing an output representing the at least one selected candidate neural network.

[0112] In Example 10, the subject matter according to Example 9 includes: wherein the at least one expressiveness metric represents a measure of the separation of samples from the machine learning problem space by the untrained candidate neural network.

[0113] In Example 11, the subject matter according to Example 10 includes, wherein the measure of the separation is magnitude.

[0114] In Example 12, the subject matter according to Examples 10-11 includes, wherein the measure of the separation is an angle.

[0115] In Example 13, the subject matter according to Examples 9-12 includes, wherein the at least one trainability metric represents stochastic gradient descent of the weights in the candidate neural network during a first stage of training.

[0116] In Example 14, the subject matter according to Examples 9-13 includes, wherein selecting the at least one candidate neural network for solving the machine learning problem includes: selecting the at least one candidate neural network having the at least one expressiveness metric that exceeds a threshold and the at least one trainability metric within a range, wherein the range is defined by a range minimum and a range maximum.

[0117] Example 15 is a method that includes: accessing a machine learning problem space related to quantities of a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem; for each untrained candidate neural network, calculating at least one expressiveness metric that captures the expressiveness of the candidate neural network with respect to the machine learning problem; for each untrained candidate neural network, calculating at least one trainability metric that captures the trainability of the candidate neural network with respect to the machine learning problem; selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressiveness metric and the at least one trainability metric; and providing an output representing the selected at least one candidate neural network.

[0118] In Example 16, the subject matter according to Example 15 includes: wherein the at least one expressiveness metric represents a measure of the separation of samples from the machine learning problem space by the untrained candidate neural network.

[0119] In Example 17, the subject matter according to Example 16 includes, wherein the measure of the separation is magnitude.

[0120] In Example 18, the subject matter according to Examples 16-17 includes, wherein the measure of the separation is an angle.

[0121] In Example 19, the subject matter according to Examples 15 - 18 includes, wherein the at least one trainability metric represents a stochastic gradient descent of weights in the candidate neural network during a first stage of training.

[0122] In Example 20, the subject matter according to Examples 15 - 19 includes, wherein selecting the at least one candidate neural network for solving the machine learning problem includes: selecting the at least one candidate neural network having the at least one expressivity metric exceeding a threshold and the at least one trainability metric within a range, wherein the range is defined by a range minimum and a range maximum.

[0123] Example 21 is at least one machine - readable medium including instructions that, when executed by a processing circuitry, cause the processing circuitry to perform operations for implementing any one of Examples 1 - 20.

[0124] Example 22 is an apparatus including modules for implementing any one of Examples 1 - 20.

[0125] Example 23 is a system for implementing any one of Examples 1 - 20.

[0126] Example 24 is a method for implementing any one of Examples 1 - 20.

[0127] Components and Logic

[0128] Certain embodiments are described herein as including logic or multiple components or mechanisms. A component can be a software component (e.g., code embodied on a machine - readable medium) or a hardware component. A “hardware component” is a tangible unit capable of performing certain operations and can be configured or arranged in a certain physical manner. In various example embodiments, one or more computer systems (e.g., a stand - alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) can be configured by software (e.g., an application or a part of an application) to operate as a hardware component for performing certain operations described herein.

[0129] In some embodiments, a hardware component may be implemented mechanically, electronically, or any suitable combination thereof. For example, a hardware component may include dedicated circuitry or logic that is permanently configured to perform certain operations. For example, a hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). A hardware component may also include programmable logic or circuitry that is temporarily configured by software to perform certain operations. For example, a hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component becomes a particular machine (or a particular component of a machine) customized to perform the configured functions and is no longer a general-purpose processor. It will be appreciated that decisions may be made, taking into account cost and time, as to whether to implement a hardware component mechanically in dedicated and permanently configured circuitry or in temporarily configured circuitry (e.g., configured by software).

[0130] Accordingly, the phrase “hardware component” should be understood to include a tangible record that is physically constructed, permanently configured (e.g., hard-wired), or temporarily configured (e.g., programmed) to operate in a particular manner or to perform certain operations described herein. As used herein, “hardware-implemented component” refers to a hardware component. Considering embodiments in which a hardware component is temporarily configured (e.g., programmed), each of the hardware components may not be configured or instantiated at any instance in time. For example, in the case where a hardware component includes a general-purpose processor that is configured by software to become a dedicated processor, the general-purpose processor may be configured to different dedicated processors (e.g., including different hardware components) at different times respectively. Thus, software configures one or more particular processors, e.g., to constitute a particular hardware component at one instance in time and a different hardware component at a different instance in time.

[0131] A hardware component may provide information to and receive information from other hardware components. Accordingly, the described hardware components may be considered to be communicatively coupled. In cases where multiple hardware components are present, communication may be achieved through signal transmission (e.g., via appropriate circuitry and buses) between or among two or more of the hardware components. In embodiments in which multiple hardware components are configured or instantiated at different times, communication between these hardware components may be achieved, for example, by storing and retrieving information in a memory structure accessible to the multiple hardware components. For example, one hardware component may perform an operation and store the output of the operation in a memory device to which it is communicatively coupled. Then, another hardware component may access the memory device at a later time to retrieve and process the stored output. A hardware component may also initiate communication with an input or output device and may operate on resources (e.g., a pool of information).

[0132] The various operations of the example methods described herein can be performed, at least in part, by one or more processors temporarily configured (e.g., by software configuration) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may comprise processor-implemented components that operate to perform one or more of the operations or functions described herein. As used herein, "processor-implemented components" refers to hardware components implemented using one or more processors.

[0133] Similarly, the methods described herein can be implemented, at least in part, by a processor, where one or more particular processors are examples of hardware. For instance, at least some operations of a method can be performed by one or more processors or processor-implemented components. Additionally, one or more processors are also operable in a "cloud computing" environment or as a "software as a service" (SaaS) to support the execution of relevant operations. For example, at least some of the operations can be performed by a group of computers (as an example of machines including processors), which can be accessed via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs).

[0134] The execution of certain operations can be distributed among processors, residing not only within a single machine but also deployed across multiple machines. In some example embodiments, the processors or processor-implemented components can be located in a single geographical location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processors or processor-implemented components can be distributed across multiple geographical locations.

[0135] Example Machine and Software Architectures

[0136] In some embodiments, the components, methods, applications, etc. described in conjunction with Figure 1 to 4 are implemented in the context of a machine and an associated software architecture. The following sections describe representative (multiple) software architectures and (multiple) machine (e.g., hardware) architectures suitable for use with the disclosed embodiments.

[0137] The software architecture is used in conjunction with the hardware architecture to create devices and machines customized for a particular purpose. For example, a particular hardware architecture combined with a particular software architecture will create a mobile device, such as a mobile phone, a tablet device, etc. Slightly different hardware and software architectures can result in smart devices for the "Internet of Things", while another combination results in a server computer for use within a cloud computing architecture. Since those skilled in the art can readily understand how to implement the disclosed subject matter in contexts different from those included herein, not all combinations of such software and hardware architectures are presented herein.

[0138] Figure 5FIG. 0 is a block diagram showing components of a machine 500 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein. Specifically, Figure 5 FIG. 1 schematically shows an example form of a computer system, within which instructions 516 (e.g., software, program, application, applet, app, or other executable code) can be executed to cause the machine 500 to perform any one or more of the methods discussed herein. The instructions 516 transform a general, unprogrammed machine into a particular machine programmed to perform the described and illustrated functions in the described manner. In an alternative embodiment, the machine 500 operates as a stand-alone device or can be coupled (e.g., networked) to other machines. In a network deployment, the machine 500 can operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 500 can include, but is not limited to, a server computer, a client computer, a PC, a tablet computer, a laptop computer, a netbook, a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smartphone, a mobile device, a wearable device (e.g., a smartwatch), a smart home device (e.g., a smart device), other smart devices, a web appliance, a network router, a network switch, a bridge, or any machine capable of sequentially or otherwise executing the instructions 516 that specify actions to be taken by the machine 500. Further, although only a single machine 500 is shown, the term "machine" shall also be taken to include a collection of machines 500 that individually or jointly execute the instructions 516 to perform any one or more of the methods discussed herein.

[0139] The machine 500 can include a processor 510, a memory 530, and I / O components 550 that can be configured to communicate with each other, for example, via a bus 502. In an example embodiment, the processor 510 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) can include, for example, a processor 512 and a processor 514 that can execute the instructions 516. The term "processor" is intended to include multi-core processors that can include two or more independent processors (sometimes referred to as "cores") that can execute instructions simultaneously. Although Figure 5Multiple processors 510 are shown, but machine 500 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0140] Memory / storage 530 may include memory 532 such as main memory or other memory storage devices, and storage unit 536, both of which may be accessed by processor 510 via bus 502. Storage unit 536 and memory 532 store instructions 516 embodying any one or more of the methods or functions described herein. During execution of instructions 516 by machine 500, instructions 516 may also reside, in whole or in part, within memory 532, within storage unit 536, within at least one of processors 510 (e.g., within a cache memory of the processor), or any suitable combination thereof. Accordingly, memory 532, storage unit 536, and the memory of processors 510 are examples of machine-readable media.

[0141] As used herein, "machine-readable medium" refers to a device that can store instructions (e.g., instructions 516) and data temporarily or permanently, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database or associated cache and servers) that can store instructions 516. The term "machine-readable medium" should also be understood to include any medium or combination of media that can store instructions (e.g., instructions 516) for execution by a machine (e.g., machine 500) such that the instructions, when executed by one or more processors of the machine (e.g., processor 510), cause the machine to perform any one or more of the methods described herein. Thus, "machine-readable medium" refers to a single storage device or apparatus, as well as a "cloud-based" storage system or storage network that includes multiple storage devices or apparatuses. The term "machine-readable medium" does not include a signal per se.

[0142] The I / O component 1050 can include a variety of components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, and so on. The specific I / O components 550 included in a particular machine will depend on the type of the machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that the I / O components 550 can include Figure 5 many other components not shown in Figure 5 . For simplicity of the following discussion only, the I / O components 550 are grouped only by function, and this grouping is in no way restrictive. In various example embodiments, the I / O components 550 can include an output component 552 and an input component 554. The output component 552 can include visual components (e.g., a display, such as a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)), acoustic components (e.g., a speaker), tactile components (e.g., a vibration motor, a resistance mechanism), other signal generators, etc. The input component 554 can include an alphanumeric input component (e.g., a keyboard, a touch screen configured to receive alphanumeric input, an optical photo keyboard, or other alphanumeric input components), a pointing-based input component (e.g., a mouse, a touchpad, a trackball, a joystick, a motion sensor, or other pointing tools), a tactile input component (e.g., a physical button, a touch screen providing the location and / or force of a touch or touch gesture, or other tactile input components), an audio input component (e.g., a microphone), etc.

[0143] In other example embodiments, the I / O component 550 may include a biometric component 556, a motion component 558, an environmental component 560, or a location component 562 among numerous other components. For example, the biometric component 556 may include components for operations such as detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye-gaze tracking), measuring biometric signals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), measuring exercise-related metrics (e.g., distance traveled, speed of movement, or time spent exercising), identifying a person (e.g., voice recognition, retina recognition, facial recognition, fingerprint recognition, or electroencephalogram-based recognition), and so on. The motion component 558 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), and the like. The environmental component 560 may include, for example, a lighting sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers for detecting ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones for detecting background noise), a proximity sensor component (e.g., an infrared sensor for detecting nearby objects), a gas sensor (e.g., a gas detection sensor for detecting the concentration of harmful gases to ensure safety or for measuring pollutants in the atmosphere), or other components that can provide an indication, measurement, or signal corresponding to the surrounding physical environment. The location component 562 may include a positioning sensor component (e.g., a Global Positioning System (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer for detecting the air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), and the like.

[0144] Communication can be implemented using a variety of techniques. The I / O component 550 may include a communication component 564 operable to couple the machine 500 to the network 580 or the device 570 via the coupling 582 and the coupling 572, respectively. For example, the communication component 564 may include a network interface component for docking with the network 580 or another suitable device. In other examples, the communication component 564 may include a wired communication component, a wireless communication component, a cellular communication component, a Near Field Communication (NFC) component, components (e.g., Low Energy), components, and other communication components for providing communication in other ways. The device 570 may be another machine or any one of a variety of peripheral devices (e.g., a peripheral device coupled via USB).

[0145] In addition, communication component 564 can detect an identifier, or include components operable to detect an identifier. For example, communication component 564 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component, or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be derived via communication component 564, such as, for example, location via Internet Protocol (IP) geolocation, location via signal triangulation, location via detection of an NFC beacon signal that can indicate a specific location, and so on.

[0146] In various example embodiments, one or more portions of network 580 can be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a WAN, a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a network, another type of network, or a combination of two or more such networks. For example, network 580 or a portion of network 580 can include a wireless or cellular network, and coupling 582 can be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, coupling 582 can implement any one of a variety of types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolved data optimized (EVDO) technology, general packet radio service (GPRS) technology, GSM enhanced data rates for GSM evolution (EDGE) technology, 3rd Generation Partnership Project (3GPP) including 5G, 4th generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), worldwide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other standards defined by various standards setting organizations, other remote protocols, or other data transmission technologies.

[0147] Instructions 516 can be transmitted or received via a network interface device (e.g., the network interface component included in communication component 564) over network 580 using a transmission medium and any one of a variety of well-known transmission protocols (e.g., HTTP). Similarly, instructions 516 can be transmitted or received to device 570 via coupling 572 (e.g., a peer-to-peer coupling) using a transmission medium. The term "transmission medium" shall be considered to include any non-transitory medium that is capable of storing, encoding, or carrying instructions 516 for execution by machine 500, and includes digital or analog communication signals or other non-transitory media for facilitating the communication of such software.

Claims

1. A computer-implemented method, comprising: Identifying a plurality of neural network (NN) architectures for solving a machine learning problem; For each of the plurality of NN architectures: Initializing weights for the NN architecture; Calculating a measure of expressiveness and gradient distortion for the NN architecture using the initialized weights; Providing the measure of expressiveness and the gradient distortion as inputs to a probabilistic machine learning (ML) algorithm to predict the performance of the NN architecture, the ML algorithm outputting a predictive distribution; And Providing the output of the predictive distribution as an input to an acquisition function; Selecting an NN architecture for which the value of the acquisition function exceeds a predetermined threshold as a selected NN architecture; Training the selected NN architecture; And Using the trained selected NN architecture to solve the machine learning problem.

2. The method according to claim 1, wherein the predictive distribution further comprises a posterior mean and variance.

3. The method according to claim 1, wherein initializing weights for the NN architecture further comprises: Initializing the weights using Glorot Normal initialization, independent normal distribution, or Laplace distribution.

4. The method according to claim 1, wherein the acquisition function is one of expected improvement, upper confidence bound, or Thompson sampling.

5. The method according to claim 1, wherein the NN architecture is a fixed-depth neural network architecture, wherein the units are recurrent basic units repeated multiple times, and wherein the selection of the NN architecture is based on reasoning about the unit with the highest accuracy.

6. The method according to claim 5, wherein the unit is defined as a directed acyclic graph having one or more blocks, wherein each block receives two inputs, performs corresponding operations on each of the inputs, and returns the sum of the outputs from the two operations.

7. The method according to claim 6, wherein the possible inputs for a block are the outputs of previous blocks within the unit and the outputs of the previous two units.

8. A system, comprising: A memory storing instructions; And One or more computer processors, wherein the instructions, when executed by the one or more computer processors, cause the system to perform operations, the operations comprising: Identifying a plurality of neural network (NN) architectures for solving a machine learning problem; For each of the plurality of NN architectures: Initializing weights for the NN architecture; Calculating a measure of expressiveness and gradient distortion for the NN architecture using the initialized weights; Providing the measure of expressiveness and the gradient distortion as inputs to a probabilistic machine learning (ML) algorithm to predict the performance of the NN architecture, the ML algorithm outputting a predictive distribution; and Providing the output of the predictive distribution as an input to an acquisition function; Selecting an NN architecture for which the value of the acquisition function exceeds a predetermined threshold as a selected NN architecture; Training the selected NN architecture; and Using the trained selected NN architecture to solve the machine learning problem.

9. The system according to claim 8, wherein the predictive distribution further comprises a posterior mean and variance.

10. The system according to claim 8, wherein initializing the weights for the NN architecture further comprises: Initializing the weights using Glorot Normal initialization, independent normal distribution, or Laplace distribution.

11. The system according to claim 8, wherein the acquisition function is one of expected improvement, upper confidence bound, or Thompson sampling.

12. The system according to claim 8, wherein the NN architecture is a fixed-depth neural network architecture, wherein the unit is a recurrent basic unit repeated multiple times, and wherein the NN architecture is selected based on reasoning the unit with the highest accuracy.

13. The system according to claim 12, wherein the unit is defined as a directed acyclic graph having one or more blocks, wherein each block receives two inputs, performs corresponding operations on each of the inputs, and returns the sum of the outputs from the two operations.

14. The system according to claim 13, wherein the possible inputs for a block are the outputs of the previous block within the unit and the outputs of the previous two units.

15. A non-transitory machine-readable medium storing instructions that, when executed by a machine, cause the machine to perform operations, the operations comprising: Identifying a plurality of neural network NN architectures for solving a machine learning problem; For each NN architecture of the plurality of NN architectures: Initializing the weights for the NN architecture; Calculating a measure expressiveness and a gradient deformation for the NN architecture using the initialized weights; Providing the measure expressiveness and the gradient deformation as inputs to a probabilistic machine learning ML algorithm to predict the performance of the NN architecture, the ML algorithm outputting a predictive distribution; And Providing the output of the predictive distribution as an input to an acquisition function; Selecting the NN architecture for which the value of the acquisition function exceeds a predetermined threshold as the selected NN architecture; Training the selected NN architecture; And Solving the machine learning problem using the trained selected NN architecture.

16. The non-transitory machine-readable medium according to claim 15, wherein the predictive distribution further comprises a posterior mean and variance.

17. The non-transitory machine-readable medium according to claim 15, wherein initializing the weights for the NN architecture further comprises: Initializing the weights using Glorot Normal initialization, independent normal distribution, or Laplace distribution.

18. The non-transitory machine-readable medium according to claim 15, wherein the acquisition function is one of expected improvement, upper confidence bound, or Thompson sampling.

19. The non-transitory machine-readable medium according to claim 15, wherein the NN architecture is a fixed-depth neural network architecture, wherein the unit is a recurrent basic unit repeated multiple times, and wherein the NN architecture is selected based on reasoning the unit with the highest accuracy.

20. The non-transitory machine-readable medium according to claim 19, wherein the unit is defined as a directed acyclic graph having one or more blocks, where each block receives two inputs, performs corresponding operations on each of the inputs, and returns the sum of the outputs from the two operations.

Images