Choosing a neural network architecture for a supervised machine learning problem

Abstract

Systems and methods for selecting a neural network for a machine learning problem are disclosed. A method includes accessing an input matrix. The method includes accessing a machine learning problem space associated with the machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem. The method includes, for each untrained candidate neural network, computing at least one expressiveness metric that captures the expressiveness of the candidate neural network with respect to the machine learning problem. The method includes, for each untrained candidate neural network, computing at least one trainability metric that captures the trainability of the candidate neural network with respect to the machine learning problem. The method includes selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressiveness metric and the at least one trainability metric.

Description

Background Art

[0001] Many different types of neural network architectures are known (e.g., convolutional neural networks, feed-forward neural networks, etc.) Selecting a neural network architecture (and sub-architectures within a given architecture type) to solve a given machine learning problem can be challenging. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Some embodiments of the present technology are shown by way of example and not limitation in the figures of the accompanying drawings.

[0003] Figure 1 An example system is shown in which selecting a neural network architecture for solving a machine learning problem can be implemented in accordance with some embodiments.

[0004] Figure 2 A flowchart is shown of an example method for selecting a neural network architecture for solving a machine learning problem in accordance with some embodiments.

[0005] Figure 3 A flow chart of an example method for reducing error rates according to some embodiments is shown.

[0006] Figures 4A to 4B A flow chart illustrating an example method for selecting a cell from a block according to some embodiments.

[0007] Figure 5 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium and performing any of the methods discussed herein, according to some embodiments. Summary of the invention

[0008] The present disclosure generally relates to machines configured to select a neural network architecture for solving a machine learning problem, including computerized variants of such special-purpose machines and improvements to such variants, and to techniques that improve such special-purpose machines over other special-purpose machines that provide neural network techniques. In particular, the present disclosure is directed to systems and methods for selecting a neural network architecture for solving a machine learning problem in a given machine learning problem space.

[0009] According to some aspects of the technology described herein, a system includes processing hardware and memory. The memory stores instructions that, when executed by the processing hardware, cause the processing hardware to perform operations. The operations include accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem. The operations include, for each untrained candidate neural network, calculating at least one expressivity measure that captures the expressivity of the candidate neural network with respect to the machine learning problem. The operations include, for each untrained candidate neural network, calculating at least one trainability measure that captures the trainability of the candidate neural network with respect to the machine learning problem. The operations include selecting at least one candidate neural network for solving the machine learning problem based on at least one expressivity measure and at least one trainability measure. The operations include providing an output representing the selected at least one candidate neural network.

[0010] According to some aspects of the technology described herein, a machine-readable medium stores instructions that, when executed by one or more machines, cause the one or more machines to perform operations. The operations include accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem. The operations include, for each untrained candidate neural network, computing at least one expressivity metric that captures the expressivity of the candidate neural network with respect to the machine learning problem. The operations include, for each untrained candidate neural network, computing at least one trainability metric that captures the trainability of the candidate neural network with respect to the machine learning problem. The operations include selecting at least one candidate neural network for solving the machine learning problem based on at least one expressivity metric and at least one trainability metric. The operations include providing an output representing the selected at least one candidate neural network.

[0011] According to some aspects of the technology described herein, a method includes accessing an input matrix. The method includes accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem. The method includes, for each untrained candidate neural network, computing at least one expressivity metric that captures the expressivity of the candidate neural network with respect to the machine learning problem. The method includes, for each untrained candidate neural network, computing at least one trainability metric that captures the trainability of the candidate neural network with respect to the machine learning problem. The method includes selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressivity metric and the at least one trainability metric. The method includes providing an output representing the selected at least one candidate neural network. DETAILED DESCRIPTION

[0012] Overview

[0013] The present disclosure particularly describes methods, systems and computer program products that individually provide various functions. In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of various aspects of the different embodiments of the present disclosure. However, it is clear to those skilled in the art that the present disclosure can be practiced without all the specific details.

[0014] As mentioned above, many different types of neural network architectures are known (e.g., feed-forward neural networks, convolutional neural networks, recurrent neural networks, etc.) Selecting a high-performance neural network architecture to solve a given machine learning problem (e.g., a regression problem, a ranking problem, or a classification problem, such as classifying data in a given space, such as classifying images of birds according to the type of bird in the image) can be challenging.

[0015] Some aspects of the technology described herein are directed to solving the following technical problem: selecting a neural network architecture for solving a given machine learning problem from a set of neural network architectures before the neural network architecture is trained. Advantageously, as a result of certain aspects, a high-performance neural network architecture is trained to solve the given machine learning problem, and a less-optimal architecture is not trained. This can save computational time and improve efficiency without resulting in the use of a non-high-performance neural network.

[0016] In some cases, a solution to the problem is implemented at a server. The server accesses, via a data repository, a machine learning problem space associated with the machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem. The server computes, for each untrained candidate neural network, at least one expressiveness metric that captures the expressiveness of the candidate neural network with respect to the machine learning problem. The server computes, for each untrained candidate neural network, at least one trainability metric that captures the trainability of the candidate neural network with respect to the machine learning problem. The server selects, based on the at least one expressiveness metric, the at least one trainability metric, and the architecture of the candidate neural network, at least one candidate neural network for solving the machine learning problem. The server provides an output representing the selected at least one candidate neural network.

[0017] In some cases, the selected at least one candidate neural network is partially or fully trained to solve the machine learning problem. As used herein, a neural network that is "partially or fully trained" may include being trained for several epochs or being trained until a certain convergence indicator has been met. The trained at least one candidate neural network is run on the machine learning problem space to solve the machine learning problem. The server then provides a solution to the machine learning problem generated by the trained at least one candidate neural network.

[0018] According to some examples, at least one expressiveness metric represents a measure (e.g., size or angle) of separation of samples from the machine learning problem space by an untrained candidate neural network. According to some examples, at least one trainability metric represents a function of gradients at the last layer and the first layer given samples from the machine learning problem space. According to some examples, expressibility and trainability metrics may include quantities, metrics, or statistics of the neural network that capture different properties of the architecture, such as expressibility and trainability.

[0019] Figure 1 An example system 100 is shown in which selection of a neural network architecture for solving a machine learning problem may be implemented according to some embodiments. As shown, the system 100 includes a server 110, a data repository 120, and a client device 130 connected to each other via a network 140. The network 140 includes one or more of the Internet, an intranet, a local area network, a wide area network, a wired network, a wireless network, a cellular network, a WiFi network, and the like.

[0020] The client device 130 may be a laptop computer, a desktop computer, a mobile phone, a tablet computer, a smart TV with a processor and a memory, a smart watch, etc. The client device 130 may be used to display output to a user or receive input from a user.

[0021] The data repository 120 may be implemented as a database or any other data storage structure. As shown, the data repository stores a machine learning problem space 122. The machine learning problem space 122 includes data to be classified by a neural network. For example, the machine learning problem space 122 may include photos of birds to be classified according to the type of bird or email messages to be classified as "important email", "unimportant email" or "spam".

[0022] Server 110 may include one or more servers. Server 110 may be implemented as a server farm including multiple servers. As shown, server 110 stores untrained candidate neural networks 112.1-n (where n is a positive integer greater than or equal to 2), selection module 114, and training module 116. Untrained candidate neural networks 112.1-n are neural networks that can be used for various classification tasks. For example, untrained candidate neural networks 112.1-n may include untrained versions of convolutional neural networks or feedforward neural networks.

[0023] The selection module 114 selects at least one of the untrained candidate neural networks 112.1-n for training to solve a machine learning problem associated with the machine learning problem space 122. More details on example operations of the selection module are provided in Figure 2The training module 116 trains the neural network(s) selected (by the selection module 114) (from the untrained candidate neural networks 112.1-n) to solve the machine learning problem. After training, the trained neural network can be used to solve the machine learning problem by classifying data in the machine learning problem space 122 (or another problem space).

[0024] Figure 2 FIG. 2 is a flowchart showing an example method 200 for selecting a neural network architecture for solving a machine learning problem according to some embodiments. As described below, the method 200 uses Figure 1 The method 200 is implemented by the selection module 114 of the server 110. However, the method 200 is not limited by the architecture of the system 100 and can be implemented in other architectures or other systems.

[0025] At operation 210, the selection module 114 accesses (eg, via the network 140) a machine learning problem space 122 associated with a machine learning problem to be solved. The selection module 114 accesses a plurality of untrained candidate neural networks 112.1-n for solving the machine learning problem.

[0026] At operation 220, the selection module 114 calculates, for each untrained candidate neural network 112.1-n, expressiveness metric(s) related to the expressiveness of the candidate neural network with respect to the machine learning problem. The expressiveness metric(s) represent a measure of the separation of samples from the machine learning problem space 122 by the untrained candidate neural network. The measure of separation may be a magnitude or an angle.

[0027] At operation 230, the selection module 114 calculates, for each untrained candidate neural network 112.1-n, trainability measure(s) related to the trainability of the candidate neural network with respect to the machine learning problem.

[0028] At operation 240, the selection module 114 selects candidate neural network(s) for solving the machine learning problem based on the expressivity metric(s) and the trainability metric(s). The selection is also based on the architecture(s) of the candidate neural network(s). The candidate neural network(s) are selected from a plurality of untrained candidate neural networks 112.1-n. In some cases, the selection module 114 selects candidate neural network(s) whose expressivity metric(s) exceed a threshold and whose trainability metric(s) are within a range. The range is defined by a range minimum and a range maximum.

[0029] At operation 250, the selection module 114 provides an output representing the selected candidate neural network(s). In some cases, the training module 116 trains the selected candidate neural network(s) to solve the machine learning problem. The training module 116 runs the trained candidate neural network(s) on the machine learning problem space 122 to solve the machine learning problem. The server 110 provides (e.g., to the client device 130 for display thereat, or to the data repository 120 for storage thereat) a solution to the machine learning problem generated by the trained candidate neural network(s).

[0030] One goal of the techniques described herein is to automatically select and configure a neural network architecture for a given task. More specifically, given a dataset, certain aspects automatically identify the layer type (e.g., convolution, max pooling, fully connected, etc.), its hyperparameters (e.g., stride size, convolution size), its connections to all other layers, and the total number of layers. Some aspects also identify the training algorithm to use (e.g., stochastic gradient descent, RMSProp, Adam, etc.) and how to initialize the weights (e.g., Glorot, Normal, Laplace, Uniform, etc.).

[0031] Some approaches can be grouped based on: (1) how they define the search space architecturally (e.g., unrestricted or restricted); (2) how they explore the space (e.g., reinforcement learning, Monte Carlo tree search); (3) what predictive model they use to guide the search (e.g., sequential model-based optimization, recurrent neural networks (RNNs), genetic algorithms); and (4) whether they use cheap surrogate functions to guide the search more efficiently. In the case of genetic algorithms and reinforcement learning, points 2 and 3 above are merged into one, since they jointly learn a predictive model and explore the space.

[0032] In some embodiments, the search space is explored by means of sequential model-based optimization (SMBO). An example model in SMBO can be a Bayesian recurrent neural network. A set of properties of random neural networks are defined to act as cheap surrogate (cheap surrogate) (or statistics) of the real performance of the neural network. In some cases, other models (e.g., Gaussian processes) that simultaneously output predictions and uncertainty about the predictions are used. In some cases, cheap substitutes (or statistics) are not given as input to the model, but are treated as outputs. Cheap substitutes can include the trainability and expressiveness metrics discussed above.

[0033] In some extended embodiments, the search space is unrestricted. Reinforcement learning or Monte Carlo tree search is used to explore the search space. In some cases, representations of the architecture itself can also be used additionally to help make predictions.

[0034] Some aspects involve two main metrics used to predict the performance of the initial model on the dataset after training before training. Both metrics are statistics of the model collected on random batches from the dataset. In some examples, the batch consists of n points Some aspects ignore the label y i The model typically consists of two components: a deep neural network that embeds the input space into a latent space, and a fully connected linear layer followed by a softmax that transforms the embedding into a probability distribution over a set of possible labels. Let f be the former, i.e., the embedding, and assume it has L layers.

[0035] The measure expressivity is defined in terms of Expression 1, which approximates Expression 2. In Expression 1, {x1, ..., x n} is a batch of input.

[0036]

[0037] Expression 1

[0038]

[0039] Expression 2

[0040] In Expression 2, E denotes the expectation and P is the data distribution for multiple x. Intuitively, this measure represents the tendency of f to expand the input space and pull the points apart. Larger measure expressiveness should be associated with better performance after training. Variations on this measure include sampling (x, x') pairs instead of selecting every pair from the same batch, and selecting powers other than 2 (i.e., for some p>0, ), as well as other methods for testing the "extensibility" (propensity to expand the input space) of neural networks.

[0041] In addition to measuring expressive features, gradient deformation also includes sampling the random gradient vector about the last layer and performing i After the forward computation, backpropagation is performed to obtain the gradient vector for each previous layer. Assume a fixed input x like this i And the last layer gradient vector is fixed, the gradient vector of the parameter w at the lth layer is expressed as Then, the gradient deformation of the parameter w is defined in Expression 3.

[0042]

[0043] Expression 3

[0044] In other words, gradient deformation is a measure of how much the gradients explode or vanish on a typical data point. The larger this is, the worse the performance after training is expected to be, because training (via stochastic gradient descent (SGD)) is expected to be difficult. The variation of this metric consists of i Sampling a new last layer gradient vector, replacing the number of sums with Expression 4, and other methods of measuring gradient explosion / vanishing. The above metrics predict the remaining network performance under fixed architecture and random initialization.

[0045]

[0046] Expression 4

[0047] Given the same settings as in the measure expressivity case, angle expressivity is defined in Expression 5.

[0048]

[0049] Expression 5

[0050] Angular expressivity measures how much f “decorrelates” the input vectors in the sense of pulling their angles apart. Therefore, one would expect that a large angular expressivity is associated with better performance. In some schemes, the prediction quantity is actually The deviation from the asymptotic limit of f as its depth reaches infinity. This asymptotic deviation is represented by the Cauchy error C l -C (l+1) , where C l is the angular expressiveness of the network up to layer l.

[0051] In some cases, statistics can be learned automatically. Although the data-based statistics described in this article are highly correlated with the final performance of the model, they may not be the most predictive statistics that one can extract to predict the generalization ability of a given model. The most predictive statistics may be complex nonlinear functions of the original statistics (i.e., the embeddings and gradient measures used to compute the statistics in the previous section).

[0052] However, like any other function approximation problem in machine learning, these complex functions can be learned if there is enough training data. This will motivate another version of the framework in which the server 110 also learns predictive statistics from the raw data. In particular, some aspects may use a neural network whose input is the data representation of the last layer (i.e., f(x) above), the gradients of the last and first layers, etc.

[0053] In a general version of some aspects, the process of Algorithm 1 is repeated until a predetermined desired error rate is reached or a predetermined total computational cost is reached.

[0054] Algorithm 1

[0055] 1. A variety of deep neural network (DNN) architectures have been proposed. DNN architectures can be sampled from the space of predefined architectures or built using common building blocks.

[0056] 2. Initialize the weights of the architecture using Glorot Normal initialization. In an alternative embodiment, independent normal and Laplace distributions may be used.

[0057] 3. Compute the measure expressivity and gradient deformation for each of the architectures.

[0058] 4. Include the measure expressiveness and gradient deformation of each architecture as input to a Bayesian Recurrent Neural Network (Bayesian RNN) to predict architecture performance. Gaussian processes can also be used for this task. In principle, any probabilistic machine learning algorithm that outputs a distribution over its predictions can be used.

[0059] 5. Use the posterior mean and variance computed using the Bayesian RNN for each in the architecture as input to an acquisition function, such as expected improvement, upper confidence interval limit, or Thompson sampling.

[0060] 6. Sample the frame(s) having value(s) for the acquisition function that exceed a threshold.

[0061] 7. Train the architecture(s) with value(s) for the acquisition function that exceed the threshold and evaluate their performance on the holdout set. Update the model used in operation 4 with the observed performance. Return to operation 1.

[0062] Figure 3 Algorithm 1 is summarized in . Figure 3 A flow 300 of an example method for reducing error rates according to some embodiments is shown.

[0063] At operation 310, a server (e.g., server 110) proposes a plurality of DNN architectures. The DNN architectures may be sampled from a space of predefined architectures or constructed using common building blocks.

[0064] The server initializes the weights of the architecture using Glorot Normal initialization at operation 320. In alternative embodiments, independent normal and Laplace distributions may be used.

[0065] At operation 330 , the server computes the measure expressivity and gradient deformation for each of the architectures.

[0066] At operation 340, the server includes the measure expressiveness and gradient deformation of each architecture as input to a Bayesian RNN to predict architecture performance. Gaussian processes can also be used for this task. In principle, any probabilistic machine learning algorithm that outputs a distribution over its predictions can be used.

[0067] At operation 350 , the server uses the posterior mean and variance computed for each architecture using the Bayesian RNN as input to an acquisition function, such as expected improvement, upper confidence interval limit, or Thompson sampling.

[0068] At operation 360 , the server samples the architecture(s) having value(s) for the acquisition function that exceed a threshold.

[0069] At operation 370, the server trains the architecture(s) with the value(s) for the acquisition function that exceeds the threshold and evaluates their performance on the holdout set. The Bayesian RNN model of operation 340 is updated based on the observed performance. After operation 370, if the value for the acquisition function (of operation 360) is sufficient, the method 300 ends. Otherwise, the method 300 returns to operation 310.

[0070] A specific implementation is described here. Given a fixed deep neural network architecture in which a recursive basic unit (cell) is repeated multiple times, certain aspects involve reasoning about the cell with the highest accuracy. First, a cell is defined as a directed acyclic graph of a certain number of blocks. Each block takes two inputs (I1, I2), performs an operation (O1, O2) on each input, and then returns the sum of the outputs from these two operations. The set of possible inputs for a block is the outputs of all previous blocks in a cell and the outputs of the previous two cells. The operations on these inputs are standard neural network layers, such as convolution, max pooling, etc. It is very important to automatically and accurately search for such basic units in deep neural networks because their performance usually generalizes when applied to larger data sets. To find the best cell with B blocks, Algorithm 2 is used.

[0071] In some cases, the search in Algorithm 2 may be exhaustive, since the number of cells on a block (C) may be small. For example, for 8 possible operations, the search space is 8*8*2*2=256, which can be exhaustively explored, for example, by using 256 graphics processing units (GPUs) in parallel.

[0072] Algorithm 2

[0073] 1. Consider a set of cells in a block (C).

[0074] 2. For each cell in C:

[0075] Initialize the weights of the architecture using Glorot Normal initialization. In alternative embodiments, identically distributed (iid) normal distribution and Laplace distribution can be used.

[0076] Compute measure expressivity and gradient deformation. This can be done over multiple initializations, and can report mean and variance.

[0077] Train the neural network for a fixed number of epochs.

[0078] Calculate the accuracy on the test set.

[0079] 3. If the number of blocks of cells in C is B, stop and return the cell with the highest test accuracy.

[0080] 4. Train a Bayesian Recurrent Neural Network (BRNN) that takes the cell architecture and random statistics as input and predicts the accuracy on the test set.

[0081] 5. Consider the set D, consisting of cells obtained by expanding every cell in C and adding new blocks.

[0082] 6. Compute measures of expressivity and gradient deformation for the units in D. If this computation is too expensive (e.g., due to the large number of units in D), one can (i) train an additional BRNN that only assumes the unit structure and predicts the test set accuracy, and (ii) use it to filter the set D based on the predicted test accuracy.

[0083] 7. Use the BRNN from operation 4 to determine the set of units to consider in the next iteration This decision is made by weighing the trade-off between exploitation and exploration within a Bayesian optimization framework.

[0084] 8.ε→C and then go to step 2.

[0085] exist Figure 4A-4B Algorithm 2 is summarized in . Figure 4A-4B A flow chart is shown of an example method 400 for selecting a cell from a block in accordance with some embodiments.

[0086] like Figure 4A As shown, operation 410 includes sub-operations 412-418. In operation 410, for each unit in block C, the server (eg, server 110) performs sub-operations 412-418.

[0087] At sub-operation 412, the server initializes the weights of the architecture using Glorot Normal initialization. In alternative embodiments, the same distribution normal distribution and Laplace distribution may be used.

[0088] At sub-operation 414, the server computes the measure expressivity and gradient deformation. This can be done over multiple initializations, and the mean and variance can be reported.

[0089] At sub-operation 416, the server trains the neural network for a fixed number of epochs.

[0090] At sub-operation 418 , the server computes the accuracy on the test set. After completing operation 410 for each cell in C, method 400 proceeds to operation 420 .

[0091] At operation 420, the server determines whether the number of blocks of C is within a predetermined range. If so, the method 400 continues to operation 430. If not, the method 400 continues to operation 440.

[0092] In operation 430, if the number of blocks of C is within a predetermined range, the server returns a unit having the highest test accuracy in C. After operation 430, the method 400 ends.

[0093] At operation 440, if the number of blocks of C is not within the predetermined range, the server trains a BRNN that takes the cell architecture and random statistics as input and predicts the accuracy of the test set. After operation 440, the method 400 continues to Figure 4B Operation 450 is shown.

[0094] like Figure 4B As shown, in operation 450, the server generates a set D including cells obtained by expanding each cell in C and adding new blocks.

[0095] At operation 460, the server computes the measure expressiveness and gradient deformation for the units in D. If this computation is too expensive (e.g., due to the large number of units in D), the server may (i) train an additional BRNN that assumes only the unit structure and predicts the test set accuracy, and (ii) use the additional BRNN to filter the set D based on the predicted test accuracy.

[0096] In operation 470, the server uses the BRNN trained in operation 440 to determine a set of units to be considered in the next iteration. . This decision is made by weighing the trade-off between exploitation and exploration within a Bayesian optimization framework.

[0097] At operation 480, the server stores ε as C. After operation 480, method 400 returns to Figure 4A Operation 420.

[0098] Numbering Example

[0099] Certain embodiments are described herein as numbered examples 1, 2, 3, etc. These numbered examples are provided by way of example only, and do not limit the subject technology.

[0100] Example 1 is a system comprising: processing hardware; and a memory storing instructions, the instructions causing the processing hardware to perform operations, the operations comprising: accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem; for each untrained candidate neural network, computing at least one expressivity metric that captures the expressivity of the candidate neural network with respect to the machine learning problem; for each untrained candidate neural network, computing at least one trainability metric that captures the trainability of the candidate neural network with respect to the machine learning problem; selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressivity metric and the at least one trainability metric; and providing an output representing the selected at least one candidate neural network.

[0101] In Example 2, the subject matter of Example 1 includes: wherein the at least one expressive metric represents a measure of separation of samples from the machine learning problem space by the untrained candidate neural network.

[0102] In Example 3, the subject matter of Example 2 includes wherein the measure of separation is size.

[0103] In Example 4, the subject matter of Examples 2-3 includes wherein the measure of separation is an angle.

[0104] In Example 5, the subject matter of Examples 1-4 includes, wherein the at least one trainability metric represents a stochastic gradient descent of weights in the candidate neural network during a first stage of training.

[0105] In Example 6, the subject matter of Examples 1-5 includes, wherein selecting the at least one candidate neural network for solving the machine learning problem comprises: selecting the at least one candidate neural network having the at least one expressiveness metric exceeding a threshold and the at least one trainability metric within a range, wherein the range is defined by a range minimum and a range maximum.

[0106] In Example 7, the subject matter of Examples 1-6 includes wherein the operations further include: training the at least one candidate neural network to solve the machine learning problem.

[0107] In Example 8, the subject matter of Example 7 includes that the operations further include: running at least one candidate neural network that has been trained on the machine learning problem space to solve the machine learning problem; and providing a solution to the machine learning problem generated by the at least one candidate neural network that has been trained.

[0108] Example 9 is a non-transitory machine-readable medium storing instructions that cause one or more machines to perform operations, the operations comprising: accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem; for each untrained candidate neural network, computing at least one expressivity metric that captures the expressivity of the candidate neural network with respect to the machine learning problem; for each untrained candidate neural network, computing at least one trainability metric that captures the trainability of the candidate neural network with respect to the machine learning problem; selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressivity metric and the at least one trainability metric; and providing output representing the selected at least one candidate neural network.

[0109] In Example 10, the subject matter of Example 9 includes: wherein the at least one expressive metric represents a measure of separation of samples from the machine learning problem space by the untrained candidate neural network.

[0110] In Example 11, the subject matter of Example 10 includes wherein the measure of separation is size.

[0111] In Example 12, the subject matter of Examples 10-11 includes wherein the measure of separation is an angle.

[0112] In Example 13, the subject matter of Examples 9-12 includes wherein the at least one trainability metric represents a stochastic gradient descent of weights in the candidate neural network during a first stage of training.

[0113] In Example 14, the subject matter of Examples 9-13 includes, wherein selecting the at least one candidate neural network for solving the machine learning problem comprises: selecting the at least one candidate neural network having the at least one expressive metric exceeding a threshold and the at least one trainability metric within a range, wherein the range is defined by a range minimum and a range maximum.

[0114] Example 15 is a method comprising: accessing a machine learning problem space of an amount related to a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem; for each untrained candidate neural network, computing at least one expressivity metric that captures the expressivity of the candidate neural network with respect to the machine learning problem; for each untrained candidate neural network, computing at least one trainability metric that captures the trainability of the candidate neural network with respect to the machine learning problem; selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressivity metric and the at least one trainability metric; and providing an output representing the selected at least one candidate neural network.

[0115] In Example 16, the subject matter of Example 15 includes: wherein the at least one expressive metric represents a measure of separation of samples from the machine learning problem space by the untrained candidate neural network.

[0116] In Example 17, the subject matter of Example 16 includes wherein the measure of separation is size.

[0117] In Example 18, the subject matter of Examples 16-17 includes wherein the measure of separation is an angle.

[0118] In Example 19, the subject matter of Examples 15-18 includes wherein the at least one trainability metric represents a stochastic gradient descent of weights in the candidate neural network during a first stage of training.

[0119] In Example 20, the subject matter of Examples 15-19 includes, wherein selecting the at least one candidate neural network for solving the machine learning problem comprises: selecting the at least one candidate neural network having the at least one expressiveness metric exceeding a threshold and the at least one trainability metric within a range, wherein the range is defined by a range minimum and a range maximum.

[0120] Example 21 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations for implementing any of Examples 1-20.

[0121] Example 22 is an apparatus comprising means for implementing any of Examples 1-20.

[0122] Example 23 is a system for implementing any of Examples 1-20.

[0123] Example 24 is a method for implementing any of Examples 1-20.

[0124] Components and Logic

[0125] Certain embodiments are described herein as including logic or multiple components or mechanisms. Components may constitute software components (e.g., code embodied on a machine-readable medium) or hardware components. A "hardware component" is a tangible unit that is capable of performing certain operations and may be configured or arranged in some physical manner. In various example embodiments, one or more computer systems (e.g., a stand-alone computer system, a client computer system, or a server computer system) or one or more hardware components of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware component that operates to perform certain operations described herein.

[0126] In some embodiments, the hardware component can be implemented mechanically, electronically, or any suitable combination thereof. For example, the hardware component may include a dedicated circuit system or logic that is permanently configured to perform certain operations. For example, the hardware component may be a dedicated processor, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The hardware component may also include a programmable logic or circuit system that is temporarily configured by software to perform certain operations. For example, the hardware component may include software executed by a general-purpose processor or other programmable processor. Once configured by such software, the hardware component will become a specific machine (or a specific component of a machine) that is specially customized for performing the configured function, and is no longer a general-purpose processor. It will be appreciated that it may be possible to consider whether to mechanically implement the hardware component in a dedicated and permanently configured circuit system or in a temporarily configured circuit system (e.g., configured by software) in terms of cost and time.

[0127] Thus, the phrase "hardware component" should be understood to include a tangible record, i.e., physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular manner or perform certain operations described herein. As used herein, a "hardware-implemented component" refers to a hardware component. Considering embodiments in which hardware components are temporarily configured (e.g., programmed), each of the hardware components in the hardware component may not be configured or instantiated at any instance in time. For example, in the case where the hardware component includes a general-purpose processor that is configured by software to be a special-purpose processor, the general-purpose processor may be configured as different special-purpose processors (e.g., including different hardware components) at different times. Thus, the software configures one or more specific processors, for example, to constitute a specific hardware component at one instance in time, and to constitute different hardware components at different instances in time.

[0128] Hardware components can provide information to other hardware components and receive information from other hardware components. Therefore, the described hardware components can be considered to be communicatively coupled. In the case of multiple hardware components simultaneously, communication can be achieved by signal transmission (e.g., by appropriate circuits and buses) between or among two or more hardware components in the hardware components. In an embodiment in which multiple hardware components are configured or instantiated at different times, communication between these hardware components can be achieved, for example, by storing and retrieving information in a memory structure accessible to multiple hardware components. For example, a hardware component can perform an operation and store the output of the operation in a memory device to which it is communicatively coupled. Then, another hardware component can access the memory device at a later time to retrieve and process the stored output. The hardware component can also initiate communication with an input or output device, and can operate on a resource (e.g., a collection of information).

[0129] The various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily configured (e.g., configured by software) or permanently configured to perform the relevant operations. Such processors, whether temporarily or permanently configured, may constitute processor-implemented components that operate to perform one or more operations or functions described herein. As used herein, "processor-implemented components" refers to hardware components implemented using one or more processors.

[0130] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are examples of hardware. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented components. In addition, one or more processors may also be operable in a "cloud computing" environment or as "software as a service" (SaaS) to support the execution of related operations. For example, at least some of the operations may be performed by a group of computers (as an example of a machine including a processor), which may be accessed over a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., APIs).

[0131] The execution of certain operations may be distributed among processors, not only residing within a single computing machine, but also deployed across multiple machines. In some example embodiments, the processor or processor-implemented components may be located in a single geographic location (e.g., in a home environment, an office environment, or a server farm). In other example embodiments, the processor or processor-implemented components may be distributed across multiple geographic locations.

[0132] Example machine and software architecture

[0133] In some embodiments, in combination Figures 1 to 4BThe described components, methods, applications, etc. are implemented in the context of a machine and associated software architecture. The following sections describe representative software architecture(s) and machine (eg, hardware) architecture(s) suitable for use with the disclosed embodiments.

[0134] Software architectures are used in conjunction with hardware architectures to create devices and machines that are tailored for specific purposes. For example, a specific hardware architecture combined with a specific software architecture will create a mobile device, such as a mobile phone, a tablet device, etc. Slightly different hardware and software architectures can produce smart devices for the "Internet of Things", while another combination produces a server computer for use within a cloud computing architecture. Not all combinations of such software and hardware architectures are presented herein, as those skilled in the art can readily understand how to implement the disclosed subject matter in contexts different from the disclosure contained herein.

[0135] Figure 5 is a block diagram illustrating components of a machine 500 capable of reading instructions from a machine-readable medium (e.g., a machine-readable storage medium) and performing any one or more of the methods discussed herein, according to some example embodiments. Specifically, Figure 5 A schematic representation of a machine 500 is shown in the form of an example computer system within which instructions 516 (e.g., software, programs, applications, applet, apps, or other executable code) may be executed to cause the machine 500 to perform any one or more of the methodologies discussed herein. The instructions 516 transform a general purpose, unprogrammed machine into a specific machine that is programmed to perform the functions described and illustrated in the manner described. In alternative embodiments, the machine 500 operates as a standalone device or may be coupled (e.g., networked) to other machines. In a network deployment, the machine 500 may operate as a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine 500 may include, but is not limited to, a server computer, a client computer, a PC, a tablet computer, a laptop computer, a netbook, a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart device), other smart devices, web appliances, a network router, a network switch, a network bridge, or any machine capable of sequentially or otherwise executing instructions 516 that specify actions to be taken by the machine 500. Furthermore, while only a single machine 500 is shown, the term "machine" should also be taken to include a collection of machines 500 that individually or collectively execute instructions 516 to perform any one or more of the methodologies discussed herein.

[0136] The machine 500 may include a processor 510, a memory 530, and I / O components 550 that may be configured to communicate with each other, for example, via a bus 502. In an example embodiment, the processor 510 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, a processor 512 and a processor 514 that may execute instructions 516. The term "processor" is intended to include multi-core processors that may include two or more independent processors (sometimes referred to as "cores") that may execute instructions simultaneously. Although Figure 5 Multiple processors 510 are shown, but machine 500 may include a single processor with a single core, a single processor with multiple cores (e.g., a multi-core processor), multiple processors with a single core, multiple processors with multiple cores, or any combination thereof.

[0137] The memory / storage 530 may include a memory 532, such as a main memory or other memory storage device, and a storage unit 536, both of which may be accessed by the processor 510 via the bus 502. The storage unit 536 and the memory 532 store instructions 516 that embody any one or more of the methods or functions described herein. During execution of the instructions 516 by the machine 500, the instructions 516 may also reside, in whole or in part, within the memory 532, within the storage unit 536, within at least one of the processors 510 (e.g., within a cache memory of the processor), or any suitable combination thereof. Thus, the memory 532, the storage unit 536, and the memory of the processor 510 are examples of machine-readable media.

[0138] As used herein, "machine-readable medium" refers to a device capable of temporarily or permanently storing instructions (e.g., instructions 516) and data, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of storage devices (e.g., erasable programmable read-only memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database or associated caches and servers) capable of storing instructions 516. The term "machine-readable medium" should also be understood to include any medium or combination of multiple media capable of storing instructions (e.g., instructions 516) for execution by a machine (e.g., machine 500), such that these instructions, when executed by one or more processors of the machine (e.g., processor 510), cause the machine to perform any one or more of the methods described herein. Therefore, "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network including multiple storage devices or devices. The term "machine-readable medium" does not include the signal itself.

[0139] I / O components 1050 may include a variety of components for receiving input, providing output, generating output, transmitting information, exchanging information, capturing measurements, and the like. The specific I / O components 550 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that I / O components 550 may include Figure 5Many other components not shown in the figure. Only to simplify the following discussion, the I / O components 550 are grouped according to function only, and the grouping is by no means limiting. In various example embodiments, the I / O components 550 may include output components 552 and input components 554. The output components 552 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 554 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical camera keyboards, or other alphanumeric input components), pointing-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or other pointing tools), tactile input components (e.g., physical buttons, touch screens that provide positioning and / or force of touch or touch gestures, or other tactile input components), audio input components (e.g., microphones), etc.

[0140] In other example embodiments, the I / O component 550 may include, among other components, a biometric component 556, a motion component 558, an environmental component 560, or a position component 562. For example, the biometric component 556 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body gestures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweat, or brain waves), measuring exercise-related metrics (e.g., distance moved, speed moved, or time spent exercising), identifying a person (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 558 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental component 560 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor for detecting hazardous gas concentrations to ensure safety or for measuring pollutants in the atmosphere), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Location component 562 may include a positioning sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects air pressure from which altitude can be derived), an orientation sensor component (e.g., a magnetometer), and the like.

[0141] Communication can be achieved using a variety of technologies. I / O components 550 may include a communication component 564 operable to couple machine 500 to network 580 or device 570 via coupling 582 and coupling 572, respectively. For example, communication component 564 may include a network interface component or another suitable device for interfacing with network 580. In other examples, communication component 564 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, Components (e.g. Low Energy), Components, and other communication components for providing communication by other means. Device 570 can be another machine or any of a variety of peripheral devices (eg, a peripheral device coupled via USB).

[0142] In addition, the communication component 564 can detect the identifier, or include a component operable to detect the identifier. For example, the communication component 564 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component, or an acoustic detection component (e.g., a microphone for an audio signal of an identification tag). In addition, various information can be derived via the communication component 564, such as positioning via Internet Protocol (IP) geolocation, Positioning is performed by signal triangulation, positioning is performed via detection of NFC beacon signals that can indicate a specific location, and so on.

[0143] In various example embodiments, one or more portions of network 580 may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a WAN, a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of a public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, 580 or a portion of network 580 may include a wireless or cellular network, and coupling 582 may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or other type of cellular or wireless coupling. In this example, coupling 582 may implement any of a variety of types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, third generation partnership project (3GPP) including 5G, fourth generation wireless (4G) network, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standard, other standards defined by various standards development organizations, other remote protocols, or other data transmission technologies.

[0144] Instructions 516 may be transmitted or received using a transmission medium via a network interface device (e.g., a network interface component included in communication component 564) over network 580 and using any of a number of well-known transmission protocols (e.g., HTTP). Similarly, instructions 516 may be transmitted or received to device 570 via coupling 572 (e.g., a peer-to-peer coupling) using a transmission medium. The term "transmission medium" should be taken to include any intangible medium capable of storing, encoding, or carrying instructions 516 for execution by machine 500, and includes digital or analog communication signals or other intangible media used to facilitate communication of such software.

Claims

1. A system comprising: Processing hardware; as well as a memory storing instructions that cause the processing hardware to perform operations comprising: accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem, the machine learning problem space comprising data to be processed by the trained neural networks; For each untrained candidate neural network, calculate at least one expressiveness metric based on the data of the machine learning problem space, the at least one expressiveness metric capturing the expressiveness of the candidate neural network with respect to the machine learning problem, the at least one expressiveness metric being calculated without training the candidate neural network; For each untrained candidate neural network, calculate at least one trainability metric based on data of the machine learning problem space, wherein the at least one trainability metric captures the trainability of the candidate neural network with respect to the machine learning problem; selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressiveness metric and the at least one trainability metric, wherein selecting the at least one candidate neural network for solving the machine learning problem comprises selecting the at least one candidate neural network having the at least one expressiveness metric exceeding a threshold and the at least one trainability metric within a range; and An output representing the selected at least one candidate neural network is provided.

2. The system of claim 1, wherein the at least one expressiveness metric represents a measure of separation of samples from the machine learning problem space.

3. The system of claim 2, wherein the metric of separation is size. The system of claim 2 , wherein the measure of separation is an angle.

5. The system of claim 1 , wherein the at least one trainability metric represents stochastic gradient descent of weights in the candidate neural network during a first phase of training.

6. The system of claim 1, wherein the operations further comprise: The selected at least one candidate neural network is trained to solve the machine learning problem.

7. The system of claim 6, wherein the operations further comprise: Running the at least one candidate neural network that has been trained on the machine learning problem space to solve the machine learning problem; as well as A solution to the machine learning problem generated by the at least one candidate neural network that has been trained is provided.

8. A non-transitory machine-readable medium storing instructions that cause one or more machines to perform operations comprising: accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem, the machine learning problem space comprising data to be processed by the trained neural networks; For each untrained candidate neural network, calculate at least one expressiveness metric based on the data of the machine learning problem space, the at least one expressiveness metric capturing the expressiveness of the candidate neural network with respect to the machine learning problem, the at least one expressiveness metric being calculated without training the candidate neural network; For each untrained candidate neural network, calculate at least one trainability metric based on data of the machine learning problem space, wherein the at least one trainability metric captures the trainability of the candidate neural network with respect to the machine learning problem; selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressiveness metric and the at least one trainability metric, wherein selecting the at least one candidate neural network for solving the machine learning problem comprises selecting the at least one candidate neural network having the at least one expressiveness metric exceeding a threshold and the at least one trainability metric within a range; as well as An output representing the selected at least one candidate neural network is provided.

9. The machine-readable medium of claim 8, wherein the at least one expressiveness metric represents a measure of separation of samples from the machine learning problem space.

10. The machine-readable medium of claim 9, wherein the metric of separation is size. The machine-readable medium of claim 9 , wherein the measure of separation is an angle.

12. The machine-readable medium of claim 8, wherein the at least one trainability metric represents stochastic gradient descent of weights in the candidate neural network during a first stage of training.

13. A method comprising: accessing a machine learning problem space associated with a machine learning problem and a plurality of untrained candidate neural networks for solving the machine learning problem, the machine learning problem space comprising data to be processed by the trained neural networks; For each untrained candidate neural network, calculate at least one expressiveness metric based on the data of the machine learning problem space, the at least one expressiveness metric capturing the expressiveness of the candidate neural network with respect to the machine learning problem, the at least one expressiveness metric being calculated without training the candidate neural network; For each untrained candidate neural network, calculate at least one trainability metric based on data of the machine learning problem space, wherein the at least one trainability metric captures the trainability of the candidate neural network with respect to the machine learning problem; selecting at least one candidate neural network for solving the machine learning problem based on the at least one expressiveness metric and the at least one trainability metric, wherein selecting the at least one candidate neural network for solving the machine learning problem comprises selecting the at least one candidate neural network having the at least one expressiveness metric exceeding a threshold and the at least one trainability metric within a range; as well as An output representing the selected at least one candidate neural network is provided.

14. The method of claim 13, wherein the at least one expressiveness metric represents a measure of separation of samples from the machine learning problem space.

15. The method of claim 14, wherein the measure of separation is size. The method of claim 14 , wherein the measure of separation is an angle.

17. The method of claim 13, wherein the at least one trainability metric represents stochastic gradient descent of weights in the candidate neural network during a first phase of training.

Images