Automatic generation of machine learning models

Abstract

To provide a method of automatically generating a machine learning model such as a neural network and so on.SOLUTION: In a system having a hardware processing unit and a storage resource, the storage resource stores computer-readable commands. The hardware processing unit executes a repetitive model growth process including acquisition of a child model through modification of a parent model by the computer readable commands. The repetitive model growth process includes selection of a candidate layer to be included in the child model based on at least weight learnt in an initial process of the candidate layer. The system outputs a final model selected from the child models.SELECTED DRAWING: Figure 1

Description

[Background technology]

[0001] background

[0001] Traditionally, machine learning models have been manually constructed by experts who define the model structure and use automated techniques to train the model. As machine learning models have grown more complex, various attempts have been made to automate the process of generating machine learning models. However, these efforts have met with limited success. Summary of the Invention [Means for solving the problem]

[0002] overview

[0002] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0003] This description generally relates to techniques for automatically generating machine learning models. An example includes a method or technique that can be executed on a computing device. The method or technique may include performing two or more iterations of an iterative model growth process. The iterative model growth process may include selecting a particular parent model from a parent model pool of one or more parent models, generating multiple candidate layers, and initializing the multiple candidate layers while reusing learned parameters and / or structure of the particular parent model. The iterative model growth process may also include selecting specific candidate components, such as layers, for inclusion in child models for training. Each child model may include the particular parent model and one or more of the particular candidate layers or other structures. The iterative model growth process may also include training multiple child models to obtain trained child models and evaluating the trained child models using one or more criteria. The iterative model growth process may also include designating each trained child model as a new parent model based at least on the evaluation and adding the new parent model to the parent model pool. The method or technique may also include selecting at least one trained child model after two or more iterations as a final model and outputting the final model.

[0004] Another example includes a system with a hardware processing unit and a storage resource. The storage resource can store computer-readable instructions that, when executed by the hardware processing unit, cause the hardware processing unit to perform an iterative model growth process that includes modifying a parent model to obtain a child model. The iterative model growth process can include selecting candidate layers for inclusion in the child model based at least on weights learned in an initialization process for the candidate layers. The computer-readable instructions can also cause the hardware processing unit to output a final model selected from the child models.

[0005] Another example includes a computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to perform operations. The operations may include performing two or more iterations of an iterative model growing process. The iterative model growing process may include selecting a particular parent model from a parent model pool of one or more parent models, initializing multiple candidate layers, and selecting multiple child models for training. Each child model may include at least one of a structure inherited from the particular parent model and the candidate layers. The iterative model growing process may also include training the multiple child models to obtain trained child models, and designating each trained child model as a new parent model based at least on one or more criteria. The iterative model growing process may also include adding the new parent model to the parent model pool. The operations may also include selecting at least one trained child model as a final model after the two or more iterations and outputting the final model.

[0006]

[0006] The examples provided above are intended to provide a quick reference to assist the reader and are not intended to define the scope of the concepts described herein.

[0007] BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The detailed description is set forth with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. Use of like reference numbers in different instances in the description and drawings may refer to similar or identical items. [Brief explanation of the drawings]

[0008] [Figure 1]

[0008] An example of a method or technique for automatically generating machine learning models consistent with some implementations of the present concepts is provided. [Figure 2]

[0009] We present an example of a method for generating candidate layers for machine learning models that is consistent with some implementations of the present concepts. [Figure 3]

[0010] We present an example of a technique for initializing candidate layers of a machine learning model that is consistent with some implementations of the present concepts. [Figure 4]

[0011] We present another example of a technique for initializing candidate layers of a machine learning model, consistent with some implementations of the present concepts. [Figure 5]

[0012] Here is an example of an approach for training a child model that is consistent with some implementations of the present concepts. [Figure 6]

[0013] 1 shows scatter plots associated with successive iterations of an iterative model growth process consistent with some implementations of the present concepts. [Figure 7]

[0013] Figure 1 shows scatter plots associated with successive iterations of an iterative model growth process consistent with some implementations of the present concepts. [Figure 8]

[0013] Figure 1 shows scatter plots associated with successive iterations of an iterative model growth process consistent with some implementations of the present concepts. [Figure 9]

[0014] An example process flow for automatically generating machine learning models consistent with some implementations of the present concepts is shown below. [Figure 10]

[0015] An example of a system consistent with implementing some of the concepts is shown below. [Figure 11]

[0016] 1 shows an example of a graphical user interface consistent with some implementations of the present concepts. DETAILED DESCRIPTION OF THE INVENTION

[0009] Detailed Description overview

[0017] There are various types of machine learning frameworks that can be trained using supervised and / or unsupervised learning. Support vector machines, decision trees, and neural networks are just a few examples of machine learning frameworks that are suitable for supervised learning, where the model can learn from labeled training data. Some machine learning frameworks, such as neural networks, use layers of nodes that perform specific operations.

[0010]

[0018] In a neural network, nodes are connected to each other by one or more edges. A neural network may include an input layer, an output layer, and one or more hidden layers. Each node processes its individual input according to a predetermined function and may provide an output to a subsequent layer or, in some cases, to a previous layer. The input to a given node may be multiplied by the corresponding weight value of the edge between the input and the node. In addition, nodes may have individual bias values that are also used to produce the output. Various training procedures may be applied to learn the edge weights and / or bias values. As used herein, the term "learned parameters" refers to parameters, such as edges and bias values, that are learned by training a hierarchical machine learning model, such as a neural network.

[0011]

[0019] Neural network structures can be constructed modularly. For example, one or more layers of nodes may collectively perform a particular operation, such as a pooling operation or a convolution operation. The various layers can then be connected to form an overall network structure. As used herein, the term "layer" refers to a group of nodes that share connectivity to one or more input layers and one or more target layers that receive output from the nodes in that layer. The term "operation" refers to a function that may be performed by one or more layers of nodes. The term "model structure" refers to the overall architecture of a hierarchical model, including the number of layers, their connectivity, and the type of operation performed by each layer. The term "neural network structure" refers to the model structure of a neural network. The disclosed implementations primarily use the neural network structure as an example of the model structure of a hierarchical machine learning model. The term "trained model" refers to the model structure as well as the learned parameters for the model structure. Note that two trained models may share the same model structure and still have different learned parameters, for example, if the two models are trained based on different training data or if the underlying stochastic process is in the training process.

[0012]

[0020] As mentioned above, one way to generate a model structure is for a human to manually define the model structure. The model structure can then be trained by a computer based on some training dataset to obtain a trained model, and then a validation dataset can be used to validate the trained model. Modifications to the model structure can then be manually generated, for example, by adding or removing layers or connections between layers. The modified structure can then be trained again from scratch to obtain additional trained models, which can be compared with each other to select a final model and corresponding structure that performs well for a given task. However, this approach requires the involvement of humans who are experts in a specific domain to create the initial model structure and modifications, and to select the final model structure.

[0013]

[0021] Another approach is to automate the process by using a computer to generate various model structures and to select a final model structure from the generated structures. However, past efforts to automate the generation of model structures have met with limited success. Although modern computers have made great advances in computational power, existing approaches for automatically generating model structures, such as neural network structures, tend to explore a limited search space or require an impractical amount of computational resources. Indeed, given currently available computing hardware, while generating model structures tends to be computationally feasible, independently training many different model structures tends to be computationally infeasible.

[0014]

[0022] One way to quantify the training time of a model structure is by defining a reference computational resource, such as a virtual machine or processor, and the time it takes to complete training on that reference computational resource. For example, one day of training on a particular type of graphics processing unit ("GPU") can be referred to as a GPU-day, and the computational cost of training a given model can be specified as the number of GPU-days. Other approaches are also possible, such as the number of hours of training on a particular CPU or FPGA model, the time spent on a virtual machine, etc.

[0015]

[0023] One approach to automating the generation of model structures is to simply define a search space and generate all model structures within that search space. All possible model structures can then be trained independently to obtain trained models, and one of the trained models can be selected as the final model. In general, the search space can be defined by limiting the depth of the model structures, the operations performed by the models, and the connectivity of the models. For example, a computer can be programmed to consider all fully connected model structures of depth 5, with each of the five layers performing one of several possible operations.

[0016]

[0024] However, this approach is not feasible for many models. First, model structures can grow very large, with many different layers and nodes. Second, such large model structures can take a very long time to train. Finally, this approach does not consider other model structures outside the search space that may be better suited to the task at hand. For example, if the task is image recognition and a six-layer model is significantly better for this task than any plausible five-layer model, limiting the search space to five layers will not find a six-layer model.

[0017]

[0025] More sophisticated approaches for automatically generating model structures include creating a separate "controller" model that is trained to generate new model structures for a given task. However, past efforts using controller models to generate new model structures still suffer from some of the drawbacks described above. When a large search space is explored, for example, when the controller model is expected to explore a wide range of potential model structures, each model structure needs to be created, fully trained, and evaluated against other models. Efforts to automate the learning of new model structures with relatively unconstrained search spaces have met with limited success, as training can take days on large datasets. For example, such approaches can take hundreds or thousands of GPU-days to output a final model structure with acceptable performance levels.

[0018]

[0026] Another approach to automatically generating model structures is to significantly constrain the search space and search a relatively limited set of models within the constrained search space. For example, some past approaches define an outer model skeleton with a specified number of modular layers and types of connections between modules. These approaches then generate and evaluate various candidate substructures or "cells," which are iterated within the outer model skeleton. However, these approaches do not consider alternative outer skeletons, alternative connections between cells, or the use of different types of substructures within each cell.

[0019]

[0027] In contrast to the deficiencies of conventional techniques outlined above, the disclosed implementations can generate model structures within a relatively unconstrained search space while saving significant training time compared to the techniques discussed above. By exploring a broad search space, the disclosed implementations can potentially find new model structures that provide better performance than might be possible in a more constrained search space. For example, the disclosed implementations can find model structures that are not limited to a predefined external skeleton and can also find model structures that have different types of operations in various layers of the model.

[0020]

[0028] The disclosed implementations may utilize several techniques to avoid separately generating and training every model structure in the search space. Rather, the disclosed implementations direct the growth of new model structures to portions of the search space that are expected to contain improved model structures compared to structures already evaluated, while avoiding exploring other portions of the search space that are unlikely to contain improved model structures.

[0021] How to generate a model structure

[0029] The following discussion provides an overview of functionality that may enable the automatic generation of model structures, such as neural networks, to be implemented. Figure 1 illustrates an example of a method 100 consistent with the present concepts. As discussed further below, method 100 may be implemented on many different types of devices, for example, by one or more cloud servers, by client devices such as laptops, tablets, or smartphones, or by a combination of one or more servers, client devices, etc.

[0022]

[0030] Method 100 begins at block 102, where a parent model pool is initialized. For example, one or more initial parent models may be added to the parent model pool. The initial parent models may be known models that have previously been determined to perform well for a particular task, may be randomly generated, or may simply be predetermined default seed models.

[0023]

[0031] The method 100 continues at block 104, where a particular parent model is selected from the pool. For example, the parent model may be selected randomly or deterministically from the pool. As discussed further below, the particular parent model has a corresponding model structure that can be modified.

[0024]

[0032] Method 100 continues at block 106, where a candidate layer is generated and initialized. Generally, generating a candidate layer may include selecting one or more layers of a particular parent model to provide inputs to the candidate layer and selecting another "target" layer of the particular parent model to receive the output of the candidate layer. Thus, the candidate layer may be considered an addition to the structure of the particular parent model. Additionally, generating a candidate layer may include selecting operations to perform on the inputs provided by the selected layers of the particular parent model. As discussed further herein, initializing a candidate layer may include performing some initial training on the candidate layer.

[0025]

[0033] Method 100 continues at block 108, where particular candidate layers are selected for inclusion in one or more child models derived from the parent model. As discussed in more detail below, for example, particular candidate layers may be selected based on initialization parameters learned when initializing the candidate layers. In other cases, each of the candidate layers is selected for inclusion in the child model. In either case, the child model may be considered a model that inherits the structure of the parent model and further includes one or more of the selected candidate layers.

[0026]

[0034] Method 100 continues at block 110, where a child model is trained. As discussed in more detail below, in some cases, the child model is trained by reusing learned parameters from the parent model. In these cases, the child model can be considered to initially inherit learned parameters from the parent model, but those learned parameters can be further adjusted during training of the child model.

[0027]

[0035] Method 100 continues at block 112, where the trained child model is evaluated according to one or more criteria. Generally, the criteria may relate to the model's performance in a given task, e.g., accuracy, and / or other factors, such as the computational cost of training the child model.

[0028]

[0036] The method 100 continues at block 114, where each child model is designated as a new parent model based on the evaluation at block 112. The new parent model is added to the parent model pool.

[0029]

[0037] The method 100 continues at decision block 116, where a determination is made as to whether a stopping condition has been reached. The stopping condition may define a specified amount of computational resources to be used (e.g., a budget in GPU-days), a specified performance criterion (e.g., a threshold accuracy), a specified amount of time, etc.

[0030]

[0038] If a stopping condition has not been reached, the method continues to block 104, where a subsequent iteration of blocks 104-116 may be performed. Generally speaking, blocks 104-116 may be viewed as an iterative model growing process that may be repeated for multiple iterations until a stopping condition is reached.

[0031]

[0039] If a stopping condition has been reached, the method proceeds to block 118, where a final model is output. For example, the final model may be selected from all of the child models according to one or more criteria such as those discussed above with respect to block 112.

[0032]

[0040] Method 100 is often performed to generate a model that is well suited to a particular application. For example, in a face recognition scenario, the training data may include labeled examples of images that indicate whether the image contains a face and where the face is located within the image. In a scene segmentation example, the training data may include labeled examples of video that has been divided into predefined segments. By iteratively generating new child models and evaluating them against a training dataset for a particular task, method 100 can output a final model that performs well at the particular task.

[0033] Candidate layer generation

[0041] 2 illustrates a parent model 210 and a candidate layer 220. An example of how a candidate layer can be derived from the structure of a parent model is described below. Note that, as discussed in more detail below, FIG. 2 illustrates an approach for generating multiple candidate layers from a given parent model, rather than a single instance of a parent model and candidate layer.

[0034]

[0042] In Figure 2, elements of the parent model are shown with solid lines, and elements of the candidate layers are shown with dotted lines. In this example, the parent model includes model input 211, layers 212, 213, 214, and 215, and model output 216. The parent model may also include one or more other layers (not shown), represented by circles in Figure 2. In general, model inputs may include features to be processed by the model, such as raw image, video, audio, and / or text data. Outputs may represent the results of the model's computations, such as the identification of faces in images, segmented video, transcribed audio, semantic representations of text, etc.

[0035]

[0043] In this example, the candidate layer includes individual operations 221, 222, and 223, and aggregation operation 224. As discussed further herein, the individual operations 221, 222, and 223 may generally include convolution, pooling, etc. The aggregation operation 224 may include manipulating the outputs of the individual operations to conform to a target layer, e.g., layer 215, of the parent model that receives the output from the aggregation operation. For example, the aggregation operation may concatenate the outputs of the individual operations and project them into a form or shape that matches the input of the target layer.

[0036]

[0044] As discussed above, block 106 of method 100 can be used to generate candidate layers. Generating candidate layers may include selecting one or more input layers from a parent model, selecting one or more target layers of the parent model to receive the output of the candidate layer, and selecting one or more operations to be performed by the candidate layer on the input. Additionally, generating candidate layers may include selecting operating parameters of the operations, such as convolution size, convolution stride, pooling window, etc.

[0037]

[0045] In this example, the search space of possible candidate layers can be defined as follows: Let x1,...,x L A candidate layer can be defined by a tuple (xout, xin,1, op1, xin,2, op2, ..., xin,J, opJ), where J is a positive integer and x out , x in,1 , ..., x in,K are the existing layers, op1, ..., op J is an operation such as convolution and pooling. x out is exactly all x in the topological order of the parent model's computation graph. in,i Therefore, no direct cycle can be formed.

number

[0038]

[0046] Candidate layer x c A particular algorithm for forming the target layer x is as follows: First, we extract the target layer x from the layers that were in the parent model 210. out Then, the three input layers x in,i You can select x c To ensure that the local layer is accessible, we use the x that was in the initial parent model. out x as the deepest input of in1 You can select x in,2 and x in,3 is x out We can randomly and uniformly sample with replacement from all layers of the parent model that are topologically earlier than . Then, we can randomly and uniformly select the operations to be performed on each input layer from the set of operations.

[0039]

[0047] For example, the set of operations can be predetermined. Specific examples of operations include convolution, pooling, and matching operations. Each type of operation can have different operation parameters. For example, a convolution operation can have a specified kernel size parameter, such as 1x1, 3x3, 5x5, 7x7, etc. A convolution can also have a filter size parameter, such as a 16, 32, or 64 filter, as well as a stride parameter, a padding parameter, etc. A pooling operation can include max and average pooling operations, which can be applied over different windows depending on a window size parameter. Generally, these parameters are referred to herein as "operation parameters" to distinguish them from "learned parameters," such as weights and bias values, obtained by training a model. Generating a candidate layer can include deterministically or randomly selecting various operations and / or various operation parameters.

[0040]

[0048] In some implementations, the separable convolution can be applied twice. The outputs of each operation are concatenated by an aggregation operation 224, and a 1x1 convolution is used to compute the output of the target layer x out The result is the candidate layer x c is.

[0041]

[0049] Some implementations can constrain the connectivity of candidate layers to reduce the search space. At depth i in a given parent model, there are i-1 potential previous layers, and any subset of the previous layers can be selected as inputs to a candidate layer. As a result, there is an exponential number of choices for inputs to the candidate layer. Some implementations can therefore limit the input choices by learning only repeatable cell modules and expanding cells into manually designed skeleton macrostructures. In these implementations, layers within a cell module can only obtain inputs from other layers within the same cell and from the outputs of the two closest previous cells. Other implementations are not limited to a particular skeleton or connectivity configuration and may allow sparse skip connections between any layers of the model skeleton that change as the model structure grows.

[0042]

[0050] Block 106 of method 100 may include generating one or more candidate layers using the techniques discussed above for each iteration of the method. In some cases, a model structure may include thousands or millions of different layers connected in any number of different combinations, and thus the number of potential candidate layers that can be generated from a single parent model in a single iteration may be large. Below, we describe some techniques that can reduce the computational burden of independently training the entire space of possible child model structures.

[0043] Initializing the candidate layer

[0051] As mentioned above, block 106 of method 100 may also include initializing the candidate layer. Generally, initializing the candidate layer can serve several purposes. First, initializing the parameters of the candidate layer allows the child model to be trained starting from the initialization rather than from scratch. In other words, when the final training occurs in block 110 of method 100, the child model is already partially trained. Second, initializing the candidate layer can provide information about the candidate layer so that it can be selectively added to the parent model while omitting other candidate layers, as discussed further below.

[0044]

[0052] 3 shows an example initialization operation of an implementation that trains a candidate layer 220 by initially preventing the candidate layer from affecting the parent model 210. To do so, operations 221, 222, and 223 can be augmented with a stop-gradient (sg) operation applied to each operation's respective input. sg(x) is x during forward propagation and zero during backpropagation. During backpropagation, gradients are calculated for each layer to adjust the model's learned parameters. The sg operation prevents the gradient from affecting any of the learned parameters in any preceding layers of the parent model.

[0045]

[0053] A stop-forward (sf) operation 302 can be applied to the output of the aggregation operation 224 before the output is provided to the target layer 215, where sf(x) is zero during forward propagation and the identity function during backward propagation. This allows the candidate layer 220 to receive gradient information during initialization without affecting the target layer. Thus, during initialization, the candidate layer can accumulate the gradient of the loss with respect to the target layer without actually affecting the values output by the target layer or any subsequent outputs.

[0046]

[0054] Thus, the initialization of a candidate layer allows the learned parameters of the parent model to remain stable while determining the initialized values of the learned parameters of the candidate layer. In some cases, different candidate layers generated within the same iteration may share edges. In such cases, the weights of the shared edges can be initialized and / or trained independently for each candidate layer. Alternatively, this initialization and / or training can be performed such that different candidate layers share the weights of edges that they have in common. For example, forward propagation and backpropagation operations can be shared during the initialization and / or training of shared edges, while being performed independently for other edges that are not shared between candidate layers. Significant computational cost savings can be achieved by the shared initialization and / or training of edges within a given candidate layer and / or child model.

[0047] Deleting a candidate layer

[0055] In some cases, block 108 of method 100 may include selecting all of the candidate layers generated in block 106 for inclusion in a child model for subsequent training and evaluation. If this is the case, method 100 can still provide significant benefits over conventional techniques because only certain child models are designated as parent models in block 114. As a result, the child models in each iteration are derived from known good parent models. This approach allows for a significant portion of the search space to be saved when growing new child models.

[0048]

[0056] However, depending on the types of potential operations, operation parameters, and connectivity under consideration, the number of possible candidate layers in any given iteration may be very large. Thus, the number of possible direct child models of a given parent may be large. As a further refinement to reduce the computational burden of training new child models, some implementations may filter out certain candidate layers before training the child models in block 110. Such filtering may further reduce the space of child models that need to be trained and evaluated in each iteration, as discussed above.

[0049]

[0057] One approach to reducing the number of children to train is to randomly sample candidate layers, which may be block 108 of method 100, so that fewer children need to be trained. In other implementations, block 108 may include using initialization parameters of the candidate layers to determine which candidate layers to incorporate into the child model for training. Consider a scenario in which multiple candidate layers share connectivity to one or more input and target layers of a given parent model, but perform different operations. Different weights can be initialized for the edges of the candidate layers that are input and / or output by different operations. These initialized weights can be used to select specific candidate layers to include in the child model for further training. Other candidate layers can be removed, so that the child model with those candidate layers is not trained, thus saving extra training time.

[0050]

[0058] Figure 4 illustrates one technique for using initialization parameters to eliminate candidate layers that are unlikely to be useful. In particular, Figure 4 illustrates a scenario in which multiple operations can be initialized together by deriving an aggregate candidate layer 402 from a parent model 210. Generally, the aggregate candidate layer 402 represents multiple candidate layers that each share connectivity to the parent model but perform different operations. As discussed further below, Figure 4 provides an alternative to adding all candidate layers to a parent model to obtain a child model, or to randomly selecting candidate layers. Note that for simplicity, Figure 4 omits layer 212, which was shown in Figures 2 and 3.

[0051]

[0059] A specific algorithm for pruning candidate layers follows. in,i For every x in,i The possible operations on i,1 , ..., op i,k 4, where k is the number of possible operations. In FIG. 4, their inputs are provided by, for example, layers 213 and 214. The input x in,i Given the sum of Jk tensors o1, o2, ..., o JkThese operations can be trained together at the same time. Once trained, we can sum a subset of the operations to get

number

[0052]

[0060] Once trained, the aggregated candidate layer 402 contains different weights for edges connected to different operations. Thus, the aggregated candidate layer can be thought of as multiple initialized candidate layers with different sets of weights for each edge connecting various operations. These weights can be input to a feature selection technique to select one or more of the candidate layers for use in the child model, and similarly, to remove one or more other candidate layers from use in the child model. This feature selection process is represented in FIG. 4 by selection operation 406. Note also that FIG. 4 illustrates the use of transfer stopping operation 408 and gradient stopping operations 410 and 412 in a manner similar to that discussed above with respect to FIG. 3.

[0053]

[0061] One approach to achieving sparse feature selection within the selection operation 406 is to Jk The solution is to linearly combine the weights and use an L-1 norm regularizer, such as lasso regression ("lasso"), on the linear weights. Sparsity is achieved by regularizing the overall loss by:

number

number

number

number

[0054]

[0062] From one perspective, the implementations discussed above use feature selection and learned parameter sharing to initialize several combinations of candidate layers and select a subset of them to include in child models for further training. This approach can favor the selection of candidate layers that are likely to improve model performance and penalize the selection of candidate layers that are unlikely to improve model performance. As a result, fewer child models in total need to be trained for evaluation as potential parent models, and the trained child models are more likely to yield offspring that exhibit the desired performance.

[0055] Training the child model

[0063] After initialization, a given child model may include learned parameters inherited from the parent model, as well as initialization parameters for any candidate layers included in the child model. As discussed above, block 110 of method 100 may include training the child model as a whole, e.g., the candidate layers may be trained together with the rest of the parent model. This may be considered a "warm start" technique, in which the child model initializes the candidate layer parameters at the start of training and inherits the parent model's parameters, both of which may be further modified when training the given child model as a whole.

[0056]

[0064] FIG. 5 shows an example approach for training a child model. In FIG. 5, the gradient stopping operation on the input can be eliminated. Additionally, the forward stopping operation can be replaced with a scalar multiplier 502. The scalar is trainable and can be initialized to zero. Thus, immediately after initialization, the child model represents the same functionality as the parent model. Thus, while the child model has a different structure than the parent model, the functionality of the parent model is preserved. The scalar multiplier can be varied over time, and the child model is trained starting from a combination of the parent's parameters and the initialized candidate parameters, as additional candidate layers gradually begin to contribute to the model's target layer and subsequent output. This approach can prevent candidate layers from destabilizing the learned parameters inherited from the parent model, which may be near-optimal given that the parent model is fully trained.

[0057]

[0065] Note that some implementations may omit the sg and sf operations shown in Figure 3 and instead allow the candidate layer to directly influence the parent model while the candidate layer is being trained. Using sf-sg as a prefix and suffix for the candidate layer as shown in Figure 3 may allow the candidate layer to converge faster. However, this may involve formulating additional objectives for the candidate during initialization.

[0058]

[0066] On the other hand, allowing values to flow freely between the parent and candidate layers allows new candidate layers to directly contribute to adapting the final loss. However, the initial values of the candidate layers may be too far from optimal compared to the model in the parent, which can negatively affect the parameters of the parent model. An alternative approach to address this issue is to use a much smaller learning rate for initializing the candidate layers than that used in training the parent model (0.1-0.02 times the original).

[0059] Evaluate and designate child models as parents

[0067] As previously mentioned, block 114 of method 100 adds a particular child model to the parent model pool. One approach for determining which child models to add to the parent model pool includes using one or more criteria to predict which child models are likely to produce offspring that show improvement in subsequent iterations compared to previously discovered models. Generally, the criteria may consider factors such as the loss of a given child model and the computational effort to train the given child model. A high loss implies low accuracy during model validation, and a high cost implies long training time based on the training dataset. Child models that exhibit relatively low loss and low computational effort may be prioritized for inclusion in the parent model pool.

[0060]

[0068] One particular technique for selecting child models for a parent pool is described herein with respect to FIG. 6. This figure shows an example of a scatter plot 600 of various trained models. For each child model that completes training, the computational cost to train that child model can be calculated and plotted on the x-axis 602. Additionally, the loss of that child model can be calculated and plotted on the y-axis 604. Once all models for a given iteration are plotted, a lower convex hull 606 can be calculated from the plotted values. Note that computational cost can be calculated as a value such as GPU-day, which reflects the time required to train a model on standardized hardware (e.g., a particular model of GPU). In other implementations, computational cost reflects the testing time cost, e.g., the number of operations involved in using a given model to make predictions, regardless of training eigenvalues such as the number of training epochs and / or data augmentation involved in training. As shown in FIG. 6, computational cost can be normalized to a number between 0 and 1.

[0061]

[0069] The lower convex hull 606 can be used as a mechanism for determining whether to add a given child model to the parent model pool. For example, together with the probabilities defined using the following specific algorithms, child models on the lower convex hull can be added to the parent model pool. If m1 and m2 are two adjacent models on the hull with computational costs c1 and c2 (c1 < c2), the probability weight of m1 can be set proportional to c2 - c1. The most accurate model without subsequent models on the curve can be selected with a probability of 0.5 to be included in the parent model pool. In FIG. 6, the most accurate model is model 608 because this model has the lowest loss.

[0062]

[0070] Generally, the lower convex hull is a subset of the Pareto frontier, so another approach is to select child models on the Pareto frontier for inclusion in the parent pool. Any of these approaches can result in excellent performance in selecting child models to add to the parent model pool. One way to consider the lower convex hull and / or the Pareto frontier is as follows. A given model on the lower convex hull or the Pareto frontier cannot be improved with respect to a certain criterion by moving to another model on the lower convex hull / Pareto frontier without degrading other criteria.

[0063]

[0071] Note that due to the randomness in forming the probability gradients, the same model can have different validation errors. As a result, the lower convex hull or the Pareto frontier can be relaxed using a multiplicative bandwidth. Thus, child models whose validation error is within the range of the validation error of the lower convex hull multiplied by (1 + γ) at the same computational cost can be considered to be on the lower convex hull and can be selected as parents. Some implementations can set γ = 0.025. This approach allows certain child models that are close to but not exactly on the lower convex hull to still be designated as parent models.

[0064]

[0072] Other techniques may be used to allow child models having positions within a predetermined neighborhood of the lower convex hull to be selected as parent models. For example, some implementations may define a threshold distance from the lower convex hull and allow child models within the threshold distance of the lower convex hull to be selected as parent models. This is just one of a variety of techniques that may be used to select a subset of one or more child models as parent models based on one or more criteria.

[0065]

[0073] Figure 6 shows models that have completed training as black dots. For purposes of illustration, assume that Figure 6 represents the state of scatter plot 600 after iteration N. One or more of the child models on or near lower convex hull 606 can be selected as parent models for the subsequent iteration N+1, in which another candidate layer can be added and initialized to form further child models as discussed above.

[0066]

[0074] Figure 7 shows the scatter plot 600 at a subsequent state after iteration N+1. The child model being trained during iteration N+1 is shown as a square in Figure 7. A new lower convex hull 702 can be calculated. The previous lower convex hull 606 is shown as a dotted line to indicate the downward movement of the lower convex hull in iteration N+1.

[0067]

[0075] Again, one or more of the child models within or near lower convex hull 702 can be selected for the subsequent iteration N+2. The child models trained during iteration N+2 are shown as triangles in Figure 8. A new lower convex hull 802 can be calculated, with the previous lower convex hulls 606 and 702 shown as dotted lines to indicate their position relative to lower convex hull 802.

[0068]

[0076] One way to view the approach shown in Figures 6-8 is as a greedy method for finding a cost-effective predictor. Note that this is a multi-objective approach that considers both the computational cost of training / validation as well as the accuracy of the model. Alternative implementations may use different and / or additional criteria, such as multi-dimensional plots of three or more criteria, objective functions defined across one or more criteria, etc.

[0069]

[0077] The above techniques generally use randomized techniques to grow networks. However, instead of a purely random technique, which may be computationally infeasible, the techniques are guided by predictive criteria that tend to favor using known good models as a basis for further refinement. As mentioned previously, training a model from scratch can be very computationally intensive. For example, a training dataset may contain millions of training data items, and a given model may need to be trained for several training epochs before convergence. A training epoch may include one forward propagation and one backpropagation operation through all models for each data item in the training dataset.

[0070]

[0078] The above approach offers various benefits over conventional approaches for automated model generation. Note that not all child models are used to derive candidate layers for subsequent training. Rather, by using a subset of child models that occur along the lower convex hull as parent models, the disclosed implementation begins each new iteration with a child model structure that inherits the structure of the parent model of a known good model. Starting an iteration in this manner allows subsequent iterations to proceed without training models that occupy a significant portion of the search space far from the lower convex hull, saving a significant amount of training time. Additionally, by using training cost as well as accuracy as a criterion for selecting which child model to use as the new parent model, the disclosed implementation avoids generating new, more computationally intensive models. This in turn reduces training time for those models as well as for training child models derived from such models.

[0071]

[0079] Additionally, recall that initialization of the candidate layer can be performed on a child model that inherits not only the structure of the parent model but also the learned parameters of the parent model. As a result, the parameters of the new candidate layer can be initialized to reasonable values relatively quickly. Furthermore, this configuration allows for "warm-start" training of the child models as a whole. Warm-start training can speed convergence, for example, by requiring far fewer training epochs or training samples than if each child model were fully trained from scratch.

[0072]

[0080] Additionally, recall that shared edges between any two child models can be initialized and trained together. Given that the child models are derived from a shared pool of parent models, the likelihood of shared edges is relatively high. This can save significant training time compared to the alternative of initializing and training shared edges separately for each child model.

[0073]

[0081] Recall that past techniques for automatically generating machine learning models have either suffered from computationally infeasible training burdens or considered only models within a very restricted search space. Taken together, the benefits described above enable the automatic generation of large, accurate machine learning models from a relatively unconstrained search space without imposing an excessive computational burden on the computational resources used to generate the models.

[0074] Processing flow example

[0082] 9 shows a high-level overview of a process flow 900 for incrementally growing a model structure. The process flow illustrates how a parent model pool can be iteratively grown. The process flow is an example of an approach for assigning specific tasks to worker processes, such as virtual machines. For example, this process flow may be suitable for a distributed system or server farm where a hypervisor or operating system gradually schedules new jobs as worker processes become available.

[0075]

[0083] At step 910, a parent model pool 912 is determined. For example, in some cases, a human or automated technique may select one or more models for the pool based on the known performance of the models. For simplicity, assume that the pool contains a single model 914 at initialization.

[0076]

[0084] At step 920, candidate layers are initialized from the parent model. For example, FIG. 9 shows candidate layers 922, 924, and 926. As previously described, each candidate layer can be added to the parent model pool 912. The candidate layer can be added to a candidate layer queue 928. When a given worker process becomes available, the worker process can take a candidate layer from the candidate layer queue and initialize it by, for example, performing a few initial iterations of training as discussed herein.

[0077]

[0085] Once initialized, a subset of candidate layers can be selected for inclusion in the child model queue 932 for training in stage 930. An available worker process can retrieve a given child model from the child model queue for training in stage 930. Figure 9 shows one such child model 934 in stage 930 as including candidate layer 924 but not candidate layers 922 or 926. This illustrates the idea that candidate layers 922 and / or 926 have been removed in a previous stage 920, as discussed, for example, in the feature selection implementation referenced above.

[0078]

[0086] As mentioned above, the child models are not necessarily trained from scratch, but can be trained starting from the learned parameters of the parent model and, if initializing candidate layers to add to the child model, from the initialization parameters determined in step 920. Furthermore, as also discussed above, it should be noted that multiple child models can be trained together as part of a larger model.

[0079]

[0087] A particular child model may be selected from the child model queue 932 at step 930 for addition to the parent model pool 912, as indicated by arrow 936. As discussed above, child models to add to the parent model pool may be selected based on various criteria. For example, some implementations may evaluate trained child models based on a validation dataset, determine a respective loss for each child model, and use the loss as the criterion for adding the child model to the parent model pool. Alternatively and / or additionally, the computational cost of training a given child model may be used as the criterion for determining whether to add that child model to the parent model pool. Other approaches may utilize a formal calculation of the net expected value of adding a child model to the parent model pool based on considering the computational cost and predictions calculated regarding the expected value of the addition. Once the parent model pool is updated, processing may return to step 910 for subsequent iterations.

[0080] System Example

[0088] Implementations herein may be performed on a variety of devices and in a variety of scenarios. Figure 10 shows an example of a system 1000 within which implementations herein may be used, as discussed further below.

[0081]

[0089] As shown in Figure 10, system 1000 includes client device 1010, server 1020, server 1030, and client device 1040 connected by one or more networks 1050. Note that the client devices can be implemented as both mobile devices, such as smartphones or tablets, and fixed devices, such as desktop and server devices. Similarly, the servers can be implemented using various types of computing devices. In some cases, any of the devices shown in Figure 10, but among other things, the servers can be implemented by a data center, server farm, etc.

[0082]

[0090] 10 may be referred to herein by reference numbers in parentheses. In the following description, a parenthetical (1) indicates the occurrence of a given component on client device 1010, a parenthetical (2) indicates the occurrence of a given component on server 1020, a parenthetical (3) indicates the occurrence on server 1030, and a parenthetical (4) indicates the occurrence on client device 1040. Unless a specific instance of a given component is identified, the component will be referred to generally herein without parentheses.

[0083]

[0091] Generally, devices 1010, 1020, 1030, and / or 1040 may have individual processing resources 1001 and storage resources 1002, which are discussed in more detail below. The devices may also have various modules that function using the processing and storage resources to perform the techniques discussed herein. The storage resources may include both persistent storage resources, such as magnetic drives or solid-state drives, and volatile storage, such as one or more random access memory devices. In some cases, modules are provided as executable instructions that are stored on persistent storage, loaded into a random access memory device, and retrieved from the random access memory by the processing resources for execution.

[0084]

[0092] The client device 1010 may include a configuration module 1011 that can interact with a model generation module 1021 on the server 1020. Generally speaking, the configuration module may provide certain configuration parameters to the model generation module. The model generation module uses those configuration parameters to perform the model generation functions discussed herein. Specifically, the model generation module may perform method 100 based on the configuration parameters. As described above, the iterative model growth process may include modifying a parent model to obtain child models, selecting candidate layers to include in the child models based at least on weights learned in an initialization process of the candidate layers, and outputting a final model selected from the child models. The model generation module may include various sub-modules (not shown) related to each block of method 100.

[0085]

[0093] The model generation module 1021 may output the final model to the server 1030. A model execution module 1031 may execute the final model in response to received input. For example, a recording module 1041 on the client device 1040 may record data such as images or audio for processing by the final model, and a local application 1042 may upload the recorded data to the server 1030 for processing. The model execution module may process the uploaded data using the final model and provide the output of the model to the local application for further processing.

[0086] Graphical Interface Example

[0094] As discussed above, the configuration module 1011 on the client device 1010 can provide initial configuration parameters to the model generation module 1021. The model generation module 1021 can perform the method 100 according to the configuration parameters provided by the configuration module. Figure 11 shows an example of a configuration graphical user interface ("GUI") 1100 that can be presented on the client device 1010 for a user to define these configuration parameters.

[0087]

[0095] The parent model element 1101 allows the user to specify what type of initial parent model should be used to begin the model growth process. FIG. 11 shows the user selecting a default parent model. For example, the model generation module 1021 can provide a default neural network structure for use as a general-purpose initial parent model. Other options may include a randomly generated model, in which the model generation module selects a random model structure for use as the initial parent. Another option is for the user to navigate to an existing model known to provide relatively good performance for a particular task. In this case, the configuration module 1011 can upload the specified model to the model generation module for use as the initial parent model.

[0088]

[0096] The operation element 1102 allows a user to specify which types of operations are considered by the model generation module 1021. For example, the model generation module may provide various options for enumerated operation groups that can be selected for a candidate layer. A first option, Group 1, may define two operations: a single convolution operation (e.g., 5x5) and a single pooling operation (e.g., max pooling). A second option, Group 2, may define two convolution operations, e.g., 5x5 and 7x7, and two pooling operations, average pooling and max pooling, for a total of four possible operations. A third option, Group 3, may define 1x1, 3x3, 5x5, and 7x7 convolutions, average and max pooling, and a match operation, for a total of seven possible operations. The listed operations may have predetermined or selectable operation parameters, such as adjustable window size, stride, etc.

[0089]

[0097] The budget input element 1103 allows a user to specify a computational budget for generating a model. For example, a user may specify a budget of 10,000 GPU-days, and the model generation module 1021 may use that budget as a stopping condition at decision block 116 of the method 100. Alternative implementations may use other metrics as the computational budget, such as the number of processing operations, the number of virtual machines, time, etc.

[0090]

[0098] Criteria 1 element 1104 allows the user to specify a first criterion for evaluating the model, and Criteria 2 element 1105 allows the user to specify a second criterion. In FIG. 11 , these criteria are shown as computational cost and loss, respectively, as discussed above. However, the user may want to specify other criteria, such as the overall size of the model structure, the number of connections, or the total number of learnable parameters of the model. In resource-constrained applications, such as running models on mobile or Internet of Things (IoT) devices, the size of the model in bytes may be an important criterion, as these devices tend to have limited storage and / or memory capacity. In addition, these devices may have constrained processing capabilities that can lead to user perception of latency when a given model takes a long time to run, so the computation time to run the model may also be a useful criterion in those scenarios.

[0091]

[0099] Note that the configuration parameters shown in Figure 11 are merely exemplary, and various other implementations are possible. For example, in some cases, the user may specify connectivity parameters. As one example, the user may specify that a candidate layer receives input from a specified number of previous layers, or from a variable (e.g., random) number of previous layers. As another example, the user may specify whether skip connections (e.g., a layer may not provide input to the immediately following layer, but instead connects to another layer following it, skipping the immediately following layer) are allowed. The user may also specify a densenet architecture in which each layer is connected to all preceding layers in the model.

[0092]

[0100] Further, it should be noted that some implementations may provide one or more GUIs to show the progress of method 100. For example, some implementations may generate a GUI that shows a scatter plot 600 that changes over various iterations of model growth, in a manner similar to that shown in Figures 6-8. Other implementations may display graphical representations of individual models and / or candidate layers as they are generated.

[0093] Application Examples

[0101] The techniques discussed herein can be used in a variety of applications without limitation. Nevertheless, the following provides some specific examples for illustrative purposes.

[0094]

[0102] As a first example, suppose an entity wants to market an application that tags all of a user's friends in the user's photo collection. The entity may have an existing model that it currently uses for that purpose, which may be executed by a model execution module 1031 on a server 1030 against photos provided by a local application 1042 on a client device 1040. However, the entity may feel that the existing model is not accurate enough and may provide a poor user experience in some circumstances.

[0095]

[0103] First, the entity can upload an existing model to the model generation module 1021 on the server 1020 and can configure various initial parameters as discussed above. The model generation module can then iteratively grow the existing model until a stopping condition is reached and export the final model to the server 1030. The server 1030 can replace the initial model with the received final model and continue tagging the user's photos as discussed.

[0096]

[0104] As one alternative, the client device can provide pre-recorded audio data for transcription by the final model. As another alternative, the client device can provide text in a first language for translation into a second language by the final model. As further examples, the final model can perform scene segmentation on video, object detection (e.g., identifying faces, vehicles, etc.) on images or videos, grammar parsing on text, etc.

[0097]

[0105] As yet another example, the final model may be used to map documents and queries to semantic vectors, in which case the server 1030 may provide a search engine function that uses vector distances between indexed documents and incoming queries to rank search results for users of client devices 1040. More broadly, the final model may perform any suitable task for which training data is available, including but not limited to classification, machine translation, and pattern recognition tasks.

[0098]

[0106] Additionally, some implementations may allow an entity to provide a dataset for training, validation, and / or testing. For example, a first entity may send a dataset from a client device 1010 to a second entity controlling the server 1020. The second entity may generate a final model using the dataset provided by the first entity. In some cases, the second entity may provide the final model to the first entity for execution on the server 1030 by the first entity, for example. In other cases, the second entity does not provide the model itself, but instead allows the first entity to invoke the model. In this case, the second entity may control both the server 1020 and the server 1030 on the same device and / or implement both model generation and execution.

[0099]

[0107] Additionally, it should be noted that the term "final model" is used herein merely to imply that a given child model is specified for practical use within an application. In some implementations, the final model output and used by an entity may also be used as the initial parent model for subsequent iterations of the model growing process described herein. In some cases, while an entity is using a given final model, new data may be acquired and the new data may be used as training / validation / test data for subsequent iterations of the process. Such use of new data may result in a new final model being output, which may be used for some time as further new data is acquired, and this process may be repeated continually over the life of a given practical application targeted by a given entity.

[0100]

[0108] Additionally, it should be noted that the disclosed implementations have focused on examples in which candidate layers are added to a model structure. However, in some cases, other types of candidate components may be used instead of layers. In general, candidate components may be implemented using any type of mathematical and / or logical function, ranging from simple arithmetic or Boolean operators to more complex trainable modules.

[0101] Device implementation

[0109] 10, system 1000 includes several devices, including client device 1010, server 1020, server 1030, and client device 1040. As also noted, not all implementations of the devices can be illustrated, and other implementations of the devices will be apparent to those skilled in the art from the above and below description.

[0102]

[0110] As used herein, the terms "device," "computer," "computing device," "client device," and / or "server device" may refer to any type of device having some amount of hardware processing power and / or hardware storage / memory capability. The processing power may be provided by one or more hardware processors (e.g., hardware processing units / cores) capable of executing data in the form of computer-readable instructions to provide functionality. The computer-readable instructions and / or data may be stored on a storage location, such as a memory / memory and / or data store. As used herein, the term "system" may refer to a single device, multiple devices, etc.

[0103]

[0111] Storage resources may be internal or external to the respective devices involved. Storage resources may include any one or more of volatile or non-volatile memory, hard drives, flash storage, and / or optical storage (e.g., CDs, DVDs, etc.), among others. As used herein, the term "computer-readable medium" may include signals. In contrast, the term "computer-readable storage medium" excludes signals. Computer-readable storage media includes "computer-readable storage devices." Examples of computer-readable storage devices include volatile storage media such as RAM, and non-volatile storage media such as hard drives, optical disks, and flash memory, among others.

[0104]

[0112] In some cases, a device is configured with a general-purpose hardware processor and storage resources. In other cases, a device may include a system-on-chip (SOC) type design. In an SOC design implementation, the functionality provided by the device may be integrated on a single SOC or multiple combined SOCs. An associated processor or processors may be configured to interface with shared resources, such as memory, storage, and / or one or more dedicated resources, such as hardware blocks configured to perform a specific function. Thus, as used herein, the terms “processor,” “hardware processor,” or “hardware processing unit” may also refer to a central processing unit (“CPU”), a graphical processing unit (“GPU”), a controller, a microcontroller, a processor core, or other type of processing device suitable for implementation in both traditional computing architectures as well as SOC designs.

[0105]

[0113] Alternatively, or in addition, the functionality described herein may be performed at least in part by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0106]

[0114] In some configurations, any of the modules / code discussed herein may be implemented by software, hardware, and / or firmware. In either case, the modules / code may be provided during the manufacture of the device or by an intermediary preparing the device for sale to an end user. In other examples, the end user may install the modules / code later, such as by downloading executable code and installing the executable code on the corresponding device.

[0107]

[0115] It should also be noted that a device may generally have input and / or output capabilities. For example, a computing device may have a variety of input mechanisms, such as a keyboard, a mouse, a touchpad, voice recognition, gesture recognition (e.g., using a depth camera such as a stereoscopic or time-of-flight camera system, an infrared camera system, an RGB camera system, or using an accelerometer / gyroscope, facial recognition, etc.) The device may also have a variety of output mechanisms, such as a printer, a monitor, etc.

[0108]

[0116] It should also be noted that the devices described herein may function in an independent or cooperative manner to implement the described techniques. For example, the methods and functions described herein may be performed on a single computing device and / or distributed across multiple computing devices communicating over network 1050. Without limitation, network 1050 may include one or more local area networks (LANs), wide area networks (WANs), the Internet, etc.

[0109] Further examples

[0117] Various device examples are described above. Further examples are described below. One example includes a method executed on a computing device, which may include performing two or more iterations of an iterative model growing process. The iterative model growing process may include selecting a particular parent model from a parent model pool of one or more parent models, generating multiple candidate layers, and initializing the multiple candidate layers while reusing learned parameters of the particular parent model. The iterative model growing process may also include selecting specific candidate layers for inclusion in child models for training, each of which may include the particular parent model and one or more of the particular candidate layers. The iterative model growing process may also include training multiple child models to obtain trained child models, evaluating the trained child models using one or more criteria, and designating each trained child model as a new parent model and adding the new parent model to the parent model pool based at least on the evaluation. The method may also include selecting at least one trained child model as a final model after the two or more iterations and outputting the final model.

[0110]

[0118] Further examples may include any of the above and / or below examples, where the method further includes determining a computational cost of training or testing the child model, and using the computational cost as a first criterion when evaluating the trained child model.

[0111]

[0119] Further examples may include any of the above and / or below examples, where the method further includes determining a loss associated with the trained child model and using the loss as a second criterion when evaluating the trained child model.

[0112]

[0120] Another example may include any of the above and / or below examples, wherein the method further includes plotting the child model on a graph having a first axis reflecting computational cost and a second axis reflecting loss, and selecting a new parent model based at least on a corresponding position of the new parent model on the graph.

[0113]

[0121] Further examples may include any of the above and / or below examples, wherein the method further includes determining at least one of a lower convex hull or a Pareto frontier on the graph, and selecting a new parent model based at least on a proximity of the new parent model to the lower convex hull or the Pareto frontier.

[0114]

[0122] Further examples may include any of the above and / or below examples, where the selecting includes identifying a subset of trained child models that fall within a predetermined neighborhood of the lower convex hull or Pareto frontier, determining individual probabilities for the subset of trained child models, and selecting a new parent model based at least on the individual probabilities.

[0115]

[0123] Other examples may include any of the above and / or below examples, where generating each candidate layer includes selecting a target layer from the particular parent model to receive the output of each candidate layer, selecting one or more input layers from the particular parent model to provide inputs to each candidate layer, and selecting particular operations that each candidate layer should perform on the inputs.

[0116]

[0124] Other examples may include any of the above and / or below examples, where selecting a particular operation includes defining a set of operations and randomly selecting a particular operation from the set of operations.

[0117]

[0125] Other examples may include any of the above and / or below examples, wherein the method further includes randomly selecting the target layer and at least one input layer from the particular parent model.

[0118]

[0126] Other examples may include any of the above and / or below examples, where the final model is a neural network.

[0119]

[0127] Another example includes a system including a hardware processing unit and a storage resource storing computer-readable instructions that, when executed by the hardware processing unit, cause the hardware processing unit to perform an iterative model growth process that includes modifying a parent model to obtain a child model. The iterative model growth process may include selecting candidate layers for inclusion in the child model based at least on weights learned in an initialization process of the candidate layers. The computer-readable instructions may further cause the hardware processing unit to output a final model selected from the child models.

[0120]

[0128] Further examples may include any of the above and / or below examples, where the computer readable instructions, when executed by a hardware processing unit, cause the hardware processing unit to generate different candidate layers that share connectivity to a parent model and perform different operations, and to jointly initialize the different candidate layers to obtain different weights for the different candidate layers.

[0121]

[0129] Further examples may include any of the above and / or below examples, wherein the computer readable instructions, when executed by a hardware processing unit, cause the hardware processing unit to select each candidate layer for inclusion in the child model based at least on different weights of the different candidate layers.

[0122]

[0130] Further examples may include any of the above and / or below examples, where the computer readable instructions, when executed by a hardware processing unit, cause the hardware processing unit to perform a feature selection technique on different weights of different candidate layers to select each candidate layer for inclusion in a child model.

[0123]

[0131] Further examples may include any of the above and / or below examples, wherein the feature selection technique includes lasso regression (LASSO).

[0124]

[0132] Further examples may include any of the above and / or below examples, where the various operations include convolution and pooling operations.

[0125]

[0133] Further examples may include any of the above and / or below examples, where the computer readable instructions, when executed by a hardware processing unit, cause the hardware processing unit to: train a final model using training data for at least one classification, machine translation, or pattern recognition task; and provide the final model for execution, wherein the final model is adapted to perform the at least one classification, machine translation, or pattern recognition task.

[0126]

[0134] Another example includes a computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to perform operations including performing two or more iterations of an iterative model growing process. The iterative model growing process may include selecting a particular parent model from a parent model pool of one or more parent models, initializing a plurality of candidate layers, selecting a plurality of child models for training, each child model inheriting a structure of the particular parent model and including at least one of the candidate layers, training the plurality of child models to obtain trained child models, and designating each trained child model as a new parent model based at least in part on one or more criteria and adding the new parent model to the parent model pool. The operations may further include selecting at least one trained child model as a final model after the two or more iterations and outputting the final model.

[0127]

[0135] Further examples may include any of the above and / or below examples, wherein the operations further include simultaneously initializing a plurality of candidate layers to obtain initialization parameters, and selecting each candidate layer for inclusion in a respective child model based at least on the initialization parameters.

[0128]

[0136] Further examples may include any of the above and / or below examples, wherein the operations further include randomly selecting operations from the listed group of operations for inclusion in the plurality of candidate layers.

[0129] conclusion

[0137] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims, and other features and acts recognized by those skilled in the art are intended to be within the scope of the claims.

Claims

1. A method executed on a computing device, the method comprising: outputting a graphical user interface having a first graphical element for specifying an operation search space of operations available to be performed by the candidate layer in the iterative model growing process; receiving a first user input via the first graphical element, the first user input identifying a set of operations for inclusion in the operation search space; Adding an initial parent model to the parent model pool; performing two or more iterations of the iterative model growing process, the iterative model growing process comprising: choosing a selected parent model from said parent model pool; inserting a plurality of candidate layers into the selected parent model, and performing initialization and training of the plurality of candidate layers while the plurality of candidate layers are connected to the selected parent model to obtain learned weights of the plurality of candidate layers, wherein the plurality of candidate layers are initialized while maintaining the weights of the plurality of layers of the selected parent model, and each candidate layer is configured to perform an individual operation selected from the set of operations identified by the first user input received by the first graphical element of the graphical user interface; selecting fewer than all layers of the plurality of candidate layers as selected candidate layers for inclusion in each child model of a plurality of child models for subsequent training based at least on the learned weights of the plurality of candidate layers, each child model including the plurality of layers of the selected parent model and one or more of the selected candidate layers; training the plurality of child models with the one or more selected candidate layers to obtain a trained child model; evaluating the trained child model using one or more criteria; and designating each trained child model as a new parent model based at least on said evaluating and adding said new parent models to said parent model pool. including, carrying out, and selecting at least one trained child model as a final model after the two or more iterations and outputting the final model; A method comprising:

2. receiving a second user input by a second graphical element of the graphical user interface, the second user input specifying a default model, specifying a randomly generated model, or navigating to an existing model for use as the initial parent model; The method of claim 1 further comprising:

3. 2. The method of claim 1 , wherein the first user input received by the first graphical element of the graphical user interface selects at least two different convolution operations, and the individual operations performed by the respective candidate layers are randomly selected from the operation search space.

4. 2. The method of claim 1 , wherein the first user input received by the first graphical element of the graphical user interface selects at least two different pooling operations, and the individual operations performed by the respective candidate layers are randomly selected from the operation search space.

5. receiving a second user input directed to a second graphical element of the graphical user interface, the second user input identifying a specified amount of computational resources to use in the iterative model growing process; and responsive to consuming the specified amount of computational resources, stopping the iterative model growing process and selecting the final model. The method of claim 1 further comprising:

6. The method of claim 5 , wherein the second user input directed to the second graphical element specifies a number of GPU-days to consume for the iterative model growing process.

7. The method of claim 5 , wherein the second user input directed to the second graphical element specifies an amount of time to consume for the iterative model growing process.

8. receiving a second user input directed to a second graphical element of the graphical user interface, the second user input directed to the second graphical element identifying a model size as a particular criterion for evaluating the trained child model; and designating each of the trained child models as the new parent model based at least on a model size of each of the trained child models. The method of claim 1 further comprising:

9. receiving a second user input directed to a second graphical element of the graphical user interface, the second user input directed to the second graphical element specifying a parameter of connectivity for the child model; and generating the child model according to the connectivity parameters; The method of claim 1 further comprising:

10. 10. The method of claim 9, wherein the connectivity parameter specified by the second user input directed to the second graphical element indicates the number of previous layers to be used as input to each added candidate layer of the child model.

11. 10. The method of claim 9, wherein the connectivity parameter specified by the second user input directed to the second graphical element indicates whether skip connections are used in the child model.

12. a hardware processing unit; and A storage resource that stores computer-readable instructions wherein the computer-readable instructions, when executed by the hardware processing unit, receiving a first user input via a first graphical element of a graphical user interface, the first user input identifying a set of operations for inclusion in an operation search space for an iterative model growth process, the set of operations available to be executed by candidate layers added during the iterative model growth process; Adding an initial parent model to the parent model pool; performing two or more iterations of the iterative model growing process, the iterative model growing process comprising: choosing a selected parent model from said parent model pool; inserting a plurality of candidate layers into the selected parent model, and performing initialization and training of the plurality of candidate layers while the plurality of candidate layers are connected to the selected parent model to obtain learned weights of the plurality of candidate layers, wherein the plurality of candidate layers are initialized while maintaining the weights of the plurality of layers of the selected parent model, and each candidate layer is configured to perform an individual operation selected from the set of operations identified by the first user input received by the first graphical element of the graphical user interface; selecting fewer than all layers of the plurality of candidate layers as selected candidate layers for inclusion in each child model of a plurality of child models for subsequent training based at least on the learned weights of the plurality of candidate layers, each child model including the plurality of layers of the selected parent model and one or more of the selected candidate layers; training the plurality of child models with the one or more selected candidate layers to obtain a trained child model; evaluating the trained child model using one or more criteria; and designating each trained child model as a new parent model based at least on said evaluating and adding said new parent models to said parent model pool. including, carrying out, and selecting at least one trained child model as a final model after the two or more iterations and outputting the final model; The system causes the hardware processing unit to perform the following:

13. The system of claim 12 , wherein the operation search space includes a plurality of convolution operations and a plurality of pooling operations specified by the first user input received by the first graphical element of the graphical user interface.

14. The computer-readable instructions, when executed by the hardware processing unit, configuring each candidate layer added to the selected parent model to perform an individual operation randomly selected from the set of operations identified by the first user input. The system of claim 12 , further comprising:

15. 15. The system of claim 14, wherein the first user input directed to the first graphical element identifies at least two different convolution operations and at least two different pooling operations, the individual operations being randomly selected.

16. 16. The system of claim 15, wherein the first user input directed to the first graphical element specifies at least two different window sizes and at least two different strides for the at least two different convolution operations.

17. A computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to perform operations, including: outputting a graphical user interface having a first graphical element for specifying an initial parent model and a second graphical element for specifying an operational search space for the iterative model growth process; receiving a first user input via the first graphical element, the first user input identifying a particular machine learning model as the initial parent model; receiving a second user input via the second graphical element, the second user input identifying a set of operations for inclusion in the operation search space; configuring the operation search space of the iterative model growing process to be limited to the set of operations identified by the second user input received by the second graphical element of the graphical user interface; adding the initial parent model to a parent model pool; performing two or more iterations of the iterative model growing process, the iterative model growing process comprising: choosing a selected parent model from said parent model pool; inserting a plurality of candidate layers into the selected parent model, and performing initialization and training of the plurality of candidate layers while the plurality of candidate layers are connected to the selected parent model to obtain learned weights of the plurality of candidate layers, wherein the plurality of candidate layers are initialized while maintaining the weights of the plurality of layers of the selected parent model, and each candidate layer is configured to perform an individual operation selected from the set of operations of the operation search space identified by the second user input received by the second graphical element of the graphical user interface; selecting fewer than all layers of the plurality of candidate layers as selected candidate layers for inclusion in each child model of a plurality of child models for subsequent training based at least on the learned weights of the plurality of candidate layers, each child model including the plurality of layers of the selected parent model and one or more of the selected candidate layers; training the plurality of child models with the one or more selected candidate layers to obtain a trained child model; evaluating the trained child model using one or more criteria; and designating each trained child model as a new parent model based at least on said evaluating and adding said new parent models to said parent model pool. including, carrying out, and selecting at least one trained child model as a final model after the two or more iterations and outputting the final model; 1. A computer-readable storage medium comprising:

18. 2. The method of claim 1 , wherein the first user input received by the first graphical element of the graphical user interface selects a group of at least two different convolution kernel sizes available for execution by the candidate layer when performing the iterative model growing process.

19. 2. The method of claim 1 , wherein the first user input received by the first graphical element of the graphical user interface selects a set of at least two different convolution stride sizes available to be performed by the candidate layer when performing the iterative model growing process.

20. 2. The method of claim 1 , wherein the first user input received by the first graphical element of the graphical user interface selects a group of pooling operations available to be performed by the candidate layer when executing the iterative model growing process, the group including at least a max pooling operation and an average pooling operation.

21. 2. The method of claim 1 , wherein the first user input received by the first graphical element of the graphical user interface selects a group of at least two different pooling window sizes available for execution by the candidate layer when performing the iterative model growing process.

Images