Automatic generation of machine learning models

Abstract

The present disclosure relates to automated generation of machine learning models, such as neural networks. An example system includes a hardware processing unit and a storage resource. The storage resource may store computer-readable instructions that cause the hardware processing unit to perform an iterative model growing process that includes modifying a parent model to obtain a child model. The iterative model growing process may also include selecting candidate layers to include in the child model based at least on weights learned during an initialization process for the candidate layers. The system may also output a final model selected from the child models.

Description

Background Art

[0001] Traditionally, machine learning models are manually constructed by experts who define the model's structure and then use automated techniques to train the model. As machine learning models have become increasingly complex, various attempts have been made to automate the process of generating machine learning models. However, these efforts have achieved limited success. Summary of the Invention

[0002] This Summary is provided to introduce some concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used as an aid in understanding the scope of the claimed subject matter.

[0003] This disclosure generally relates to techniques for automated generation of machine learning models. One example includes a method or technique that can be performed on a computing device. The method or technique can include performing two or more iterations of an iterative model growing process. The iterative model growing process can include selecting a specific parent model from a pool of one or more parent models, generating multiple candidate layers, and initializing the multiple candidate layers while reusing learned parameters and / or structure of the specific parent model. The iterative model growing process can also include selecting specific candidate components, such as layers, to include in child models for training. Each child model can also include a specific parent model and one or more specific candidate layers from a specific candidate layer, or other structure. The iterative model growing process can also include training multiple child models to obtain trained child models and evaluating the trained child models using one or more criteria. The iterative model growing process can also include designating an individual trained child model as a new parent model based at least on the evaluation, and adding the new parent model to the pool of parent models. The method or technique can also include, after two or more iterations, selecting at least one trained child model as a final model and outputting the final model.

[0004] Another example includes a system comprising a hardware processing unit and a storage resource. The storage resource may store computer-readable instructions that, when executed by the hardware processing unit, cause the hardware processing unit to perform an iterative model growing process involving modifying a parent model to obtain a child model. The iterative model growing process may select candidate layers to include in the child model based at least on weights learned during an initialization process for the candidate layers. The computer-readable instructions may also cause the hardware processing unit to output a final model selected from the child models.

[0005] Another example includes a computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to perform an action. The action may include performing two or more iterations of an iterative model growing process. The iterative model growing process may include selecting a particular parent model from a parent model pool consisting of one or more parent models, initializing multiple candidate layers, and selecting multiple child models for training. Each child model may include a structure inherited from a particular parent model and at least one candidate layer from the candidate layers. The iterative model growing process may also include training multiple child models to obtain trained child models, and designating individual trained child models as new parent models based on at least one or more criteria. The iterative model growing process may also include adding a new parent model to the parent model pool. The action also includes selecting at least one trained child model as a final model after two or more iterations, and outputting the final model.

[0006] The examples listed above are intended to provide a quick reference to aid the reader and are not intended to define the scope of the concepts described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The detailed description is described with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. Like reference numbers are used in different instances in the description and the drawings to indicate the same or similar items.

[0008] Figure 1 Illustrated are example methods or techniques for automated generation of machine learning models according to some implementations of the presently disclosed concepts.

[0009] Figure 2 Illustrated are example methods for generating candidate layers for a machine learning model according to some implementations of the presently disclosed concepts.

[0010] Figure 3 Illustrated are example methods for initializing candidate layers of a machine learning model according to some implementations of the presently disclosed concepts.

[0011] Figure 4 Another example method for initializing candidate layers of a machine learning model, according to some implementations of the presently disclosed concepts, is illustrated.

[0012] Figure 5 An example method for training a sub-model according to some embodiments of the presently disclosed concepts is illustrated.

[0013] Figures 6 to 8 Illustrated are scatter plots associated with successive iterations of an iterative model growing process, in accordance with some embodiments of the disclosed concepts.

[0014] Figure 9An example process flow for automated generation of machine learning models according to some implementations of the presently disclosed concepts is illustrated.

[0015] Figure 10 An example system according to some implementations of the presently disclosed concepts is illustrated.

[0016] Figure 11 An example graphical user interface according to some implementations of the presently disclosed concepts is illustrated. DETAILED DESCRIPTION

[0017] Overview

[0018] There are various types of machine learning frameworks that can be trained using supervised and / or unsupervised learning. Support vector machines, decision trees, and neural networks are some of the machine learning frameworks suitable for supervised learning, where the model learns from labeled training data. Some machine learning frameworks, such as neural networks, use layers of nodes that perform specific operations.

[0019] In a neural network, nodes are connected to each other via one or more edges. A neural network can include an input layer, an output layer, and one or more intermediate layers. According to predefined functions, individual nodes can process their respective inputs and provide outputs to subsequent layers, or in some cases provide outputs to a previous layer. A given node input can be multiplied by the corresponding weight value of the edge between the input and the node. In addition, a node can have an individual bias value, which is used to generate an output. Various training programs can be used to learn the weights and / or bias values ​​of the edges. For the purposes of this disclosure, the term "learning parameters" refers to parameters learned by training a hierarchical machine learning model (e.g., a neural network), such as edges and bias values.

[0020] Neural network structures can be constructed in a modular manner. For example, one or more node layers can work together to perform a specific operation, such as a pooling operation or a convolution operation. Different layers can then be connected together to form the entire network structure. For the purposes of this disclosure, the term "layer" refers to a group of nodes that share connectivity with one or more input layers and one or more target layers that receive input from nodes in the layer. The term "operation" refers to a function that can be performed by one or more node layers. The term "model structure" refers to the overall architecture of a hierarchical model, including the number of layers, the connectivity of the layers, and the types of operations performed by individual layers. The term "neural network structure" refers to the model structure of a neural network. The disclosed embodiments primarily use neural network structures as example model structures for hierarchical machine learning models. The term "trained model" refers to the model structure and the learning parameters for the model structure. Note that two trained models can share the same model structure and still have different learning parameters, for example, the two models are trained on different training data, or there is an underlying random process during the training process.

[0021] As mentioned above, the method of generating a model structure is for humans to manually define the model structure. The model structure can then be trained on some training data sets set by a computer to obtain a trained model, and a validation data set can then be used to validate the trained model. Subsequently, modifications to the model structure can be manually generated, for example, by adding or removing layers or connections between layers. Afterwards, the modified data can be trained again to reacquire additional trained models, which can be compared with each other to select the final model and the corresponding structure that is well suited for a given task. However, this method requires human experts in the relevant field to intervene to create the initial model data and modifications, and requires them to select the final model structure.

[0022] Another approach is to automate the process by using a computer to generate different model structures and selecting the final model from the generated structures. However, previous efforts to automate model generation have only achieved limited results. Although modern computers have greatly improved in computing performance, existing methods for the automated generation of model structures (such as neural network structures) tend to explore limited search spaces or require unrealistic amounts of computing resources. In practice, generating model structures tends to be computationally feasible, but given the currently available computing hardware, independently training a large number of different model structures tends to be computationally infeasible.

[0023] One way to quantify the time it takes to train a model architecture is by defining a reference computational resource, such as a virtual machine or processor, and the amount of time required to complete training on the reference computational resource. For example, one day of training on a particular type of graphics processing unit ("GPU") may be referred to as a GPU-day, and the computational cost of training a given model may be specified as a number of GPU-days. Other approaches are also possible, such as the number of training hours on a particular CPU or FPGA model, the amount of time spent on a virtual machine, etc.

[0024] One approach to automated generation of model architectures is to simply define a search space and generate all model architectures in the search space. All possible model architectures can then be independently trained to obtain trained models, and one of the trained models can be selected as the final model. Typically, the search space can be defined by constraining the depth of the model architecture, the operations performed by the model, and the connectivity of the model. For example, a computer can be configured to consider all fully connected model architectures with a depth of five, where each of the five layers performs one of several possible operations.

[0025] However, this approach is not feasible for many models. First, model architectures can grow very large, with varying layers and nodes. Second, training can take an extremely long time for such large architectures. Finally, this approach fails to consider other model architectures outside the search space that might be more suitable for the task at hand. For example, if the task is image recognition, a six-layer model is clearly more suitable for the task than any clever five-layer model, and this six-layer model would remain undiscovered if the search space were restricted to five layers.

[0026] Another complex approach for automated generation of model structures involves the creation of a separate "controller" model that is trained to generate new model structures for a given task. However, previous efforts to use a controller model to generate new model structures still have the drawbacks described above. If a large search space is considered, for example, the controller model is expected to consider a wide range of possible model structures, each model structure needs to be generated, fully trained, and then evaluated against the other models. Because training can take days on large datasets, efforts to automate learning of new model structures using a relatively unrestricted search space have had limited success. For example, such an approach may take hundreds or thousands of GPU talents to output a final model structure with an acceptable level of performance.

[0027] Another approach for automated model generation is to significantly restrict the search space and then search in a relatively limited set of models within the restricted search space. For example, some previous methods define an external model skeleton with a specified number of modular layers and the types of connections between modules. These methods then generate and evaluate different candidate substructures or "cells", which can then be repeated in the external model skeleton. However, these methods do not consider alternative external skeletons, alternative connections between cells, or the use of different types of substructures in each cell.

[0028] Unlike the shortcomings of the above-mentioned traditional techniques, the embodiments of the present disclosure can generate model structures in a relatively uncontrolled search space while saving significant training time relative to the above-mentioned techniques. By considering a broad search space, the embodiments of the present disclosure are more likely to find new model structures that provide better performance than more restricted search spaces. For example, the embodiments of the present disclosure can find model structures that are not limited to a predefined external skeleton and can find model structures with various operation types in different layers of the model.

[0029] Embodiments of the present disclosure can utilize several techniques to avoid individually generating and training each model structure in the search space. Instead, embodiments of the present disclosure direct the growth of new model structures into portions of the search space that are expected to contain improved model structures compared to those already evaluated, while avoiding searching other portions of the search space that are less likely to contain improved model structures.

[0030] Model structure generation method

[0031] The following discussion presents an overview of functionality that can allow for automated generation of model structures, such as neural network structures. Figure 1 An example method 100 according to the present disclosure is illustrated. As described in detail below, the method 100 can be implemented on many different device types, such as through one or more cloud servers, through client devices such as laptops, tablets, smartphones, or through a combination of one or more servers, client devices, etc.

[0032] Method 100 begins at block 102, where a pool of parent models is initialized. For example, one or more initial parent models may be added to the pool of parent models. The initial parent models may be known models that have been previously identified as performing well in a particular task, randomly generated, or simply predefined default seed models.

[0033] Method 100 continues to block 104 where a particular parent model is selected from the pool. For example, the parent model may be selected randomly or deterministically from the pool. The particular parent model has a corresponding model structure that may be modified, as described in detail below.

[0034] Method 100 continues to block 106 where a candidate layer is generated and initialized. Generally, generating a candidate layer can include selecting one or more layers of a particular parent model to provide input to the candidate layer and selecting another "target" layer of the particular parent model to receive the output of the candidate layer. Thus, a candidate layer can be considered an addition to the structure of a particular parent model. Furthermore, generating a candidate layer can include selecting an operation to be performed on the input provided by the selected layers of the particular parent model. Initializing the candidate layer can include performing some initial training on the candidate layer, as discussed further herein.

[0035] Method 100 continues to block 108, where a particular candidate layer is selected for inclusion in one or more child models derived from the parent model. For example, a particular candidate layer may be selected based on initialization parameters learned when initializing the candidate layer, as described in detail below. In other cases, each candidate layer is selected for inclusion in a child model. In either case, the child model can be considered a model that inherits the structure of the parent model and additionally includes one or more selected candidate layers.

[0036] Method 100 continues to block 110 where the child model is trained. In some cases, the child model is trained by reusing learned parameters from the parent model, as described in detail below. In these cases, the child model can be considered to initially inherit the learned parameters from the parent model, but these learned parameters may be further adjusted during the training of the child model.

[0037] Method 100 continues to block 112 where the trained sub-model is evaluated according to one or more criteria. Typically, the criteria may relate to the performance of the model in a given task, such as accuracy and / or other factors, such as the computational cost of training the sub-model.

[0038] The method 100 continues at block 114 where the individual child model is designated as a new parent model based on the evaluation at block 112. The new parent model is added to the parent model pool.

[0039] The method 100 continues to decision block 116 where it is determined whether a stopping condition has been reached. The stopping condition may define a specified amount of computing resources to be used (e.g., a budget in GPU days), a specified performance criterion (e.g., a threshold accuracy), a specified amount of time, etc.

[0040] If the stopping condition is not reached, the method continues to block 104 where subsequent iterations of blocks 104-116 may be performed. Generally speaking, blocks 104-116 may be considered an iterative model growing process that may be repeated multiple times until the stopping condition is reached.

[0041] If the stopping condition is reached, the method continues to block 118 where the final model is output. For example, the final model may be selected from all sub-models based on one or more criteria (eg, as discussed above with respect to block 112).

[0042] In many cases, method 100 is performed to generate a model that is well-suited for a specific application. For example, in a facial recognition scenario, the training data may include labeled examples of images, indicating whether a face is included in the image and the location of the face in the image. In the example of scene segmentation, the training data may include labeled examples of videos that have been segmented into predetermined segments. By iteratively generating new sub-models and evaluating them against a training dataset for a specific task, method 100 can output a final model that performs well on that specific task.

[0043] Candidate layer generation

[0044] Figure 2 The parent model 210 and the candidate layer 220 are shown. The following describes an example of how to get the candidate layer from the parent model structure. Figure 2 Represents a method for generating multiple candidate layers from a given parent model, rather than a single instance of the parent model and candidate layers, as detailed below.

[0045] Figure 2 In the example, the elements of the parent model are shown in solid lines and the elements of the candidate layers are shown in dashed lines. In this case, the parent model includes model input 211, layers 212, 213, 214 and 215, and model output 216. The parent model may also include one or more other layers (not shown) Figure 2 (denoted by ellipsis). Typically, model inputs can include features to be processed by the model, such as raw image, video, sound, and / or text data. Outputs can represent the results of the model's computations, such as facial landmarks in an image, segmented video, transcribed sound, semantic representations of text, and so on.

[0046] In this example, the candidate layer includes individual operations 221, 222, and 223 and an aggregation operation 224. Typically, the individual operations 221, 222, and 223 may include convolution, pooling, etc., as further discussed herein. The aggregation operation 224 may include manipulating the outputs of the individual operations to conform to a target layer of the parent model, such as layer 215, that will receive the output from the aggregation operation. For example, the aggregation operation may concatenate the outputs of the individual operations and transform them into a form or shape that matches the input of the target layer.

[0047] As described above, block 106 of method 100 can be used to generate a candidate layer. Generating a candidate layer can include selecting one or more input layers from a parent model, selecting one or more target layers of the parent model to receive the output of the candidate layer, and selecting one or more operations to be performed by the candidate layer on the input. Generating a candidate layer also includes selecting operational parameters of the operation, such as convolution size, convolution stride, pooling window, etc.

[0048] In this example, the search space of possible candidate layers can be defined as follows. In the parent model, let x1,...,x L For all existing layers of the parent model. Candidate layers can be represented by a tuple (x out ,x in,1 ,op1,x in,2 ,op2,...,x in,J ,op J ) definition, where J is a positive integer, x out , x in,1 ...x in,K is an existing layer, and op1…op J is an operation (such as convolution and pooling, etc.). out can be defined as computing the topological ordering of the image according to the parent model, strictly in all x in,i After that, this can avoid forming a direct loop. The candidate layer starts from is calculated and then added to take input from one or more input layers and provide their output to x out .

[0049] Used to form the candidate layer x c A specific algorithm is as follows. First, randomly sample the target layer x from the layers of the parent model 210 out . Secondly, the three input layers x in,i (i=1,2,3) are selected. To ensure x c Access to local layer, x in1 Can be selected as the parent model in the initial x out The deepest input of . From topologically earlier than x out In all layers of the parent model, x in,2 and x in,3 can be sampled with uniform random replacement. Next, the operation to be performed on each input layer is randomly and uniformly selected from a set of operations.

[0050] For example, groups of operations can be predefined. Specific examples of operations include convolution, pooling, and identification operations. Each operation type can have different operation parameters. For example, a convolution operation can have a specified kernel size parameter - 1x1, 3x3, 5x5, 7x7, etc. Convolution can also include filter size parameters, such as 16, 32, or 64 filters, step parameters, padding parameters, etc. Pooling operations can include maximum and average pooling operations, and can be applied to windows that vary according to window size parameters. Generally, in this document, these parameters are referred to as "operation parameters" to distinguish them from "learning parameters" obtained through model training, such as weights and bias values. Generating candidate layers can include deterministically or randomly selecting different operations and / or different operation parameters.

[0051] In some embodiments, the separable convolution may be applied twice. The output of each operation may be concatenated together by an aggregation operation 224 and then projected to the target layer x using a 1x1 convolution. out The same shape. The result is the candidate layer x c .

[0052] Some embodiments may restrict the connections of candidate layers to reduce the search space. At a given depth i of the parent model, there are i-1 possible earlier layers and any subset of the earlier layers can be selected as input to the candidate layer. Therefore, the input selection for the candidate layer is exponential. Therefore, some embodiments will limit the input options by learning only repeatable cell modules and deploying the cells into a manually designed skeleton macrostructure. In these embodiments, the layers of the cell module can only take input from other layers of the same cell and from the output of the two previous most recent cells. Other embodiments are not limited to any particular skeleton or connectivity arrangement and may allow sparse skip connections in any layer of the bone model that changes as the model grows.

[0053] Block 106 of method 100 may include generating one or more candidate layers using the above-described method for each iteration of the method. In some cases, the model structure may include thousands or millions of different layers connected in any number of different combinations, and therefore, the number of possible candidate layers generated from a single parent model in a single iteration may be large. Some methods that can reduce the computational burden of independently training the entire space of possible child model structures are discussed below.

[0054] Candidate layer initialization

[0055] As noted above, block 106 of method 100 may also include initializing the candidate layers. Generally, candidate layer initialization serves multiple purposes. First, initializing the parameters of the candidate layers allows the child model to be trained from the initialization, rather than starting over. In other words, when final training occurs at block 110 of method 100, the child model has already been partially trained. Second, candidate layer initialization provides information about the candidate layers so that the candidate layers can be selectively added to the parent model when other layers are omitted, as described in detail below.

[0056] Figure 3 The diagram illustrates an exemplary initialization operation for an embodiment in which a candidate layer 220 is trained by initially preventing the candidate layer from influencing the parent model 210. To do this, operations 221, 222, and 223 can be augmented with a stop-gradient operation (sg), which is applied to the input of each operation. sg(x) is x during the forward pass and zero during the backward pass. During the backward pass, gradients are calculated for each layer to adjust the model's learning parameters. The sg operation prevents the gradients from influencing any learned parameters in any preceding layer of the parent model.

[0057] Before the output is provided to the target layer 215, a stop-forward (sf) operation 302 can be applied to the output 224 of the aggregation operation. Here, sf(x) is zero in the forward propagation and is the identification function in the backward propagation. This allows the candidate layer 220 to receive gradient information during initialization without affecting the target layer. Therefore, during initialization, the candidate layer can accumulate loss gradients with respect to the target layer without actually affecting the value output by the target layer or any subsequent output.

[0058] Therefore, when the candidate layer is initialized, the learning parameters of the parent model can remain stable while determining the initialized values ​​of the learning parameters for the candidate layer. In some cases, different candidate layers generated in the same iteration can share edges. In this case, the weights of the shared edges can be initialized and / or trained independently for each candidate layer. Optionally, initialization and / or training can be performed so that different candidate layers share the weights of the edges they have in common. For example, during the initialization and / or training of shared edges, forward propagation and backward propagation operations can be shared and performed independently by other edges that are not shared by the candidate layers. By sharing the initialization and / or training of edges in a given candidate layer and / or sub-model, computational costs will be significantly saved.

[0059] Pruning candidate layer

[0060] In some cases, block 108 of method 100 may involve selecting all of the candidate layers generated in block 106 to include in the child model for subsequent training and evaluation. When this is the case, method 100 can still provide significant advantages over previous techniques because only some of the child models are designated as parent models in block 114. Thus, the child model for each iteration is derived from a known good parent model. This approach allows a significant portion of the search space to be omitted when growing new child models.

[0061] However, depending on the possible operations, operation parameters, and the types of connections being considered, the number of possible candidate layers in any given iteration can be very large. Consequently, the number of possible intermediate child models for a given parent model can also be very large. As a fine-tuning measure to further reduce the computational burden of training new child models, some embodiments can filter out certain candidate layers before training the child models in block 110. This can further reduce the space of child models that need to be trained and evaluated in each iteration, as described above.

[0062] One way to reduce the number of sub-models for training is to randomly sample possible candidate layers in block 108 of method 100 so that fewer sub-models need to be trained. In some other embodiments, block 108 may include using initialization parameters of the candidate layers to determine which candidate layers should be incorporated into the sub-model for training. Consider a scenario where multiple candidate layers share connections to one or more input layers and a target layer of a given parent model, and also perform different operations. Different weights can be initialized for edges of candidate layers that are input and / or output by different operations. These initialized weights can be used to select some candidate layers to include in the sub-model for further training. Other candidate layers can be pruned so that the sub-models with these candidate layers are not trained, thereby saving additional training time.

[0063] Figure 4 Illustrate a technique for using initialized parameters to prune away candidate layers that are unlikely to be useful. In particular, Figure 4 A scenario is shown where multiple operations can be initialized together by getting an aggregate candidate layer 402 from the parent model 210. Typically, the aggregate candidate layer 402 represents multiple candidate layers, each of which shares a connection to the parent model but performs different operations. As detailed below, Figure 4 Alternatives are provided for: adding all candidate layers to the parent model to obtain a child model, or randomly selecting candidate layers. Note that for brevity, Figure 4 Omitted Figure 2 and Figure 3 Layer 212 is shown in FIG.

[0064] A special algorithm for pruning candidate layers is as follows. For each input x in,i , for x in,i Possible operations can be implemented as op i,1 …op i,k , where k is the number of possible operations. For example, in Figure 4 , these inputs are provided by layers 213 and 214. For example, given J inputs x in,i , which gives a total of JK tensors o1,o2…o Jk These operations can be trained together simultaneously. After training, the subset of operations can be summed together to obtain the value of x via the following formula c to complete the aggregation candidate layer operation 404, where

[0065] After training, the aggregate candidate layer 402 may include different weights for edges connecting different operations. Thus, the aggregate candidate layer may be designed as multiple initialized candidate layers with different sets of weights for edges connecting different operations. These weights may be input to a feature selection technique to select one or more of the candidate layers for use in a sub-model, and similarly prune one or more candidate layers so that they are not used in the sub-model. The feature selection process is Figure 4 This is indicated by selecting operation 406. Note that, as in Figure 3 In a similar way to what was discussed, Figure 4 Utilization of a stop-forward operation 408 and stop-gradient operations 410 , 412 is also shown.

[0066] One method of implementing the sparse feature selection in the selection operation 406 is to linearly combine the selections o1, ..., o on the linear weights. Jk and using L-1 norm regularization (e.g. least absolute shrinkage and the selection operator "lasso"). This sparsification can be performed by adding regularization to the total loss:

[0067]

[0068] where λ out is a parameter associated with the target layer 215 to manage sparsity. Another alternative is to replace the linear loss with a square loss, as follows:

[0069]

[0070] Recall that in some embodiments So the previous equation is equivalent to:

[0071]

[0072] From one perspective, the implementations discussed above employ feature selection and learning parameter sharing to initialize a combined number of candidate layers and select a subset of them to include in child models for further training. This approach can bias the selection of candidate layers that are likely to improve model performance and deselect those that are unlikely to improve model performance. As a result, fewer total child models need to be trained as possible parent models, and the trained child models are more likely to produce offspring with the desired performance.

[0073] Sub-model training

[0074] After initialization, in the child model, a given child model may include the learned parameters inherited from the parent model and the initialized parameters for all candidate layers. As described above, block 110 of method 100 may include training the child model as a whole, for example, the candidate layers may be trained together with the remaining parent model. This can be considered a "warm start" technique, where the child model has initialized the candidate layer parameters and inherited the parent model parameters at the beginning of training, both of which can be further modified when training the given child model as a whole.

[0075] Figure 5 An example method for training a sub-model is shown. Figure 5 , the stop-gradient operation on the input can be removed. In addition, the stop-forward operation can be replaced by a scalar multiplier 502. The scalar is trainable and can be initialized to 0. Therefore, immediately after initialization, the child model represents the same functionality as the parent model. Accordingly, the child model has a different structure from the parent model, but the functionality of the parent model is preserved. The child model is then trained from the combination of the parent model and the initialized candidate parameters, as the scalar multiplier changes over time and the added candidate layers slowly begin to contribute to the target layer and subsequent outputs of the model. This approach can avoid the candidate layers from destabilizing the learned parameters inherited from the parent model, which may be close to the optimal solution given that the parent model has been fully trained.

[0076] Note that some implementations may omit Figure 3 The sg and sf operations shown, in contrast, allow the candidate layer to influence the parent model immediately while the candidate layer is being trained. Figure 3 The shown sf-sg as prefix and suffix of the candidate layer can make the candidate layer converge faster. However, this involves formulating additional targets for the candidate during initialization.

[0077] On the other hand, allowing values to flow freely between the parent model and the candidate layer can enable the new candidate layer to directly contribute to fitting the final loss. However, compared to the parent model, the initial values for the candidate layer may be too far from the optimal values, and this can adversely affect the parameters of the parent model. An alternative solution to this problem is to use a learning rate that is much smaller than the one used to train the parent model (0.1 - 0.02 times the original) for initializing the candidate layer.

[0078] Evaluate and designate the sub - model as the parent model

[0079] As described above, some sub - models are added to the parent model pool in box 114 of method 110. One way to determine which sub - models to add to the parent model pool involves using one or more criteria to predict which sub - models are more likely to produce descendants that are better than known models in subsequent iterations. Generally, the criteria can consider factors such as the loss of a given sub - model, the computational effort to train the given sub - model. In model validation, a higher loss means lower accuracy, and a higher cost means a longer training time on the training dataset. Sub - models that exhibit relatively low loss and low training effort can tend to be included in the parent model pool.

[0080] A specific method for selecting sub - models for the parent pool is as Figure 6 shown. The figure shows an example scatter plot 600 for various trained models. For each sub - model that has completed training, the computational cost of training that sub - model can be calculated and labeled on the x - axis 602. Additionally, the loss of that sub - model can be calculated and labeled on the y - axis 604. Once all the models for a given iteration are plotted, the lower convex hull 606 can be calculated from the plotted values. Note that the computational cost can be calculated as a value that reflects the amount of time to train the model on standard hardware (e.g., a specific model of GPU), such as GPU days. In other embodiments, the computational cost reflects the test - time cost, e.g., the number of operations involved in using a given model to make predictions, without considering training - specific values (e.g., the number of training epochs and / or data augmentation during training). The computational cost can be normalized to a number between 0 and 1, as Figure 6 shown.

[0081] The lower convex hull 606 can be used as a mechanism to determine whether a given sub - model is added to the parent model pool. For example, using the probability defined by the following specific algorithm, sub - models on the lower convex hull can be added to the parent model pool. If m1 and m2 are two adjacent models on the convex hull, with computational costs c1 and c2 (c1 < c2), then the probability weight of m1 can be set proportionally to c2 - c1. The most accurate model with no successor model on the curve can be selected to be included in the parent model pool with a probability of 0.5. In Figure 6The most accurate model is 608 because it has the lowest loss.

[0082] Typically, the lower convex hull is a subset of the Pareto frontier, and therefore another approach is to select child models on the Pareto frontier for inclusion in the parent pool. Both approaches offer good performance for selecting child models to add to a parent model. One approach for observing the lower convex hull and / or Pareto frontier is as follows. A given model on the lower convex hull or Pareto frontier cannot be improved on one criterion by moving to another lower convex hull or Pareto frontier model without lowering the other criterion.

[0083] Note that the same model may have different validation errors due to the randomness of the stochastic gradient formation. As a result, the lower convex hull or Pareto frontier can be relaxed using the multiplicative bandwidth. Therefore, within the same computational cost, if the validation error of a child model is within (1 + γ) times the validation error of the lower convex hull, the child model can be considered to be on the lower convex hull and selected as the parent. Some embodiments may set γ = 0.025. This approach allows a specific child model that is close to the lower convex hull, although not strictly located, to be designated as the parent model.

[0084] Other methods may also be used to allow child models to be selected as parent models if they are located near a predetermined lower convex hull. For example, some embodiments may define a threshold distance from the lower convex hull and allow child models within the threshold distance to be selected as parent models. This is just one of many methods that can be used to select one or more child models as subsets of a parent model based on one or more criteria.

[0085] Figure 6 The trained models are shown as black dots. For explanation purposes, assume Figure 6 600 after iterations N. One or more of the sub-models on or near the lower convex hull 606 may be selected as parent models for a subsequent iteration (N+1), where additional candidate layers may be added and initialized to form further sub-models, as described above.

[0086] Figure 7 A scatter plot 600 is shown of the subsequent state after N+1 iterations. Figure 7 The sub-model trained during the N+1 iteration is shown in FIG. A new lower convex hull 702 can be calculated. The previous lower convex hull 606 is represented by a dotted line to illustrate the downward movement of the lower convex hull during the N+1 iteration.

[0087] Again, one or more sub-models in or near the lower convex hull can be selected for subsequent iteration N+2. The sub-models trained during iteration N+2 are as follows Figure 8A new lower convex hull 802 may be calculated, and the previous lower convex hulls 606 and 702 are shown in dashed lines to illustrate their positions relative to the lower convex hull 802 .

[0088] A view like Figure 6-Figure 8 The method described is a greedy approach that seeks for cost-effective predictions. Note that this is a multi-objective approach, i.e., it considers both the computational cost of training / validation and model accuracy. Alternative implementations may use different and / or additional criteria, e.g., a multi-dimensional graph of three or more criteria, an objective function defined on one or more criteria, etc.

[0089] The above methods typically use randomization to grow the network. However, in contrast to purely computationally infeasible randomization methods, this method is guided by predictive criteria, which tends to use known good models as a basis for further modification. As noted above, retraining a model is computationally intensive. For example, the training dataset may include millions of training items, and a given model needs to be trained over several training epochs before convergence. A training epoch may include one forward propagation and one backward propagation through the entire model for each data item in the training dataset.

[0090] The above method provides advantages of various traditional methods for automated model generation. Note that not every child model is used to derive candidate layers for subsequent training. Instead, by using a subset of child models that appear with the lower convex hull as new parent models, embodiments of the present disclosure start each new iteration with child models that inherit the parent model structure of a known good model. This allows subsequent iterations to start when the training model does not occupy a significant part of the search space and is far away from the lower convex hull, and can save a lot of training time. In addition, by using not only accuracy but also training cost as criteria for selecting which child model to use as the new parent model, embodiments of the present disclosure do not support the generation of new models that are more computationally intensive. Instead, the present method not only reduces these models, but also reduces the training time for training the child models derived therefrom.

[0091] Furthermore, recall that candidate layer initialization can be performed on child models, which inherit not only the structure of their parent model but also its learned parameters. Therefore, the parameters of new candidate layers can be quickly initialized to reasonable values. Furthermore, this allows for "warm-starting" the training of the child models as a whole. This can accelerate convergence relative to retraining each child model, for example by requiring fewer training epochs or training samples.

[0092] Furthermore, recall that shared edges between two child models can be initialized and trained together. Given that the child models are drawn from the shared pool of the parent model, the likelihood of shared edges is very high. Therefore, a significant amount of training time can be saved compared to the alternative approach where each shared edge is initialized and trained separately for each child model.

[0093] Recall that previous techniques for automated generation of machine learning models have either incurred computationally infeasible training burdens or considered models only in very restricted search spaces. Together, the benefits described above enable the automated generation of a large number of accurate machine learning models from a relatively unconstrained search space, without adding an excessive computational burden to the computing resources used to generate the models.

[0094] Example Processing Flow

[0095] Figure 9 A high-level overview of a process flow 900 for incrementally growing a model structure is provided. The process flow illustrates how a parent model pool is iteratively grown. The process flow is an example of a method for assigning certain tasks to worker processes (e.g., virtual machines). For example, the process flow may be suitable for a distributed system or server farm where a hypervisor or operating system schedules new tasks over time as worker processes become available.

[0096] At stage 910, a parent model pool is determined 912. For example, in some cases, manual or automated techniques may select one or more models for the pool based on known model performance. For simplicity, at initialization, it is assumed that the pool includes a single model 914.

[0097] At stage 920, candidate layers are initialized from the parent model. For example, Figure 9 Candidate layers 922, 924, and 926 are shown. As described above, each candidate layer can be added to the parent model pool 912. The candidate layer is added to the candidate layer queue 928. When a given worker process becomes idle, the worker process can pull a candidate layer from the candidate layer queue and initialize the candidate layer as described herein, for example, by performing several initial iterations of training.

[0098] Once initialized, a subset of candidate layers may be selected to be included in a sub-model queue 932 for training in stage 930. An idle worker process may pull a given sub-model from the sub-model queue for training in stage 930. Figure 9 One such sub-model 934 is shown in stage 930, which includes candidate layer 924, but not candidate layers 922 or 926. This indicates that candidate layers 922 and / or 926 have already been pruned in the previous stage 920, as discussed in the feature selection embodiment above.

[0099] As previously mentioned, the child model does not have to be trained from scratch, but rather can be trained starting from the learned parameters of the parent model and the initialization parameters predetermined in stage 920 when initializing the candidate layers to be added to the child model. Also, note that, as mentioned above, multiple child models can be trained collectively as part of a larger model.

[0100] At stage 930, some of the child models from the child model queue may be added to the parent model pool 912, as indicated by arrow 936. As described above, the child models to be added to the parent model pool may be selected based on various criteria. For example, some embodiments may evaluate the trained child models on a validation dataset and determine a corresponding loss for each child model, which loss may be used as a criterion for adding the child models to the parent model pool. Alternatively and / or additionally, the computational cost of training a given child model may be used to determine whether the model is added to the parent model pool. Other approaches may utilize a form of calculation of the net expected value of adding the child model to the parent model pool based on the computational overhead and a projection calculated on the expected value of the addition. Once the parent model pool has been updated, processing may return to stage 910 for subsequent iterations.

[0101] Example System

[0102] The embodiments of the present disclosure can be implemented in various scenarios on various devices. Figure 10 An example system 100 is shown in which embodiments of the present disclosure are applied, as described in detail below.

[0103] like Figure 10 As shown, system 1000 includes client device 1010, server 1020, server 1030, and client device 1040 connected via one or more networks 1050. Note that client devices can be implemented as mobile devices such as smartphones or tablet computers, or as fixed devices such as desktop computers, server devices, etc. Similarly, servers can be implemented using various types of computing devices. In some cases, Figure 10 Any of the devices shown, particularly the servers, may be implemented in a data center, server farm, or the like.

[0104] Figure 10 Certain components of the apparatus shown in FIG may be represented herein by reference numerals within parentheses. For purposes of the following description, parentheses (1) indicate the presence of a given component on client device 1010, (2) indicate the presence of a given component on server 1020, (3) indicate the presence on server 1030, and (4) indicate the presence on client device 1040. This disclosure generally does not use parentheses to refer to components unless identifying a specific instance of a given component.

[0105] Typically, devices 1010, 1020, 1030, and / or 1040 may have corresponding processing resources 1001 and storage resources 1002, as described in detail below. The devices may also include various modules that can use the processing resources and the storage resources to perform the techniques described in this disclosure. The storage resources may include permanent storage resources (such as magnetic or solid-state drives) and volatile storage (such as one or more random access memory devices). In some cases, the modules are provided as executable instructions that can be stored on a permanent storage device, loaded into a random access memory device, and read from the random access memory for execution by the processing resources.

[0106] The client device 1010 may include a configuration module 1011, which may interact with a model generation module 1021 on the server 1020. Typically, the configuration module may provide some configuration parameters to the model generation module. The model generation module uses these configuration parameters to perform the model generation function as described in the present disclosure. In particular, the model generation module may execute method 100 based on the configuration parameters. As described above, the iterative model growth process may include modifying the parent model to obtain a child model, selecting a candidate layer to be included in the child model based on at least the learned weights during the initialization process of the candidate layer, and outputting a final model selected from the child model. The model generation module may include various submodules (not shown) for each box of method 100.

[0107] The model generation module 1021 can output the final model to the server 1030. In response to the received input, the model execution module 1031 can execute the final model. For example, the recording module 1041 on the client device 1040 can record data such as images or speech for processing by the final model, and the local application 1042 can upload the recorded data to the server 1030 for processing. The model execution module can process the uploaded data using the final model and provide the model output to the local application for further processing.

[0108] Sample graphical interface

[0109] As described above, the configuration module 1011 on the client device 1010 may provide initial configuration parameters to the model generation module 1021. The model generation module 1021 may execute the method 100 according to the configuration parameters provided by the configuration module. Figure 11 An example configuration graphical user interface ("GUI") 1100 is shown that may be presented on the client device 1010 for a user to define these configuration parameters.

[0110] The Parent Model element 1101 allows the user to specify which type of initial parent model should be used to start the model growing process. Figure 11, it is shown that the user has selected a default parent model. For example, the model generation module 1021 can provide a default neural network structure to be used as a general initial parent model. Other options include randomly generated models, where the module generation module selects a random model structure to be used as the initial parent model. Another option is for the user to navigate to an existing model that is known to provide relatively good performance for a specific task. In this case, the configuration module 1011 can upload the specified model to the model generation module to be used as the initial parent model.

[0111] The operation unit 1102 allows the user to specify which types of operations are considered through the model generation module 1021. For example, the model generation module may provide various options for listing groups of operations, the listed operations being used to select candidate layers. The first option, Group 1, may define two operations, a single convolution operation (5x5) and a single pooling operation (such as max pooling). The second option, Group 2, may define a total of four possible operations: two convolution operations (such as 5x5 and 7x7) and two pooling operations (average pooling and max pooling). The third option, Group 3, may define a total of seven possible operations: 1x1, 3x3, 5x5, and 7x7 convolutions, average and max pooling, identity operations. The listed operations may also have predetermined or optional operating parameters, such as adjustable window size, step size, etc.

[0112] Budget input unit 1103 allows the user to specify a computational budget for model generation. For example, the user may specify a budget of 10,000 GPU days, and model generation module 1021 may use this budget as a stopping condition at decision block 116 of method 100. Alternative embodiments may use other metrics as computational budgets, such as the number of processing operations, the number of virtual machines, the amount of time, etc.

[0113] Criteria 1 element 1104 allows the user to specify a first criterion for evaluating the model, and Criteria 2 element 1105 allows the user to specify a second criterion. Figure 11 In

[15] , as described above, these criteria are expressed as computational cost and loss, respectively. However, users may wish to specify other criteria, such as the overall size of the model structure, the number of connections, the total number of learnable parameters of the model, etc. For resource-constrained applications, such as running models on mobile devices or Internet of Things (IoT) devices, the model size in bytes can be an important criterion, as these devices tend to have limited storage and / or memory capacity. Additionally, in these cases, the computational time to execute the model can also be a useful criterion, as these devices may have limited processing power, which may result in user-perceived delays when a given model takes too long to execute.

[0114] Notice, Figure 11The configuration parameters shown are merely exemplary, and various other implementations may be contemplated. For example, in some cases, a user may specify connectivity parameters. For example, a user may specify that a candidate layer receives input from a specified number of previous layers or a varying (e.g., random) number of previous layers. As another example, a user may specify whether skip connections are allowed, e.g., where a layer may not provide input to the immediately following layer, but instead skips the immediately following layer and connects to another subsequent layer. A user may also specify a dense network architecture, where each layer is connected to all previous layers in the model.

[0115] It should also be noted that some embodiments may provide one or more GUIs to display the progress of method 100. For example, some embodiments may display the progress of method 100 in a manner similar to Figure 6-Figure 8 The illustrated manner generates a GUI showing a scatter plot 600 that changes at different iterations of model growing. Other embodiments may display graphical representations of individual models and / or candidate layers as they are generated.

[0116] Sample Application

[0117] The technology described in this disclosure can be used in various applications without limitation. However, for the purpose of illustration, some specific examples are given below.

[0118] As a first example, suppose an entity wishes to operate an application that tags all of the friends in a user's photo collection. The application may have a pre-existing model for this purpose, and the model can be executed by the model execution module 1031 of the server 1030 on the photos provided by the local application 1042 on the client device 1040. However, the entity may feel that the pre-existing model is not accurate enough and may provide an unsatisfactory user experience in some cases.

[0119] First, the entity may upload a pre-existing model to the model generation module 1021 on the server 1020 and configure various initial parameters as described above. The model generation module may then iteratively grow the pre-existing model until a stopping condition is reached and output the final model to the server 1030. The server 1030 may replace the initial model with the received final model and continue tagging the user's photos as described above.

[0120] As an alternative example, the client device can use the final model to transcribe recorded audio data. As another alternative, the client device can use the final model to translate text from a first language into a second language. As additional examples, the final model can perform scene segmentation on videos, object detection (e.g., identifying faces, cars, etc.) on images or videos, and grammatical evaluation on text.

[0121] In another example, the final model can be used to map documents and queries into semantic vectors. In this case, server 1030 can provide a search engine function that uses the vector distance between indexed documents and received queries to rank search results for users of client device 1040. More generally, the final model can perform any suitable task for which training data is available, including but not limited to classification, machine translation, and pattern recognition tasks.

[0122] Similarly, some embodiments may allow entities to provide datasets for training, validation, and / or testing. For example, a first entity may send a dataset from client device 1010 to a second entity that controls server 1020. The second entity may use the dataset provided by the first entity to generate a final model. In some cases, the second entity may provide the final model to the first entity, for example, for the first entity to execute on server 1030. In some cases, the second entity may not provide the model itself, but rather allow the first entity to call the model. In this case, the second entity may control server 1020 and server 1030, and / or implement both model generation and model execution on the same device.

[0123] Furthermore, note that the term "final model" as used in this disclosure only means that a given sub-model is designated for actual use by an application. In some embodiments, the final model output and used by an entity may also be used as the initial parent model for subsequent iterations of the model growth process described in this disclosure. In some cases, an entity acquires new data during the process of using a given final model, and this new data may be used as training / validation / test data for subsequent iterations of the process. This may result in the output of a new final model, which may then be used for a period of time before updated data is acquired, and the process may be repeated continuously for the lifetime of a given actual application targeted by a given entity.

[0124] Furthermore, it is noted that the embodiments disclosed herein focus on examples of adding candidate layers to a model structure. However, in some cases, other types of candidate components can be used in place of layers. In general, candidate components can be implemented using any type of mathematical and / or logical functions, ranging from simple arithmetic or Boolean operators to more complex trainable modules.

[0125] Equipment implementation

[0126] Reference Figure 10As noted, system 1000 includes several devices, including client device 1010, server 1020, server 1030, and client device 1040. It should also be noted that not all device implementations may be shown, and other device implementations will be apparent to those skilled in the art based on the above and following descriptions.

[0127] As used herein, the terms "device," "computer," "computing device," "client device," and / or "server device" may refer to any type of device having a certain amount of hardware processing power and / or hardware storage / memory power. The processing power may be provided by one or more hardware processors (e.g., hardware processing units / cores) that may execute data in the form of computer-readable instructions to provide functionality. The computer-readable instructions and / or data may be stored on a storage device, such as a storage device / memory and / or a data storage device. As used herein, the term "system" may refer to a single device, a plurality of devices, and the like.

[0128] Storage resources can be internal or external to the corresponding devices to which they are associated. Storage resources can include one or more of any volatile or non-volatile memory, hard drives, flash memory devices, and / or optical storage devices (e.g., optical disks, CDs, DVDs, etc.). The term "computer-readable medium" as used in this disclosure may include signals. In contrast, the term "computer-readable storage medium" does not include signals. Computer-readable storage media includes "computer-readable storage devices." Examples of computer-readable storage devices include volatile storage media (e.g., RAM) and non-volatile storage media (e.g., hard drives, optical disks, and flash memory, etc.).

[0129] In some cases, the device is configured with a general-purpose hardware processor and storage resources. In other cases, the device may include a system-on-chip (SOC) type design. In an SOC design embodiment, the functions provided by the device may be integrated on a single SOC or multiple coupled SOCs. One or more associated processors may be configured to coordinate with shared resources (such as memory, storage devices, etc.) and / or one or more dedicated resources (such as hardware blocks configured to perform specific functions). Therefore, the term "processor" used in this disclosure may also refer to a central processing unit (CPU), a graphics processing unit (GPU), a controller, a microcontroller, a processor core, or other types of processing devices suitable for implementation in traditional computing architectures as well as SOC designs.

[0130] Alternatively or additionally, the functions described in the present disclosure may be performed or partially performed by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), etc.

[0131] In some configurations, any of the modules / codes described herein may be implemented in software, hardware, and / or firmware. In any case, the modules / codes may be provided during the production of the device or in preparation for sale to the end user. In other cases, the end user may later install the modules / codes, such as by downloading an executable code and installing the executable code on the corresponding device.

[0132] Note that devices can generally have input and / or output capabilities. For example, a computing device can have various input mechanisms, such as a keyboard, mouse, trackpad, voice recognition, gesture recognition (using a depth camera, such as a stereo or time-of-flight camera system, an infrared camera system, an RGB camera, or using an accelerometer / gyroscope, facial recognition, etc.). A device can also have various output mechanisms, such as a printer, a monitor, etc.

[0133] It should also be noted that the devices described in this disclosure can implement the described techniques individually or in a collaborative manner. For example, the methods or functions described in this disclosure can be executed on a single computing device and / or distributed across multiple computing devices communicating via a network 1050. Without limitation, the network 1050 can include one or more local area networks (LANs), wide area networks (WANs), the Internet, etc.

[0134] Additional Examples

[0135] Various device examples are described above. Additional examples are described below. One example includes a method performed on a computing device, the method comprising performing two or more iterations of iterative model growing. The iterative model growing process comprises selecting a particular parent model from a pool of parent models consisting of one or more parent models, generating multiple candidate layers, and initializing the multiple candidate layers while reusing the learning parameters of the particular parent model. The iterative model growing process may also include selecting a particular candidate layer to include in a child model for training, and the corresponding child model may include the particular parent model and one or more candidate layers of the particular candidate layer. The iterative model growing process may also include training multiple child models to obtain trained child models, evaluating the trained child models using one or more criteria, and designating individual trained child models as new parent models based at least on the evaluation, and adding the new parent model to the pool of parent models. The method may include, after two or more iterations, selecting at least one trained child model as a final model, and outputting the final model.

[0136] Another example may include any of the above and / or following examples, wherein the method further comprises determining a computational cost of training or testing the sub-model when evaluating the trained sub-model, and using the computational cost as the first criterion.

[0137] Another example may include any of the above and / or following examples, wherein the method further comprises determining a loss associated with the trained sub-model when evaluating the trained sub-model, and using the loss as the second criterion.

[0138] Another example may include any of the above and / or following examples, wherein the method further includes plotting the child model on a graph having a first axis reflecting computational cost and a second axis reflecting loss, and selecting a new parent model based at least on a corresponding position of the new parent model on the graph.

[0139] Another example may include any of the above and / or following examples, wherein the method further comprises determining at least one lower convex hull or Pareto frontier on the graph, and selecting a new parent model based at least on a proximity of the new parent model to the lower convex hull or Pareto frontier.

[0140] Another example may include any of the above and / or following examples, wherein the selecting includes identifying a subset of the trained sub-models that are within a predetermined neighborhood of the lower convex hull or the Pareto frontier, determining respective probabilities for the subset of the trained sub-models, and selecting a new parent model based at least on the respective probabilities.

[0141] Another example may include any of the above and / or following examples, where generating an individual candidate layer includes selecting a target layer from a specific parent model to receive the output of the individual candidate layer, selecting one or more input layers from the specific parent model to provide input to the individual candidate layer, and selecting a specific operation to be performed on the input by the individual candidate layer.

[0142] Another example may include any of the above and / or following examples, wherein selecting the specific operation includes defining an operation group and randomly selecting the specific operation from the operation group.

[0143] Another example may include any of the above and / or following examples, wherein the method further comprises randomly selecting the target layer and at least one input layer from a particular parent model.

[0144] Another example may include any of the above and / or following examples, wherein the final model is a neural network.

[0145] Another example includes a system comprising a hardware processing unit and a storage resource storing computer-readable instructions that, when executed by the hardware processing unit, cause the hardware processing unit to perform an iterative model growing process comprising modifying a parent model to obtain a child model. The iterative model growing process may include selecting a candidate layer to include in the child model based at least on learned weights during an initialization process for the candidate layer. The computer-readable instructions may further cause the hardware processing unit to also output a final model selected from the child models.

[0146] Another example may include any of the above and / or following examples, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to generate different candidate layers that share connectivity with the parent model and perform different operations, and initialize the different candidate layers together to obtain different weights for the different candidate layers.

[0147] Another example may include any of the above and / or following examples, wherein the computer-readable instructions, when executed by a hardware processing unit, cause the hardware processing unit to select individual candidate layers for inclusion in the sub-model based at least on different weights of different candidate layers.

[0148] Another example may include any of the above and / or following examples, wherein the computer-readable instructions, when executed by a hardware processing unit, cause the hardware processing unit to perform a feature selection technique on different weights of different candidate layers to select individual candidate layers for inclusion in the sub-model.

[0149] Another example may include any of the above and / or following examples, wherein the feature selection technique includes a least absolute shrinkage and selection operator (LASSO).

[0150] Another example may include any of the above and / or following examples, wherein the different operations include a convolution operation and a pooling operation.

[0151] Another example may include any of the above and / or following examples, wherein, when executed by a hardware processing unit, the computer-readable instructions cause the hardware processing unit to train a final model using training data for at least one classification, machine translation, or pattern recognition task, and provide a final model for execution, the final model being suitable for performing at least one classification, machine translation, or pattern recognition task.

[0152] Another example includes a computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to perform actions including two or more iterations of an iterative model growing process. The iterative model growing process may include: selecting a particular child model from a parent model pool of one or more parent models, initializing a plurality of candidate layers, selecting a plurality of child models for training, each child model inheriting the structure of the particular parent model and including at least one candidate layer, training the plurality of child models to obtain trained child models, and designating a single trained child model as a new parent model based at least in part on one or more criteria, and adding the new parent model to the parent model pool. The actions also include, after two or more iterations, selecting at least one trained child model as a final model and outputting the final model.

[0153] Another example may include any of the above and / or following examples, wherein the action further includes simultaneously initializing multiple candidate layers to obtain initialized parameters and selecting individual candidate layers to include in separate sub-models based at least on the initialized parameters.

[0154] Another example may include any of the above and / or following examples, wherein the action further comprises randomly selecting an operation from the enumerated group of operations to include in the plurality of candidate layers.

[0155] in conclusion

[0156] Although the subject matter of the present disclosure has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims, and other features and acts that those skilled in the art will recognize are intended to fall within the scope of the claims.

Claims

1. A method for automatically generating a machine learning model, the method comprising: Performing two or more iterations of an iterative model growing process, the iterative model growing process comprising: Selecting a specific parent model from a parent model pool of two or more parent models; Generate multiple candidate layers, and initialize the multiple candidate layers while reusing the learning parameters of the specific parent model, wherein initializing the multiple candidate layers includes: Applying stop-gradient to the corresponding input of each operation in the candidate layer, wherein the stop-gradient is a function that prevents gradients from affecting learned parameters in previous layers; and applying a stop-and-forward to the outputs of the plurality of candidate layers, wherein the stop-and-forward is zero during forward propagation and is a signature function during backward propagation; selecting a specific candidate layer to be included in a child model for training, the corresponding child model including the specific parent model and one or more specific candidate layers among the specific candidate layers; Training the sub-model to obtain a trained sub-model, wherein the training comprises: replacing the stop-gradient with a scalar multiplier, wherein the scalar multiplier is initialized to zero; removing the stop-gradient; and increasing the scalar multiplier over time to increase contributions of the plurality of candidate layers to the output of the parent model; determining a computational cost of training or testing the trained sub-model; and designating an individual trained child model as a new parent model based at least on the computational cost of training or testing the trained child model, and adding the new parent model to the pool of parent models; and After the two or more iterations, at least one trained sub-model is selected as a final model, and the final model is output.

2. The method according to claim 1, further comprising: determining a computational cost of training or testing the sub-model; as well as The computational cost is used as a first criterion when evaluating the trained sub-models.

3. The method according to claim 2, further comprising: determining a loss associated with the trained sub-model; as well as Based at least on the loss, the individually trained child model is selected as the new parent model, and the new parent model is added to the pool of parent models.

4. The method according to claim 3, further comprising: plotting the submodel on a graph having a first axis reflecting the computational cost and a second axis reflecting the loss; as well as The new parent model is selected based at least on a corresponding position of the new parent model on the graph.

5. The method according to claim 4, further comprising: determining at least one of a lower convex hull or a Pareto frontier on the graph; as well as The new parent model is selected based at least on a proximity of the new parent model to the lower convex hull or the Pareto frontier.

6. The method of claim 5, wherein the selecting comprises: identifying a subset of the trained sub-models that is within a predetermined neighborhood of the lower convex hull or the Pareto frontier; determining corresponding probabilities for the subsets of the trained sub-models; as well as The new parent model is selected based at least on the corresponding probabilities.

7. The method of claim 1 , wherein generating individual candidate layers comprises: selecting a target layer from the specific parent model to receive the output of the individual candidate layer; selecting one or more input layers from the particular parent model to provide input to the individual candidate layer; as well as A particular operation is selected to be performed on the input by the individual candidate layer.

8. The method according to claim 7, wherein selecting the specific operation comprises: Define action groups; as well as The specific operation is randomly selected from the group of operations.

9. The method according to claim 7, further comprising: The target layer and at least one input layer are randomly selected from the specific parent model.

10. The method of claim 1, wherein the final model is a neural network.

11. A system for automatically generating a machine learning model, the system comprising: Hardware processing unit; as well as A storage resource storing computer-readable instructions that, when executed by the hardware processing unit, cause the hardware processing unit to: An iterative model growing process is performed, the iterative model growing process comprising: Selecting a specific parent model from a parent model pool of two or more parent models; Generate multiple candidate layers and initialize the multiple candidate layers while reusing the learning parameters of the specific parent model, wherein initializing the multiple candidate layers includes: applying a stop-gradient operation to the corresponding input of each operation in the candidate layer, wherein the stop-gradient operation is a function that prevents gradients from affecting learned parameters in a previous layer; and applying a stop-and-forward to the outputs of the plurality of candidate layers, wherein the stop-and-forward is zero during forward propagation and is a signature function during backward propagation; selecting a specific candidate layer to be included in a child model for training, the corresponding child model including the specific parent model and one or more specific candidate layers among the specific candidate layers; Training the plurality of sub-models to obtain trained sub-models, wherein the training comprises: replacing the stop-gradient operation with a scalar multiplier, wherein the scalar multiplier is initialized to zero; removing the stop-gradient; and increasing the scalar multiplier over time to increase contributions of the plurality of candidate layers to the output of the parent model; evaluating the trained sub-model using one or more criteria; determining a computational cost of training or testing the trained sub-model; and designating an individual trained child model as a new parent model based at least on at least one computational cost of training or testing the trained child model, and adding the new parent model to the pool of parent models; and After the two or more iterations, at least one trained sub-model is selected as a final model, and the final model is output.

12. The system of claim 11 , wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to: generating different candidate layers that share connectivity to the parent model and perform different operations; and The different candidate layers are initialized together to obtain different weights for the different candidate layers.

13. The system of claim 12, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to: Based at least on the different weights of the different candidate layers, an individual candidate layer is selected for inclusion in the sub-model.

14. The system of claim 13, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to: A feature selection technique is performed on the different weights of the different candidate layers to select the individual candidate layers for inclusion in the sub-model.

15. The system of claim 14, wherein the feature selection technique comprises least absolute shrinkage and selection operator LASSO.

16. The system of claim 13, wherein individual candidate layers perform different operations, the different operations comprising at least a convolution operation and a pooling operation.

17. The system of claim 11, wherein the computer-readable instructions, when executed by the hardware processing unit, cause the hardware processing unit to: training the final model using training data for at least one classification, machine translation, or pattern recognition task; and The final model is provided for execution, the final model being suitable for performing the at least one classification, machine translation, or pattern recognition task.

18. A computer-readable storage medium storing instructions that, when executed by a processing device, cause the processing device to perform actions for automatically generating a machine learning model, the actions comprising: Performing two or more model growing iterations of an iterative process, the iterative model growing process comprising: Selecting a specific parent model from a parent model pool of two or more parent models; Generate multiple candidate layers and initialize the multiple candidate layers while reusing the learning parameters of the specific parent model, wherein initializing the multiple candidate layers includes: Applying stop-gradient to the corresponding input of each operation in the candidate layer, wherein the stop-gradient is a function that prevents gradients from affecting learned parameters in previous layers; and applying a stop-and-forward to the outputs of the plurality of candidate layers, wherein the stop-and-forward is zero during forward propagation and is a signature function during backward propagation; selecting a specific candidate layer to be included in a child model for training, the corresponding child model including the specific parent model and one or more of the specific candidate layers; Training the sub-model to obtain a trained sub-model, wherein the training comprises: replacing the stop-gradient with a scalar multiplier, wherein the scalar multiplier is initialized to zero; removing the stop-gradient; and increasing the scalar multiplier over time to increase contributions of the plurality of candidate layers to the output of the parent model; determining a computational cost of training or testing the trained sub-model; and designating an individual trained child model as a new parent model based at least in part on one or more criteria based at least on at least one computational cost of training or testing the trained child model, and adding the new parent model to the pool of parent models; and After the two or more iterations, at least one trained sub-model is selected as a final model, and the final model is output.

19. The computer-readable storage medium of claim 18, the actions further comprising: Initialize multiple candidate layers simultaneously to obtain initialized parameters; as well as Based at least on the initialized parameters, individual candidate layers are selected for inclusion in the individual sub-model.

20. The computer-readable storage medium of claim 19, the actions further comprising: Operations are randomly selected from the enumerated group of operations to include in the plurality of candidate layers.

Images