Resource-constrained neural network architecture search

Abstract

A method (200) includes defining a neural network computational unit (122) comprising a directed graph of nodes (402) representing respective neural network latent representations and edges (404) representing respective operations that transform the respective neural network latent representations; replacing each operation that transforms a respective neural network latent representation with a respective linear combination of candidate operations, wherein each candidate operation in the respective linear combination has a respective mixing weight parameterized by one or more computational unit hyperparameters; iteratively adjusting values of the computational unit hyperparameters (132a) and weights (132b) to optimize a validation loss function subject to a computational resource constraint (106); and generating a neural network (152) for performing a machine learning task using the defined computational unit and the adjusted values of the computational unit hyperparameters and weights.

Description

Technical Field

[0001] This specification relates to determining the architecture of a neural network. Background Technology

[0002] A neural network is a machine learning model that uses one or more non-linear units to predict the output from a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from the received input based on the current values ​​of its corresponding parameter set. Summary of the Invention

[0003] This specification describes an end-to-end neural architecture search framework for one-off neural architecture search under resource constraints, wherein it is capable of learning customized network architectures for any machine learning task dataset.

[0004] Typically, an innovative aspect of the subject matter described in this specification includes a method for searching neural network architectures, the method comprising: defining computational units of a neural network, wherein the computational units comprise a directed graph of nodes and edges, each node representing a corresponding neural network latent representation and each edge representing a corresponding operation that transforms the corresponding neural network latent representation; replacing each operation that transforms the corresponding neural network latent representation with a corresponding linear combination of candidate operations from a predefined set of candidate operations, wherein each candidate operation in the corresponding linear combination has a corresponding mixed weight parameterized by one or more computational unit hyperparameters; iteratively adjusting i) the values ​​of the computational unit hyperparameters and ii) the values ​​of the computational unit weights to optimize a validation loss function constrained by one or more computational resource constraints, for each iteration including: performing unconstrained optimization of the validation loss function to update the values ​​of the computational unit hyperparameters from the previous iteration and obtaining adjusted values ​​of the computational unit weights; and projecting the updated values ​​of the computational unit hyperparameters onto the nearest points in a feasible set defined by one or more resource constraints to obtain adjusted values ​​of the computational unit hyperparameters; and generating a neural network for performing a machine learning task using the defined computational units and the adjusted values ​​of the computational unit hyperparameters and the adjusted values ​​of the computational unit weights.

[0005] Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each corresponding computer system, apparatus, and computer program configured to perform actions of the method. A system of one or more computers can be configured to perform a specific operation or action by means of software, firmware, hardware, or any combination thereof installed on a system that can cause the system to perform actions during operation. One or more computer programs can be configured to perform a specific operation or action by means of instructions that, when executed by a data processing apparatus, cause the apparatus to perform the action.

[0006] The foregoing and other embodiments may optionally include one or more of the following features, individually or in combination. In some embodiments, generating a neural network for performing a machine learning task using defined computational units and adjusted values ​​of computational unit hyperparameters and adjusted values ​​of computational unit weights includes: defining a discrete computational unit architecture by replacing each linear combination of candidate operations with a single operation having a maximum mixed weight, which is parameterized by one or more adjusted computational unit hyperparameters; and generating a neural network for performing a machine learning task using the defined discrete computational unit architecture and the adjusted values ​​of the computational unit weights.

[0007] In some implementations, generating a neural network for performing machine learning tasks using defined computational units and adjusted values ​​of computational unit hyperparameters and adjusted values ​​of computational unit weights includes stacking multiple copies of defined computational units, wherein each copy has the same unit architecture defined by the adjusted values ​​of computational unit hyperparameters.

[0008] In some semi-dormant states, the method further includes: generating multiple copies of the defined computational unit; and training each generated copy of the defined computational unit on the corresponding training data.

[0009] In some implementations, stacking multiple copies of the defined computational unit includes interleaving one or more additional neural network layers between the copies of the defined computational unit.

[0010] In some implementations, the one or more additional neural network layers include a connection computation unit comprising an input node and an intermediate node, and wherein the method further includes learning the connection unit by iteratively adjusting i) the values ​​of the connection computation unit hyperparameters and ii) the values ​​of the connection computation unit weights to optimize a validation loss function constrained by one or more computational resources.

[0011] In some implementations, the validation loss function represents a measurement of the error obtained after running the validation dataset through a defined computation set.

[0012] In some implementations, iteratively adjusting the values ​​of i) computational unit hyperparameters and ii) computational unit weights to optimize the validation loss function includes performing a two-layer optimization of the validation loss function and a training loss function representing a measurement of the error obtained on the training data, wherein the computational unit hyperparameters include upper-layer parameters and the computational unit weights include lower-layer parameters.

[0013] In some implementations, one or more computing resource constraints include user-defined constraints on one or more of the following: memory, the amount of floating-point operations, or inference speed.

[0014] In some implementations, iteratively adjusting the values ​​of i) computational unit hyperparameters and ii) computational unit weights to optimize a verification loss function constrained by one or more computational resource constraints includes defining a corresponding cost function for each computational resource constraint, wherein each defined cost function maps computational unit hyperparameters to a corresponding resource cost.

[0015] In some implementations, the corresponding resource cost of an edge in a computation unit is calculated as a softmax over the cost of the operation in the candidate operation set.

[0016] In some implementations, the method further includes setting lower and upper bound constraints for each defined cost function.

[0017] In some implementations, performing unconstrained optimization of the validation loss function to update the values ​​of computational unit hyperparameters used in previous iterations and obtain adjusted values ​​of computational unit weights includes: iteratively performing unconstrained optimization of the validation loss function, for each iteration including: obtaining values ​​of computational unit hyperparameters and computational unit weights for the iteration, including obtaining randomly initialized values ​​of computational unit hyperparameters and computational unit weights for the first iteration, or obtaining values ​​of computational unit hyperparameters and said computational unit weights from previous iterations; and iteratively, reaching a predetermined number of iterations or until a predefined convergence criterion is met: fixing the obtained values ​​of computational unit hyperparameters used in the iterations and obtaining updated computational unit weights based on the gradient of the training loss function relative to the computational unit weights; and fixing the updated computational unit weights and obtaining updated values ​​of computational unit hyperparameters based on the gradient of the validation loss function relative to the computational unit hyperparameters.

[0018] In some implementations, fixing the updated computational unit weights and obtaining the updated computational unit hyperparameters based on the gradient of the validation loss function with respect to the computational unit hyperparameters includes assuming that the computational unit hyperparameters and computational unit weights are independent.

[0019] In some implementations, projecting the updated value of the computational unit hyperparameter to the nearest point in a feasible set defined by one or more resource constraints includes identifying an element in the feasible set that minimizes the 2-norm of the difference between i) the updated value of the computational unit hyperparameter and ii) the element.

[0020] In some implementations, the predefined set of candidate operations includes pooling operations, convolution operations, or connection operations.

[0021] In some implementations, the method further includes: defining a plurality of computational units of a neural network, wherein each defined computational unit can be represented by a corresponding directed graph of nodes and edges; and for each defined computational unit among the plurality of defined computational units: replacing each operation that transforms the latent representation of the corresponding neural network with a corresponding linear combination of candidate operations from a predefined set of candidate operations, wherein each candidate operation in the corresponding linear combination has a corresponding mixed weight parameterized by one or more computational unit hyperparameters; iteratively adjusting the values ​​of i) the computational unit hyperparameters and ii) the computational unit weights to optimize a validation loss function constrained by one or more computational resources, for each iteration including: performing unconstrained optimization of the validation loss function to update the values ​​of the computational unit hyperparameters of the previous iteration and obtain adjusted values ​​of the computational unit weights; and projecting the updated values ​​of the computational unit hyperparameters onto the nearest points in a feasible set defined by one or more resource constraints to obtain adjusted values ​​of the computational unit hyperparameters; and generating a neural network for performing a machine learning task using the defined plurality of computational units and the adjusted values ​​of the corresponding computational unit hyperparameters and computational unit weights.

[0022] In some implementations, the method further includes: training the generated neural network on training data to obtain a trained neural network; and using the trained neural network to perform a machine learning task.

[0023] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.

[0024] Systems implementing the techniques described herein are capable of learning neural network architectures that satisfy task-related resource constraints such as model size and computational complexity. For example, the system can learn lightweight neural network architectures that can be efficiently implemented on mobile platforms with constrained computational resources.

[0025] Furthermore, lightweight neural network architectures can be learned under resource constraints without compromising their quality, such as accuracy and performance. For example, the learned neural network architecture can achieve state-of-the-art performance in terms of accuracy, model size, and complexity. The techniques described herein for performing neural network architecture search also enable improved speed for neural architecture search.

[0026] Furthermore, the techniques described in this specification are not limited to specific machine learning applications—systems implementing the techniques described herein can learn customized neural network architectures for any specific machine learning task and dataset. The techniques described herein are also applicable to neural architecture search with one-off resource constraints.

[0027] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0028] Figure 1 An example neural network architecture search system is shown.

[0029] Figure 2 This is a flowchart illustrating an example process for generating a neural network to perform machine learning tasks.

[0030] Figure 3 This is a flowchart of an example process for performing an iterative projection method.

[0031] Figure 4 It is a conceptual visualization of an example for learning discrete computing unit architecture.

[0032] The same reference numerals and names in the various figures denote the same elements. Detailed Implementation

[0033] Designing and implementing neural network architectures for performing machine learning tasks such as image recognition, speech recognition, or language modeling can be a time-consuming and expensive process that requires expert knowledge and experience in the field.

[0034] One example technique used for automating the design of neural network architectures is Neural Architecture Search (NAS). NAS techniques can be categorized into two main groups. The first group of NAS techniques uses black-box optimization methods, such as reinforcement learning or genetic algorithms, to optimize the reward function. This technique typically requires training thousands of deep learning models to learn the neural network architecture, thus incurring significant computational costs. Furthermore, NAS techniques using black-box optimization are computationally prohibitive for one-off NAS. One-off NAS is important for resource-constrained applications because different tasks require different neural network architectures. For example, for a simple problem such as classifying the colors of an image, a simple neural network architecture, such as a two-layer neural network, is sufficient. On the other hand, classifying cats and dogs from images requires complex neural networks.

[0035] The second group of NAS techniques formulates the neural architecture search task as a differentiable optimization problem and utilizes gradient descent to find the optimal solution. Compared to the NAS techniques in the first group, the second group of NAS techniques is generally more computationally efficient.

[0036] This specification describes an end-to-end resource-constrained differentiable architecture search framework for one-off NAS. The differentiable architecture search task is formulated as a constrained optimization task by incorporating resource constraints, where the search space of the resource-constrained optimization task is mapped to a continuous search space to enable the application of gradient descent methods. An iterative projection algorithm is applied to solve the constrained optimization task, and neural network architectures are learned within the feasible set defined by the constraints. Multi-level search strategies can be applied to learn different architectures for neural network layers of varying depths.

[0037] The neural network architecture learned by the techniques described herein can be configured to receive any type of digital data input and generate any type of score, classification, or regression output based on the input.

[0038] For example, if the input to a neural network defined by a neural network architecture is an image or features extracted from an image, the output generated by the neural network for a given image can be a score for each of the object classification sets, where each score represents an estimated probability that the image contains an object belonging to that classification.

[0039] As another example, if the input to a neural network defined by a neural network architecture is an internet resource (e.g., a webpage), a document, or a portion of a document, or features extracted from an internet resource, document, or portion of a document, then the output generated by the neural network for a given internet resource, document, or portion of a document can be a score for each of the topics in the set, with each score representing an estimated probability that the internet resource, document, or portion of a document is about that topic.

[0040] As another example, if the input to a neural network defined by a neural network architecture is features of the impression context of a particular advertisement, then the output generated by the neural network can be a score representing the estimated probability that the particular advertisement will be clicked.

[0041] As another example, if the input to a neural network defined by a neural network architecture is features for personalized recommendations to a user, such as features characterizing the context of the recommendation, or features characterizing the user's previous actions, then the output generated by the neural network can be a score for each of the content items in the set, with each score representing an estimated probability that the user will respond positively to the recommended content item.

[0042] As another example, if the input to a neural network defined by a neural network architecture is a sequence of texts in one language, the output generated by the neural network can be a score for each of the text fragments in another language, each score representing an estimated probability that the text fragment in the other language is a correct translation of the input text into that other language.

[0043] As another example, if the input to a neural network defined by a neural network architecture is a sequence representing spoken utterances, then the output generated by the neural network can be a score for each of the text fragments in the set, each score representing an estimated probability that the text fragment is a correct transcript of the utterance.

[0044] Example hardware

[0045] Figure 1 An example neural architecture search system 100 is shown. The neural architecture search system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below can be implemented.

[0046] The neural architecture search system 100 is a system that receives training data 102 for training a neural network to perform a specific machine learning task, a validation set 104 for evaluating the performance of the neural network on the specific machine learning task, and data 106 specifying the resource constraints of the computing device that implements the neural network when performing the specific machine learning task.

[0047] The neural architecture search system 100 uses training data 102 and a validation set 104 to determine the neural network architecture configured to perform a specific task. This architecture defines the number of layers in the neural network, the operations performed by each layer, and the connectivity between layers—that is, which layers receive input from which other layers in the neural network. Typically, both training data 102 and validation set 104 include sets of neural network inputs, and for each network input, include the corresponding target output that should be generated by the neural network to perform the specific task. For example, a larger training dataset may have been randomly split to generate training data 102 and validation set 104.

[0048] The neural architecture search system 100 uses resource constraints when determining the neural network architecture for a neural network configured to perform a specific task. That is, the neural architecture search system 100 learns the neural network architecture under resource constraints. Example resource constraints include, but are not limited to, the amount of available memory, the number of floating-point operations, inference speed, or model size. Learning neural networks under resource constraints is particularly useful for mobile platforms and real-time applications.

[0049] System 100 can receive training data 102, validation set 104, and computational resource constraints in any of a variety of ways. For example, system 100 can receive training data as an upload from a remote user of the system via a data communication network, for example, using an application programming interface (API) available to system 100, and randomly divide the uploaded data into training data 102 and validation set 104. System 100 can also, for example, receive user-defined computational resource constraints as an upload from a remote user of the system via a data communication network using an application programming interface (API) available to system 100.

[0050] As another example, system 100 can receive input from a user specifying which data, maintained by system 100, should be used to train the neural network, and then divide the specified data into training data 102 and a validation set 104. Similarly, system 100 can receive input from a user specifying which of a plurality of resource constraints maintained by system 100 should be used to determine the neural network architecture.

[0051] The neural architecture search system 100 includes a computational unit generator 110, a computational unit parameter tuning engine 120, and a neural network architecture generator 130.

[0052] The computational unit generator 110 is configured to define computational units of a neural network whose neural network architecture is determined by the system 100. The computational units defined by the computational unit generator 110 are architectural building blocks of the neural network, such as subnetworks, whose neural network architecture is determined by the system 100. For example, multiple instances of defined computational units, each with a corresponding learned architecture and independently learned weights, can be stacked to generate deeper neural networks.

[0053] Each computational unit defined by computational unit generator 110 can be represented as a directed acyclic graph G = (V, E) with a predetermined number of nodes V and edges E. Each node x in the computational unit i ∈V represents a latent representation, such as a feature map in a convolutional network. Each directed edge (ij) is associated with operation O. i,j Related. Operation O i,j Transform node x i For example, this operation will potentially represent x i As input and output, the latent representation x j Each node can be computed based on the transformation outputs of all its predecessors, for example, in, x represents the set of predecessors j See below for reference. Figure 4 Explanation and description of example calculation units.

[0054] The computation unit generator 110 can be configured to define computation units based on the received inputs 102-106. For example, the number of nodes and edges included in the defined computation units can depend on the machine learning task to be performed by the neural network, the neural network architecture of which is determined by system 100, and the computational resources available to implement the neural network.

[0055] The computational unit parameter update engine 120 is configured to receive data representing defined computational units 122 from the computational unit generator 110. The computational unit parameter tuning engine 120 is configured to iteratively tune the values ​​of computational unit hyperparameters and computational unit weights to optimize the verification loss function constrained by resource constraint 106. Tuning the values ​​of computational unit hyperparameters and computational unit weights to optimize the verification loss function constrained by resource constraint 106 includes implementing a continuous relaxation strategy to expand the architecture search space from a set of predefined discrete candidate operations O. i,j The discrete search space defined by the set is mapped to a continuous search space, allowing the architecture to be determined using gradient descent. A constrained optimization problem is then performed on the continuous search space to determine the adjustment values ​​for the hyperparameters of the computational units and the weights of the computational units for optimizing the validation loss function. See below for reference. Figure 2 and 3The operations performed by the computational unit parameter update engine 120 are described in more detail.

[0056] Neural network architecture generator 130 is configured to receive data representing adjusted computational unit parameter values ​​132, such as hyperparameters 132a and weights 132b, from computational unit parameter tuning engine 120. Neural network architecture generator 130 is configured to determine a neural network architecture 150 using the adjusted computational unit parameter values ​​(and the defined computational units 122). For example, neural network architecture generator 130 is capable of determining a neural network architecture 150 as a stack of multiple copies of defined computational units, wherein the architecture of each copy of the computational units has a unit architecture defined by the adjusted computational unit parameter values ​​132. In some cases, neural network architecture generator 130 is capable of including additional layers, such as one or more filter layers, between the stacks of computational units in the determined neural network architecture. (See below for reference.) Figure 2 A more detailed description is given of how the defined computational units and learned computational unit parameters are used to determine the neural network architecture.

[0057] The neural network search system 100 can output architecture data 150 specifying the architecture of a neural network, that is, data specifying the layers that are part of the neural network, the connectivity between layers, and the operations performed by the layers. For example, the neural network search system 100 can output the architecture data 150 to a user who has submitted training data and resource constraints. The user can then use the resource-constrained device to train an instance of the neural network with the determined architecture and use the trained neural network to process the neural network input.

[0058] In some implementations, instead of or in addition to the output architecture data 150, system 100 trains an instance of a neural network with a determined architecture, for example, by starting from scratch or fine-tuning parameter values ​​generated as a result of training the neural network with that architecture, and then uses the trained neural network to process requests received by the user, for example, through an API provided by the system. That is, system 100 is capable of receiving input to be processed, processing the input using the trained neural network, and providing output generated by the trained neural network or data derived from the generated output in response to the received input.

[0059] Programming the hardware

[0060] Figure 2 This is a flowchart of an example method 200 for generating a neural network 152 for performing machine learning tasks. For convenience, method 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural architecture search system appropriately programmed according to this specification, such as... Figure 1The neural architecture search system 100 is capable of executing method 200.

[0061] The system defines computational units for the neural network (step 202). A computational unit can be considered a building block of the neural network architecture generated by example method 200, such as a subnetwork. For example, as described in more detail below with reference to step 206, multiple instances of computational units with the same learned architecture and independently learned weights can be stacked to generate a deeper neural network.

[0062] A defined computational unit can be represented as a directed acyclic graph G = (V, E) with a predetermined number of nodes V and edges E. Each node x in the computational unit... i ∈V represents a latent representation, such as a feature map in a convolutional network. Each directed edge (ij) is associated with operation O. i,j Related. Operation O i,j Transform node x i For example, this operation will potentially represent x i As input and output, the latent representation x j Each node can be computed based on the transformation outputs of all its predecessors, for example, in, x represents the set of predecessors j .

[0063] The number of nodes and edges included in a defined computational unit can depend on the machine learning task to be performed by the final generated neural network and the computational resources available for searching the neural network architecture. For example, since searching the architecture of an entire neural network or a large subnetwork of a neural network on a large dataset can be computationally expensive, the size of the defined computational unit can be chosen to reduce computational costs while maintaining the final accuracy of the neural network.

[0064] In some implementations, a computational unit can include one or more input nodes and one or more output nodes, for example, two input nodes and a single output node. An input node can be defined as a node that transforms the output of a previous computational unit. For example, in the case of a convolutional neural network architecture, the computational unit can be a convolutional unit, and the input node can be defined as the unit output from the previous two layers; for example, the input node represents the input image. In the case of a recurrent neural network architecture, the computational unit is a recurrent unit, and the input node includes the input of the current step and the state carried from the previous step. An output node can be defined as a node that provides the output of the computational unit. The output of the computational unit can be obtained by applying a reduction operation, such as concatenation, to all nodes; for example, x... N-1 =concat(x2,…,x N-2), where N represents the total number of nodes in the computing unit.

[0065] The operations associated with directed edges form the discrete architecture search space. A portion of this includes a predefined set of operations that the neural network architecture can perform. For example, the predefined set of operations can include: pooling operations, such as max pooling or average pooling; convolution operations with different kernel sizes, such as separable convolution or dilated separable convolution; or connections, such as zero connections or identity connections.

[0066] Architecture search space It can be transformed into a continuous search space by: converting a predefined set of operations. Each candidate operation in the algorithm is replaced with a corresponding linear combination of candidate operations from a predefined set of candidate operations—also referred to as a hybrid operation in this paper. Each candidate operation in the corresponding linear combination has a corresponding hybrid weight parameterized by one or more computational unit hyperparameters. In some implementations, each hybrid operation... Output The softmax weighted sum of all possible operations. An example of mixed operations is given by the following equation (1).

[0067]

[0068] In equation (1), the mixed weights of the operations on a pair of nodes (i,j) are determined by the vector. Parameterization.

[0069] In the architecture search space After being transformed into a continuous search space, the task of generating a neural network for performing machine learning tasks includes learning the continuous variable set θ = {θ...} (i,j) Once the continuous variable set has been learned, the corresponding discrete computational unit architecture can be obtained by first determining the node x based on the strength of the corresponding edge. j The strongest predecessors, where the strength of edge (i,j) is defined in equation (2) below.

[0070]

[0071] For example, the system can determine the relationship with node x j The strongest senior's X j Prestige Collection The number of operations of equal size. Then, the blending operation at edge (i,j) is replaced with a single operation with the maximum blending weight, as defined in equation (3) below.

[0072]

[0073] The system learns by optimizing the validation loss function: i) defining the computational unit hyperparameters of the computational unit architecture; and ii) computational unit weights (step 204).

[0074] Validation loss function This represents the error metric obtained after running the validation dataset through the trained neural network. The validation loss depends on the hyperparameters θ of the computational unit and the neural network weights ω, i.e., Furthermore, optimizing the validation loss function includes determining how to minimize the validation loss. Minimize the optimal computational unit hyperparameter θ * , where ω * =ω * (θ) represents the loss function optimized during training. The obtained computational unit weights, the training loss function This represents the error metric obtained on the training data (specific to the machine learning task to be performed by the neural network), i.e., That is, the system performs two-level optimization, where the hyperparameter θ of the computational unit is the upper-level parameter, and the weight ω of the computational unit is the lower-level parameter.

[0075] The system optimization is constrained by one or more resource constraints on the validation loss function. One or more resource constraints can be defined by the user based on the computational resources available when implementing the neural network generated using example method 200. For example, the system can receive data as input representing target values ​​for different resource costs, such as available memory, FLOPs, or inference speed.

[0076] The system can associate each user-defined resource constraint with a corresponding cost function that maps computational unit hyperparameters to corresponding resource costs. For example, the system can create a discretized network architecture from the computational unit hyperparameter θ according to equation (3) and calculate the cost of the discretized network to determine the exact cost of the computational unit architecture. Since the objective function of the discrete network architecture is not continuous, optimizing the objective function using gradient descent is challenging. Therefore, the system implements a continuous relaxation strategy for user-defined resource constraints, where the cost of edge (i,j) in the defined computational unit is calculated as the softmax of the costs of all possible operations, as given in the following equation (4).

[0077]

[0078] In equation (4), express The resource cost of all operations in the process, where F represents the softmax function, σ represents the indicator function, and... This represents the set of predecessor nodes of node j.

[0079] Equation (4) uses the expected value of the resource cost in the cell as an approximation of the actual cost of the discrete architecture derived from θ. Using the functional form in Equation (4) has several advantages. For example, since Equation (4) is differentiable wrtθ, it is possible to optimize the validation loss function using gradient descent. As another example, Equation (4) is easy to implement because the resource cost of each candidate operation at edge (i,j) is related to θ. (i,j) The value of u is independent. Therefore, it is possible to fix and compute u before training. If more complex resource constraints are to be implemented, such as inference speed on a specific platform, it is possible to learn a neural network that maps architectural hyperparameters to resource costs.

[0080] The system can set lower and upper bound constraints for each cost function to prevent the model from learning an oversimplified architecture. Lower bound constraints can be set to ensure the model has sufficient representational power.

[0081] In summary, the system performs optimization under the constraints given by the following equation (5).

[0082]

[0083]

[0084] C L ≤Φ(θ)≤C H (5)

[0085] In equation (5), Representing the training loss, Φ(θ)=[Φ 0 (θ),…,Φ M-1 (θ)] T ,in, Let M be a set of cost functions, and These represent the lower and upper bounds of the user-defined cost constraint, respectively. That is, the cost Φ m (θ) is constrained by Within the range. To optimize the validation loss function. The system executes the iterative projection method. The system executes the iterative projection method because, due to the softmax function in equation (4), the cost function with respect to θ is non-convex, and the objective function has no solution in closed form.

[0086] The iterative projection method optimizes the validation loss function in two alternating phases: an unconstrained training phase, which searches for an improved architecture by performing unconstrained optimization on the validation loss function to learn the computational unit hyperparameters θ in a larger, unconstrained parameter space; and an architecture projection phase, which projects the computational unit hyperparameters θ output from the unconstrained training phase onto the nearest point in the feasible set defined by the constraints in equation (5). See below for reference. Figure 3 The unconstrained training phase and architecture projection phase are described in more detail.

[0087] The system uses the defined computational units and the learned computational unit hyperparameters and computational unit weights to generate a neural network for performing machine learning tasks (step 206).

[0088] In some implementations, generating a neural network using defined computational units and learned computational unit hyperparameters and weights can include multiple copies of stacked computational units. For example, the system can generate multiple copies of computational units, each copy having the same unit architecture defined by the learned computational unit hyperparameters. The system can train each copy of the computational units on a corresponding training dataset (specific to the machine learning task to be performed by the neural network). In this way, the multiple copies of the computational units have independently learned computational unit weights. The system can then stack the trained multiple copies of the computational units to create deeper neural networks for performing machine learning tasks.

[0089] When the defined computational units include input and output nodes, stacking multiple trained replicas of the computational units can include removing the output node from the first replica of the computational unit, removing the input and output nodes from the intermediate replicas of the computational units, and removing the input node from the last replica of the computational unit before stacking. When the defined computational units do not include input and output nodes, stacking multiple trained replicas of the computational units can include adding one or more input and output nodes to the stack. In either case, the system can further add additional nodes and / or layers, such as one or more filter layers, to the stacked trained replicas of the computational units.

[0090] In some implementations, the system can implement a multi-level search strategy when executing example method 200 for generating a neural network. To implement the multi-level search strategy, the system can define multiple computational units in step 202, wherein each defined computational unit can be represented as a corresponding directed acyclic graph of a predetermined number of nodes and edges. That is, the defined multiple computational units can have different architectures. The system can then perform step 204 for each defined computational unit and combine the learned computational units and / or copies of the learned computational units to generate a neural network, as described with reference to step 206.

[0091] Implementing multi-level search strategies can be advantageous for several reasons. For example, units at different network depths can exhibit significant variations in resource costs, such as the number of parameters and FLOPs, because the number of filter channels increases whenever the input resolution decreases. This design is widely used in deep networks to avoid bottlenecks in information flow, where lower-level layers, i.e., layers closer to the input, have larger FLOPs than higher-level layers, while higher-level layers have more parameters than lower-level layers. It can also be advantageous to vary the unit architecture with layer depth to ensure the learned architecture meets given resource constraints. As another example, units at different depths can have different effects on the overall network performance; for instance, lower-level layers (closer to the input) are less sensitive to reducing the number of parameters.

[0092] Furthermore, to achieve a more lightweight architecture, the system can apply steps 202-204 to learn the connection units or layers between the computation unit stacks, instead of predefining connections as, for example, 1x1 transformations. Connection units can be defined as described above with reference to step 202. In some implementations, a connection unit may include an input node and a node within the connection unit.

[0093] Figure 3 It is used to execute the above reference. Figure 2 A flowchart of an example process 300 describing the iterative projection method. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural architecture search system appropriately programmed according to this specification, such as... Figure 1 The neural architecture search system 100 is capable of executing process 300.

[0094] The system randomly initializes the values ​​of the hyperparameters θ = θ0 and the weights ω = ω0 of the computational unit (step 302).

[0095] The system iteratively performs an unconstrained training process (i.e., verifying the unconstrained optimization of the loss function, as described below) and an architecture projection process until a predetermined number of iterations is reached or until a predefined convergence criterion is met (step 304). For example, each iteration can include an implementation of the unconstrained training process, followed by an implementation of the architecture projection process.

[0096] For each iteration, the system obtains the hyperparameter θ of the computational unit. t and the calculation unit weight ω t (Step 304a). In the case of the first iteration of the iterative projection method, the system can obtain randomly initialized computational unit hyperparameters θ. t =θ0 and randomly initialized computational unit weights ω t=ω0. When the iteration is a subsequent iteration of the iterative projection method, the system is able to receive the computational unit hyperparameter θ from the previous iteration t-1 of the iterative projection method. t-1 and the calculation unit weight ω t-1 .

[0097] The system determines the optimization (minimization) based on the above reference. Figure 3 Defined verification loss The adjusted computational unit hyperparameters are used to perform the unconstrained training process (step 304b). As mentioned above, ω * =ω * (θ) represents the loss function optimized during training. The obtained computational unit weights. That is, the system solves the optimization problem given in equation (6) below.

[0098]

[0099]

[0100] Since it is difficult to obtain an exact solution to equation (6) for both the computational unit weight ω and the computational unit hyperparameter θ at the same time, the system implements coordinate gradient descent to iteratively and alternately update the weight ω and hyperparameter θ while fixing the value of the other.

[0101] For example, in the first step, the system can fix the received hyperparameter θ of the computing unit. t The value, and by along Descent to obtain the updated computational unit weights ω t+1 In the second step, the system fixes the computational unit weights ω obtained in the first step. t+1 The value, and by along Descent yields updated computational unit hyperparameters θ t+1 The system can iteratively execute the first and second steps to reach a predetermined number of iterations or until a predetermined convergence criterion is met. During the second step, ω and θ can be assumed to be independent to improve computational efficiency and achieve satisfactory performance.

[0102] The system uses the adjusted computational unit hyperparameter θ t+1 Perform the architecture projection process (step 304c). The system will update the hyperparameters θ of the computational units. t+1 Projecting onto the nearest point θ in the feasible set of resource constraints defined in equation (5) p The purpose of projection can be described by the following equation (7).

[0103]

[0104] Because Φ(θ)p ) is θ p Since it is a non-convex function, equation (7) has no solution in closed form. Therefore, the system transforms equation (7) into a Lagrange function given by the following equation (8).

[0105]

[0106] The system performs gradient descent to optimize equation (8). At time step t=0, the system sets... At the subsequent time step t, the system proceeds along... descent get The system iteratively performs updates until all constraints are satisfied or until a predetermined maximum number of iterations, e, is reached. p .

[0107] In some implementations, for all constraints, the system sets the weights λ1 and λ2 to be the same: λ1 = λ2 = λ. To promote convergence, λ can be set to decrease exponentially during training. At the end of training, λ → 0 and θ p ∶=0. This is because for simple resource constraint calculations... Therefore, the architecture iteration projection phase (step 404c) is faster than the unconstrained training phase (step 404b).

[0108] The system provides the computational unit parameter θ obtained by optimizing the Lagrangian function given by equation (8). p This serves as input for the next iteration of the unconstrained training process and the architecture projection process. If the iteration is the final iteration, the system provides the hyperparameter θ. p For use in deriving the above reference Figure 3 The described discrete architecture (step 306).

[0109] Executing Example Process 400 offers several advantages. For example, by jointly optimizing computational unit weights and hyperparameters, the system is able to learn an improved starting point for the architecture projection process. As another example, after the architecture projection phase is performed, an unconstrained training phase is executed to learn the computational unit architecture in a larger unconstrained parameter space. Therefore, even if the architecture projection phase results in a suboptimal computational unit architecture, the unconstrained training phase can still learn an improved computational unit architecture. Furthermore, since neural networks are sensitive to perturbations on weights during the initial training phase, in some implementations, the system can implement a warm-start strategy, where the first and second steps of the unconstrained training phase of the first iteration of the iterative projection method have a larger number of iterations compared to subsequent iterations of the iterative projection method. This reduces the likelihood of the model getting trapped in poor local optima during the architecture projection phase.

[0110] Figure 4 It is an example conceptual visualization for learning discrete computing unit architecture.

[0111] Phase (a) corresponds to step 202 of example method 200. In phase (a), an initial computation unit 400 is defined. The initial computation unit 400 includes four intermediate nodes, for example, node 402. The initial computation unit 400 can further include one or more input nodes and one output node, but for clarity, Figure 4 The input and output nodes are omitted. The initial computation unit 400 further includes six directed edges between intermediate nodes, for example, edge 404.

[0112] As described above with reference to equation (1), in stage (b), the operation on each of the six edges is replaced by a mixture of all candidate operations from a predefined set of candidate operations. Figure 4 In the example visualization shown, there are three candidate operations, but in some implementations there may be fewer or more candidate operations.

[0113] Phase (c) corresponds to step 204 of example method 200. In phase (c), refer to the above... Figure 3 The described iterative projection method is applied to solve the constrained optimization problem given by equation (5), where the architectural parameters and the weights in the cells are jointly optimized to satisfy resource constraints.

[0114] In stage (d), the final computational unit architecture 406 is derived from the learned weights in the hybrid operation. The computational unit can then be used to generate a neural network, as described in step 206 of the example method 200 above.

[0115] The embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their equivalents, or in one or more combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or control of the operation of a data processing device. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiving device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination thereof. However, the computer storage medium is not a propagation signal.

[0116] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus can include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus can also include code that creates the execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof.

[0117] A computer program (also referred to as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but must, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, for example, in one or more scripts stored in a markup language document, in a single file dedicated to the program involved, or in multiple collaborating files, for example, in a file storing one or more modules, subroutines, or code portions. A computer program can be deployed to be executed on a single computer or on multiple computers located in one location or distributed across multiple locations and interconnected by a communication network.

[0118] As used herein, "engine" or "software engine" refers to a software-implemented input / output system that provides outputs distinct from its inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device, such as a server, mobile phone, tablet, laptop, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device, including one or more processors and computer-readable media. Furthermore, two or more engines can be implemented on the same computing device or on different computing devices.

[0119] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as such dedicated logic circuitry.

[0120] Computers suitable for executing computer programs include, for example, any one or more processors capable of being based on general-purpose and special-purpose microprocessors, or both, or any kind of central processing unit. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The main components of a computer are the central processing unit for executing or running instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer may be operatively coupled to such one or more mass storage devices to receive data from or transfer data to or both. However, a computer does not necessarily have to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few.

[0121] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, such as: semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Processors and memory can be supplemented by or incorporated into dedicated logic circuitry.

[0122] To provide interaction with the user, the implementation of the subject matter described in this specification can be implemented on a computer having: a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user; and a keyboard and pointing devices such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the user's device; for example, by sending a webpage to a webpage received from a webpage on the user's client device in response to a request.

[0123] Embodiments of the subject matter described herein can be implemented in computing systems that include, for example, backend components as data servers, or middleware components such as application servers, or frontend components such as client computers with graphical user interfaces or web browsers, or any combination of one or more such backend, middleware, or frontend components, through which users can interact with embodiments of the subject matter described herein. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0124] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact through a communication network. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other.

[0125] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather as descriptions of features specific to particular embodiments of the invention. Certain features described in this specification within the context of individual embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination can be removed from the combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0126] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order, or that all shown operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0127] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method (200), the method (200) comprising: defining a computational unit (122) of a neural network, wherein the computational unit (122) comprises a directed graph of nodes (402) and edges (404), each node (402) representing a respective neural network latent representation, and each edge (404) representing a respective operation that transforms the respective neural network latent representation; replacing each operation that transforms the respective neural network latent representation with a respective linear combination of candidate operations from a pre-defined set of candidate operations, wherein each candidate operation in the respective linear combination has a respective mixing weight parameterized by one or more computational unit hyperparameters (132a); iteratively adjusting i) values of the computational unit hyperparameters (132a) and ii) values of computational unit weights (132b) to optimize a validation loss function subject to one or more computational resource constraints (106), wherein a respective cost function is defined for each of the one or more computational resource constraints, wherein each defined cost function maps the computational unit hyperparameters (132a) to a respective resource cost, the adjusting comprising, for each iteration: performing an unconstrained optimization of the validation loss function to update the values of the computational unit hyperparameters (132a) of a previous iteration and obtain adjusted values of the computational unit weights (132b); and projecting the updated values of the computational unit hyperparameters (132a) to a nearest point in a feasible set defined by the one or more resource constraints (106) to obtain adjusted values of the computational unit hyperparameters (132a), wherein the one or more computational resource constraints (106) comprise user-defined constraints on one or more of an amount of available memory and a number of floating point operations of a computing device that implements the neural network; and generating a neural network (152) for performing a machine learning task using the defined computational unit (122) and the adjusted values of the computational unit hyperparameters (132a) and the adjusted values of the computational unit weights (132b).

2. The method (200) of claim 1, wherein Generating a neural network (152) for performing a machine learning task using the defined computational unit (122) and the adjusted values of the computational unit hyperparameters (132a) and the adjusted values of the computational unit weights (132b) comprises: defining a discrete (152) computational unit architecture by replacing each linear combination of candidate operations with a single operation having a maximum mixing weight parameterized by the one or more adjusted computational unit hyperparameters (132a); and generating a neural network for performing a machine learning task using the defined discrete computational unit architecture and the adjusted values of the computational unit weights (132b).

3. The method (200) of claim 1, wherein generating a neural network (152) for performing a machine learning task using the defined computational units (122) and the adjusted values of the computational unit hyperparameters (132a) and the adjusted values of the computational unit weights (132b) comprises stacking multiple copies of the defined computational units (122), wherein each copy has the same unit architecture defined by the adjusted values of the computational unit hyperparameters (132a).

4. The method (200) of claim 3, further comprising: generating the multiple copies of the defined computational units (122); and training each generated copy of the defined computational units (122) on respective training data (102). stacking multiple copies of the defined computational units (122) comprises interleaving one or more additional neural network layers between the copies of the defined computational units (122).

5. The method (200) of claim 3, wherein the one or more additional neural network layers comprise a connection computational unit (122) comprising one input node (402) and one intermediate node (402), and wherein the method (200) further comprises learning the connection unit by iteratively adjusting i) values of the connection computational unit hyperparameters (132a) and ii) values of connection computational unit weights (132b) to optimize the validation loss function subject to one or more computational resource constraints (106).

6. The method (200) of claim 5, wherein the validation loss function represents a measure of error obtained after running a validation dataset through the defined computational set.

7. The method (200) of claim 1, wherein iteratively adjusting i) values of the computational unit hyperparameters (132a) and ii) values of computational unit weights (132b) to optimize a validation loss function comprises performing a bi-level optimization of the validation loss function and a training loss function representing a measure of error obtained on training data (102), wherein the computational unit hyperparameters (132a) comprise upper-level parameters and the computational unit weights (132b) comprise lower-level parameters.

8. The method (200) of claim 1, wherein a respective resource cost of an edge (404) in the computational unit (122) is computed as a softmax over the costs of operations in the candidate operation set.

9. The method (200) of claim 1, wherein 10. The method (200) of claim 1, further comprising setting a lower bound constraint and an upper bound constraint for each defined cost function. performing an unconstrained optimization of the validation loss function to update values of the computational unit hyperparameters (132a) for a previous iteration and obtain adjusted values of the computational unit weights (132b) comprises iteratively performing the unconstrained optimization of the validation loss function, for each iteration comprising:

11. The method (200) of claim 1, wherein ​ Obtaining the values ​​of the computational unit hyperparameter (132a) and the computational unit weight (132b) for the iteration includes obtaining randomly initialized values ​​of the computational unit hyperparameter (132a) and the computational unit weight (132b) for the first iteration, or obtaining values ​​of the computational unit hyperparameter (132a) and the computational unit weight (132b) from previous iterations; and Iteratively, until a predetermined number of iterations are reached or until a predefined convergence criterion is met: The obtained values ​​of the computational unit hyperparameters (132a) used in the iteration are fixed, and the updated computational unit weights (132b) are obtained based on the gradient of the training loss function with respect to the computational unit weights (132b); and The updated computation unit weights (132b) are fixed, and the updated values ​​of the computation unit hyperparameters (132a) are obtained based on the gradient of the validation loss function with respect to the computation unit hyperparameters (132a).

12. The method (200) of claim 11, wherein The process of fixing the updated computational unit weights (132b) and obtaining the updated computational unit hyperparameters (132a) based on the gradient of the validation loss function with respect to the computational unit hyperparameters (132a) includes assuming that the computational unit hyperparameters (132a) and the computational unit weights (132b) are independent.

13. The method (200) of claim 1, wherein Projecting the updated value of the computation unit hyperparameter (132a) onto the nearest point in the feasible set defined by the one or more resource constraints (106) includes: identifying an element in the feasible set that minimizes the 2-norm of the difference between i) the updated value of the computation unit hyperparameter (132a) and ii) the element.

14. The method (200) of claim 1, wherein The predefined set of candidate operations includes pooling operations, convolution operations, or connection operations.

15. The method (200) according to claim 1, further comprising: The neural network is defined with a plurality of computational units (122), wherein each defined computational unit (122) can be represented by a corresponding directed graph of nodes (402) and edges (404); and For each of the multiple defined computation units (122): Each operation that transforms the latent representation of the corresponding neural network is replaced with a corresponding linear combination of candidate operations from a predefined set of candidate operations, wherein each candidate operation in the corresponding linear combination has a corresponding mixed weight parameterized by one or more computational unit hyperparameters (132a). Iteratively adjust the values ​​of i) the hyperparameters (132a) of the computational unit and ii) the weights (132b) of the computational unit to optimize the verification loss function, which is constrained by one or more computational resource constraints (106), including, for each iteration: Perform unconstrained optimization of the validation loss function to update the values ​​of the computational unit hyperparameters (132a) from the previous iteration and obtain the adjusted values ​​of the computational unit weights (132b); and The updated value of the computation unit hyperparameter (132a) is projected onto the nearest point in the feasible set defined by the one or more resource constraints (106) to obtain the adjusted value of the computation unit hyperparameter (132a); and A neural network for performing machine learning tasks is generated using the defined plurality of computing units (122) and the adjusted values ​​of the corresponding computing unit hyperparameters (132a) and the adjusted values ​​of the computing unit weights (132b).

16. The method (200) according to any one of claims 1-15, further comprising: The generated neural network (152) is trained on the training data (102) to obtain the trained neural network (152); as well as The machine learning task is performed using the trained neural network (152).

17. A system (100) for searching neural network architectures, comprising one or more computers and one or more storage devices storing instructions, said instructions, when executed by said one or more computers, being operable to cause said one or more computers to perform the following operations: defining a computational unit (122) of a neural network, wherein The computing unit (122) includes a directed graph of nodes (402) and edges (404), where each node (402) represents a corresponding neural network latent representation, and each edge (404) represents a corresponding operation for transforming the corresponding neural network latent representation; Each operation that transforms the latent representation of the corresponding neural network is replaced with a corresponding linear combination of candidate operations from a predefined set of candidate operations, wherein each candidate operation in the corresponding linear combination has a corresponding mixed weight parameterized by one or more computational unit hyperparameters (132a). Iteratively adjust the values ​​of i) the computational unit hyperparameter (132a) and ii) the computational unit weight (132b) to optimize the verification loss function constrained by one or more computational resource constraints (106), wherein a corresponding cost function is defined for each of the one or more computational resource constraints, wherein each defined cost function maps the computational unit hyperparameter (132a) to a corresponding resource cost, the adjustment including, for each iteration: Perform unconstrained optimization of the validation loss function to update the values ​​of the computational unit hyperparameters (132a) from the previous iteration and obtain the adjusted values ​​of the computational unit weights (132b); and The updated values ​​of the computational unit hyperparameters (132a) are projected onto the nearest points in a feasible set defined by the one or more resource constraints (106) to obtain the adjusted values ​​of the computational unit hyperparameters (132a), wherein the one or more computational resource constraints (106) include one or more user-defined constraints on the amount of available memory and the number of floating-point operations of the computing device implementing the neural network; and A neural network (152) for performing machine learning tasks is generated using the defined computational unit (122) and the adjusted values ​​of the computational unit hyperparameters (132a) and the computational unit weights (132b).

18. A computer-readable storage medium comprising instructions stored thereon, the instructions being executable by a processing device and, upon such execution, causing the processing device to perform the following operations: defining a computational unit (122) of a neural network, wherein The computing unit (122) includes a directed graph of nodes (402) and edges (404), where each node (402) represents a corresponding neural network latent representation, and each edge (404) represents a corresponding operation for transforming the corresponding neural network latent representation; Each operation that transforms the latent representation of the corresponding neural network is replaced with a corresponding linear combination of candidate operations from a predefined set of candidate operations, wherein each candidate operation in the corresponding linear combination has a corresponding mixed weight parameterized by one or more computational unit hyperparameters (132a). Iteratively adjust the values ​​of i) the computational unit hyperparameter (132a) and ii) the computational unit weight (132b) to optimize the verification loss function constrained by one or more computational resource constraints (106), wherein a corresponding cost function is defined for each of the one or more computational resource constraints, wherein each defined cost function maps the computational unit hyperparameter (132a) to a corresponding resource cost, the adjustment including, for each iteration: Perform unconstrained optimization of the validation loss function to update the values ​​of the computational unit hyperparameters (132a) from the previous iteration and obtain the adjusted values ​​of the computational unit weights (132b); and The updated values ​​of the computational unit hyperparameters (132a) are projected onto the nearest points in a feasible set defined by the one or more resource constraints (106) to obtain the adjusted values ​​of the computational unit hyperparameters (132a), wherein the one or more computational resource constraints (106) include one or more user-defined constraints on the amount of available memory and the number of floating-point operations of the computing device implementing the neural network; and A neural network (152) for performing machine learning tasks is generated using the defined computational unit (122) and the adjusted values ​​of the computational unit hyperparameters (132a) and the computational unit weights (132b).

Images