Composite model scaling for neural networks

Abstract

A method for determining a final architecture of a neural network for performing a specific machine learning task is described. The method includes: receiving a baseline architecture for a neural network, wherein the baseline architecture has a network width dimension, a network depth dimension, and a resolution dimension; receiving data defining a compound coefficient, the compound coefficient controlling additional computing resources for scaling the baseline architecture; performing a search to determine a baseline width coefficient, a baseline depth coefficient, and a baseline resolution coefficient, the coefficients specifying how to assign additional computing resources to the network width dimension, the network depth dimension, and the resolution dimension of the baseline architecture, respectively; determining a width coefficient, a depth coefficient, and a resolution coefficient based on the baseline width coefficient, the baseline depth coefficient, the baseline resolution coefficient, and the compound coefficient; and generating a final architecture that scales the network width dimension, the network depth dimension, and the resolution dimension of the baseline architecture based on the corresponding width coefficient, depth coefficient, and resolution coefficient.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application Serial No. 62 / 796,034 filed on January 23, 2019. The disclosure of the prior application is considered part of and incorporated by reference into the disclosure of the present application. Background Art

[0003] This specification relates to determining the architecture of a convolutional neural network.

[0004] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of the corresponding set of parameters.

[0005] Some neural networks are recursive neural networks. A recursive neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recursive neural network can use some or all of the internal state of the network from previous time steps when computing the output of the current time step. An example of a recursive neural network is a long short-term memory (LSTM) neural network that includes one or more LSTM memory blocks. Each LSTM memory block can include one or more cells that each include an input gate, a forget gate, and an output gate, which allows the cell to store the previous state of the cell, such as for generating a current activation or being provided to other components of the LSTM neural network. Summary of the invention

[0006] This specification describes a system implemented as a computer program on one or more computers in one or more locations that determines a final architecture of a neural network based on a baseline architecture. As described below, the final architecture is determined based on a target computing resource usage of the final architecture. In this way, the system can determine a final architecture that is suitable for the low-level computing resources (e.g., number of floating point operations per second, FLOPS) available for use by the final architecture. Moreover, it can do so in a more efficient manner than conventional methods that may require extensive hyperparameter tuning. The neural network is configured to perform a specific machine learning task, such as, but not limited to, image processing (e.g., image classification).

[0007] The subject matter described in this specification can be implemented in a specific embodiment to achieve one or more of the following advantages. Convolutional neural networks are typically developed at a fixed computational cost and then scaled to a larger network for better accuracy if more resources are given. The model scaling techniques described in this specification use simple but efficient compound coefficients for scaling a baseline neural network model to a target model given a target resource constraint while maintaining model efficiency. Conventional methods arbitrarily increase network width or depth or resolution when scaling, which results in low accuracy and efficiency, and the process of scaling up the neural network model is inefficient and computationally expensive due to the hyperparameter adjustments required to balance these three factors. In contrast, the described techniques use compound coefficients to linearly scale the network width, depth, and resolution of the baseline model, thereby avoiding spending a lot of computing resources for hyperparameter adjustments. In addition, the model scaling techniques described herein allow the baseline model to be scaled up very efficiently, resulting in a larger model that exceeds the accuracy of the prior art while having fewer parameters than the previous prior art model. At the same time, these larger models can perform specific machine learning tasks using fewer operations than the previous prior art models, and therefore they are faster in training and inference.

[0008] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An example neural architecture search system for determining a final architecture of a neural network for performing a particular machine learning task is shown.

[0010] Figure 2 is a flow chart of an example process for determining a final architecture of a neural network for performing a particular machine learning task.

[0011] Figure 3 An example baseline architecture of a neural network is shown.

[0012] Figure 4 Another example baseline architecture of a neural network is shown.

[0013] Like reference numbers and designations throughout the various drawings represent like elements. DETAILED DESCRIPTION

[0014] This specification describes a neural architecture search system implemented as a computer program on one or more computers in one or more locations that determines a final architecture of a neural network based on a baseline architecture. The neural network is configured to perform a specific machine learning task.

[0015] Typically, a neural network is configured to receive a network input and process the network input to generate a network output for the input.

[0016] In some cases, the neural network is a convolutional neural network that is configured to receive an input image and process the input image to generate a network output for the input image, i.e., perform some image processing task.

[0017] For example, the task may be image classification, and the output generated by the neural network for a given image may be a score for each of a set of object categories, where each score represents an estimated likelihood that the image contains an object belonging to that category.

[0018] As another example, the task may be image embedding generation, and the output generated by the neural network may be a numerical embedding of the input image. For example, the proximity in the embedding space between the embeddings generated for the corresponding input images may represent the similarity between these images.

[0019] As yet another example, the task may be object detection, and the output generated by the neural network may identify locations in the input image where a particular type of object is depicted.

[0020] In some other cases, the task may be video classification, and the neural network is configured to receive as input a video or a portion of a video, and to generate an output that determines what subject matter the input video or portion of a video relates to.

[0021] In some other cases, the task may be speech recognition, and the neural network is configured to receive audio data as input and to generate an output that determines, for a given spoken utterance, one or more words that the utterance represents.

[0022] In some other cases, the task may be text classification, and the neural network is configured to receive an input text segment and generate an output that determines what topic the input text segment relates to.

[0023] Figure 1 An example neural architecture search system 100 is shown that is configured to determine a final architecture of a neural network configured to perform a specific machine learning task based on a baseline architecture. The neural architecture search system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below can be implemented.

[0024] In general, a neural architecture search system 100 is a system that obtains a baseline architecture 102 of a neural network and input data 104 that defines compound coefficients 103, where the compound coefficients control additional computational resources used to scale the baseline architecture, using simple but efficient compound coefficients to uniformly scale all dimensions of the baseline architecture 102 to generate a final architecture 126 for the neural network to perform a specific machine learning task.

[0025] The system 100 may receive the baseline architecture 102 and the input data 104 in any of a variety of ways. For example, the system 100 may receive the baseline architecture 102 and the target resource usage data 104 as an upload from a remote user of the system over a data communications network, e.g., using an application programming interface (API) available to the system 100. As another example, the system 100 may receive input from a user specifying which data already maintained by the system 100 should be used as data identifying the baseline architecture 102 and which data should be used as the target resource usage data 104.

[0026] The input data 104 defines a compounding coefficient 103 that controls the additional computational resources used to scale the baseline architecture. In other words, the compounding coefficient 103 controls the amount of computational resources that can be used by the scaled final architecture compared to the resources used by the baseline architecture. In some cases, the system 100 can generate compounding coefficient values ​​based on constraints of the user. For example, if the user wants to make the baseline architecture (102) N times larger, the system can generate a compounding coefficient φ=log2(N). For example, if the user wants to scale up the baseline architecture by a factor of 4, then φ=2. If the user wants to scale up the baseline model by a factor of 32, then φ=5.

[0027] The input data 104 may also specify a target amount of computing resources used by the final architecture. In particular, the target resource usage data specifies (i) a target memory size indicating the maximum memory size allowed for creating the final architecture, i.e., the maximum memory that can be occupied by the parameters and architecture data of the final architecture, and (ii) a target number of operations (e.g., FLOPS) indicating the maximum number of operations that the final architecture can perform in order to perform a specific machine learning task.

[0028] The baseline architecture 102 is an architecture that has been trained to perform a specific machine learning task. The baseline architecture 102 has a network width dimension, a network depth dimension, and a resolution dimension. The baseline architecture 102 has multiple neural network layers. In some implementations, the baseline architecture 102 has multiple network levels, and each of the multiple network levels has multiple neural network layers. The network level includes a set of neural network layers of the same type. This means that the multiple neural network layers in each network level of the baseline architecture share the same architecture, that is, perform the same type of operations on input tensors of the same size to generate output tensors of the same size.

[0029] The network depth dimension of the baseline architecture 102 is the set of the number of layers in the multiple network levels of the baseline architecture. Each neural network layer in the baseline structure 102 is configured to receive an input tensor from the previous layer and generate an output tensor for the input tensor, which is fed as input to the next neural network layer. The input tensor of any given layer has a height dimension, a width dimension, and a channel dimension, which specifies the number of channels in the input tensor.

[0030] The network width dimension of the baseline architecture 102 is the set of the number of input channels associated with the input tensors of the multiple neural network layers of the baseline architecture 102.

[0031] The resolution dimension of the baseline architecture 102 is the set of the height dimension and the width dimension of the input tensors of the multiple neural network layers of the baseline architecture 102.

[0032] For example, the baseline architecture 102 may be represented as:

[0033]

[0034] in Presentation Layer F i Repeat L in level i i times, (H i , W i , C i ) Presentation Layer F i The shape of the input tensor X is H i is the height dimension, W i It is width dimension, and C i is the channel size.

[0035] Figure 3 An example of a baseline architecture 102 with 5 levels is shown. In this example architecture, the spatial dimensions (height and width dimensions) gradually shrink, but the channel dimensions expand across the layers, for example, from an initial input shape of (224, 224, 3) to a final output shape of (7, 7, 512).

[0036] The main focus is on finding the best layer architecture F i (That is, find the i Unlike previous methods that perform the best type of operations), the model scaling techniques described in this specification extend the network length (L i )、Width(C i ) and / or resolution (H i ; W i ) without changing the predefined F in the baseline architecture 102 i Through the fixed layer F i The model scaling techniques described here simplify the design problem for new resource constraints relative to previous methods. However, there is still a large design space to explore different L i ; C i ;H i ; W i To further reduce the design space, system 100 may constrain all layers to scale uniformly at a constant ratio.

[0037] Given a target amount of computing resources specified by input data 104, system 100 aims to maximize model accuracy for a given computing resource constraint, which can be formulated as an optimization problem as follows:

[0038]

[0039]

[0040]

[0041]

[0042] Where w, d, r are the width, depth and resolution coefficients used to scale the network width, depth and resolution size respectively; is a predefined parameter in the obtained baseline architecture 102; target_memory indicates the maximum memory size allowed for creating the final architecture, and target_flop indicates the maximum number of operations that the final architecture can perform in order to perform a specific machine learning task; represents a neural network architecture that has been created by scaling the network width, depth, and resolution dimensions of the baseline architecture 102 using the depth, width, and resolution coefficients d, w, r; FLOPS (N) represents the number of operations performed by the neural network architecture N in order to perform a specific machine learning task; Memory (N) represents the memory used to create the neural network architecture N, i.e., the memory occupied by the parameters and architecture data of the neural network architecture N.

[0043] Typically, to determine the final architecture from the baseline architecture, the system 100 employs a compound scaling method that uniformly scales the network width, depth, and resolution dimensions in a principled manner using a compound coefficient φ. For example, the compound scaling method can be expressed as:

[0044] Depth: d = α φ

[0045] Width: w = β φ

[0046] Resolution: r = γ φ (3)

[0047] stα·β 2 γ 2 ≈2

[0048] α≥1, β≥1, γ≥1

[0049] where α, β, γ are the baseline depth coefficient, baseline width coefficient, and baseline resolution coefficient, respectively. α, β, γ can be determined, for example, by a search based on a grid search of equations 2 and 3 above. Intuitively, φ controls how many more resources are available for model scaling, while α, β, γ specify how these additional resources are assigned to the network depth, width, and resolution size, respectively. It should be noted that the number of operations (i.e., FLOPS) of a conventional convolution operation is similar to d, w 2 , r 2 proportionally, i.e., doubling the network depth dimension will double the FLOPS, but doubling the network width or resolution dimension will quadruple the FLOPS. When the baseline architecture is a convolutional neural network, the convolution operation will dominate the computational cost in the baseline architecture, so scaling the baseline architecture using Equation 3 will approximately increase the total FLOPS by (α.β 2 .γ 2 ) φ The constraint α.β 2 .γ 2 ≈2 ensures that for any composite coefficient φ, the total operations (i.e. FLOPS()) will increase by approximately 2 φ .

[0050] To determine the final architecture 126, the system 100 performs a search (e.g., a grid search, a random search, or other search strategy) to search for a baseline depth coefficient α (110), a baseline width coefficient β (108), and a baseline resolution coefficient γ (112), which specify how to assign additional computing resources to the network depth dimension, network width dimension, and resolution dimension of the baseline architecture, respectively. Figure 2 Describes in detail the search that was performed.

[0051] System 100 generates width coefficient 114 based on the value of compound coefficient 103 and baseline width coefficient 108 , depth coefficient 116 based on the value of compound coefficient 103 and baseline depth coefficient 110 , and resolution coefficient 118 based on the value of compound coefficient 103 and baseline resolution coefficient 112 .

[0052] For example, in some implementations, system 100 may use Equation 3 above to generate width, depth, and resolution coefficients.

[0053] In some other implementations, as shown in Equation 4 below, the system 100 may generate a width coefficient 114 by summing the product of the baseline width coefficient 108 and the value of the composite coefficient 103 with a constant. The system 100 may generate a depth coefficient 116 by summing the product of the baseline depth coefficient 110 and the value of the composite coefficient 103 with the constant. The system 100 may generate a resolution coefficient 118 by summing the product of the baseline resolution coefficient 112 and the value of the composite coefficient 103 with the constant.

[0054] Equation 4:

[0055] Network depth coefficient: d = 1 + α.φ

[0056] Network width coefficient: w = 1 + β.φ

[0057] Network resolution coefficient: r = 1 + γ.φ

[0058] After generating the depth, width, and resolution coefficients (d, w, r) for the composite coefficients, the system 100 generates the final architecture 126 by scaling the network depth, network width, and resolution dimensions of the baseline architecture 102 based on the corresponding depth, width, and resolution coefficients. In particular, the system 100 scales the network width dimension of the baseline architecture 102 by the width coefficient, scales the network depth dimension of the baseline architecture 102 by the depth coefficient, and scales the resolution of the baseline architecture 102 by the resolution coefficient to generate the final architecture 126.

[0059] The system 100 may then output architecture data 150 that specifies the final architecture of the neural network, i.e., data that specifies the layers that are part of the final architecture, the connectivity between the layers, and the operations performed by the layers. For example, the system 100 may output the architecture data 150 to a user who submitted the training data. In some cases, the data 150 also includes training values ​​for parameters of the final architecture from the training of the candidate architecture identified as the final architecture.

[0060] In some implementations, instead of or in addition to outputting architecture data 150, system 100 trains an instance of a neural network having a final architecture (e.g., training from scratch or fine-tuning parameter values ​​generated as a result of training a candidate architecture identified as the final architecture), and then uses the trained neural network to process requests received by a user, e.g., through an API provided by system 100. That is, system 100 may receive an input to be processed, process the input using a trained neural network having a final architecture, and provide output generated by the trained neural network or data derived from the generated output in response to the received input.

[0061] Figure 2 is a flow chart of an example process for determining a final architecture of a neural network for performing a particular machine learning task. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed neural architecture search system such as Figure 1 The neural architecture search system 100 is capable of performing process 200.

[0062] The system receives a baseline architecture of a neural network (step 202). The baseline architecture has been trained to perform a specific machine learning task. The baseline architecture has a network width dimension, a network depth dimension, and a resolution dimension. The baseline architecture has a plurality of network levels, and each of the plurality of network levels has a plurality of neural network layers. The plurality of neural network layers in each network level of the baseline architecture share the same architecture.

[0063] The network depth dimension of the baseline architecture is the set of the number of layers in the multiple network levels of the baseline architecture. Each neural network layer in the baseline structure is configured to receive an input tensor from the previous layer and generate an output tensor for the input tensor, which is fed as input to the next neural network layer. The input tensor has a height dimension, a width dimension, and a channel dimension, which specifies the number of channels in the input tensor.

[0064] The network width dimension of the baseline architecture is the set of the number of input channels associated with the input tensors of multiple neural network layers of the baseline architecture.

[0065] The resolution dimension of the baseline architecture is a set of height dimensions and width dimensions of the input tensors of the plurality of neural network layers of the baseline architecture.

[0066] The system receives input data defining a compound coefficient that controls the additional computational resources used to scale the baseline architecture (step 204). In other words, the compound coefficient controls the amount of computational resources that can be used by the scaled final architecture compared to the resources used by the baseline architecture. In some cases, the system can generate compound coefficient values ​​based on constraints of the user. For example, if the user wants to make the baseline architecture (102) N times larger, the system can generate a compound coefficient φ=log2(N). For example, if the user wants to scale the baseline architecture 4 times, then φ=2. If the user wants to scale the baseline model 32 times, then φ=5.

[0067] The input data may also specify a target amount of computing resources used by the final architecture. In particular, the target resource usage data specifies (i) a target memory size indicating the maximum memory size allowed for creating the final architecture, i.e., the maximum memory that the parameters and architecture data of the final architecture can occupy, and (ii) a target number of operations indicating the maximum number of operations that the final architecture can perform in order to perform a specific machine learning task.

[0068] To generate the final architecture, the system performs a search (e.g., a grid search, a random search, or other search strategy) to determine a baseline width coefficient β, a baseline depth coefficient α, and a baseline resolution coefficient γ, which specify how to allocate additional computing resources to the network width dimension, network depth dimension, and resolution dimension of the baseline architecture, respectively (step 206). For example, assuming that the composite coefficient value is 1 and twice the resources are available for the larger model, the system performs a grid search for α, β, γ based on equations 2 and 3 and finds the optimal values ​​of these baseline coefficients as follows: 2 γ 2 Under the constraint of ≈2., α=1.2, β=1.1, γ=1.15.

[0069] To perform a grid search, the system repeatedly performs the following steps:

[0070] (a) selecting a search value for a baseline width coefficient, a search value for a baseline depth coefficient, and a search value for a baseline resolution coefficient;

[0071] (b) generating a search width coefficient w', a search depth coefficient d', and a search resolution coefficient r' based on (i) given values ​​of the composite coefficients, and (ii) search values ​​of the baseline width, depth, and resolution coefficients, respectively (e.g., by using Equations 3 or 4);

[0072] (c) generating a search candidate architecture using the baseline architecture and a search width factor, a search depth factor, and a search resolution factor (e.g., by scaling the network width dimension of the baseline architecture by the search width factor; scaling the network depth dimension of the baseline architecture by the search depth factor; and scaling the resolution of the baseline architecture by the search resolution factor); and

[0073] (d) determining a performance score for the search candidate architecture, the performance score representing the performance of the search candidate architecture on a specific machine learning task. For example, the performance score may be an accuracy score representing the accuracy of the search candidate architecture on a specific machine learning task. In particular, the system may train an instance of a neural network having a search candidate architecture on a specific machine learning task to determine the values ​​of the parameters of the instance of the neural network having the search candidate architecture. The system may then determine the accuracy score of the trained instance of the neural network based on the execution of the trained instance of the neural network on the specific neural network task. For example, the accuracy score may represent the accuracy of the trained instance on the validation set as measured by an appropriate accuracy metric. For example, when the output is a sequence, the accuracy score may be a perplexity measure, or when the specific neural network task is a classification task, the accuracy score may be a classification error rate. As another example, the accuracy score may be the average or maximum value of the accuracy of the instance for each of the last two, five, or ten epochs of training of the instance.

[0074] The system then selects the search value associated with the maximum performance score among the performance scores of all search candidate architectures that have been generated as the final value of the baseline depth, width, and resolution coefficients.

[0075] The system can select search values ​​for the baseline width, depth, and resolution coefficients at step (a) by sampling a value for each coefficient (subject to constraints) from a grid of possible values ​​to search. For example, the system can select search values ​​for the baseline width, depth, and resolution coefficients at step (a). 2 γ 2 The value of each coefficient is sampled from a grid of possible values ​​between 1 and 2 under the constraint that ≈ 2.

[0076] After obtaining the baseline width coefficient β, the baseline depth coefficient α, and the baseline resolution coefficient γ, the system determines the width coefficient, the depth coefficient, and the resolution coefficient (step 208).

[0077] In some implementations, the system can generate width, depth, and resolution coefficients based on the baseline width, depth, and resolution coefficients and the values ​​of the compound coefficients using Equation 3.

[0078] In some other implementations, the system may generate a width coefficient by summing the product of the baseline width coefficient and the value of the composite coefficient with a constant, as shown in Equation 4. The system may generate a depth coefficient by summing the product of the baseline depth coefficient and the value of the composite coefficient with the constant. The system may generate a resolution coefficient by summing the product of the baseline resolution coefficient and the value of the composite coefficient with the constant.

[0079] The system generates a final architecture by scaling the network width, network depth, and resolution dimensions of the baseline architecture based on corresponding width, depth, and resolution factors, respectively (step 210). In particular, the system scales the network width dimension of the baseline architecture by the width factor, scales the network depth dimension of the baseline architecture by the depth factor, and scales the resolution of the baseline architecture by the resolution factor to generate the final architecture.

[0080] Then, the neural network according to the identified final architecture can be used to perform the machine learning task.In addition or alternatively, the information characterizing the identified final architecture can be output (e.g., to a remote computer system) for use in building a neural network with the final architecture.

[0081] In some implementations, the system can generate multiple larger architectures by performing process 200 for multiple different compounding coefficients. This can allow the system or a user to select which larger architecture is most appropriate (e.g., which larger architecture has the best performance score among the multiple larger architectures generated) given the amount of resources available to run the architecture at any given time (e.g., given the target amount of computing resources specified by the input data).

[0082] Figure 4 Another example baseline architecture that can be used to generate various final architectures using the above-described model scaling techniques is shown. The baseline architecture 300 includes a convolutional neural network layer 302, followed by a plurality of FusedConv neural network layers 304, followed by a plurality of MBConv neural network layers 306.

[0083] FuedConv is a type of CNN layer that includes a regular CNN sublayer followed by a point-wise CNN sublayer. No depthwise CNN sublayer is used in FusedConv.

[0084] MBConv is a type of convolutional neural network layer that includes a point-wise convolutional neural network sublayer, followed by a depth-wise convolutional neural network sublayer, followed by a point-wise convolutional neural network sublayer. Examples of MBConv can be found in Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. CVPR, 2018, and Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., and Le, Q. V. M. NasNet: Platform-aware neural architecture search for mobile. CVPR, 2019.

[0085] The final architecture generated by scaling the baseline architecture 300 using the composite coefficients may also be referred to as an "EfficientNet-Edge TPU" architecture, which is customized to run optimally on a dedicated hardware accelerator designed to run on a device (e.g., on Google's Edge TPU) rather than in the cloud. With such model customization, the EfficientNet-Edge TPU architecture is able to provide real-time image classification performance while achieving accuracy typically only seen with much larger, computationally intensive models running in data centers.

[0086] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform the operation or action. For one or more computer programs configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0087] Embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, or in tangibly embodied computer software, firmware, or in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-volatile program carrier, which are used to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information sent to an appropriate receiver device for execution by the data processing device.

[0088] The term "data processing apparatus" refers to data processing hardware and covers various apparatuses, devices and machines for processing data, including, for example, a programmable processor, a computer, a multiprocessor or a computer. The apparatus may also be or further include a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to the hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0089] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script or code) may be written in any form of programming language, including a compiled or interpreted language, or a descriptive or procedural language; and the computer program may be deployed in any form, including as a standalone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A computer program may, but need not correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., stored in one or more scripts in a markup language document, a single file dedicated to the program in question, or multiple collaborative files), such as a file storing one or more modules, subroutines or code portions. A computer program may be deployed to be executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network. In this specification, the term "database" is used broadly to refer to any data collection: data does not need to be structured in any particular way or at all, and it may be stored on a storage device at one or more locations. Thus, for example, an index database may include multiple data sets, each of which may be organized and accessed differently.

[0090] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same computer or multiple computers.

[0091] The processes and logic flows described in this specification may be performed by one or more programmable processors that execute one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by a dedicated logic circuit (e.g., FPGA or ASIC), or by a combination of a dedicated logic circuit and one or more programmed computers.

[0092] A computer suitable for executing a computer program can be based on a general-purpose microprocessor or a special-purpose microprocessor, or both, or any other kind of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or merged into the special logic circuit. Typically, the computer will also include one or more large-capacity storage devices for storing data--for example, a disk, a magneto-optical disk, or an optical disk, or be operatively coupled to receive data from the one or more large-capacity storage devices or to send data to the one or more large-capacity storage devices or to receive and send both. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, for example, a universal serial bus (USB) flash drive, just to name a few examples.

[0093] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0094] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, with which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and any form of input may be received from the user, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from the device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser to interact with the user. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device, such as a smart phone running a messaging application, and receiving a response message from the user in response.

[0095] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0096] Machine learning models can be implemented and deployed using a machine learning framework such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0097] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, or includes a middleware component, or includes a front-end component, or includes any combination of one or more such back-end, middleware, or front-end components, wherein the back-end component is, for example, a data server, the middleware component is, for example, an application server, and the front-end component is, for example, a client computer with a graphical user interface, a web browser, or an app, through which a user can interact with the implementation method of the subject matter described in this specification. The components of the system may be interconnected by any digital data communication form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0098] The computing system may include a client and a server. The client and the server are generally far from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of a computer program that runs on a corresponding computer and has a client-server relationship with each other. In some embodiments, the server transmits data such as an HTML page to a user device, for example, for displaying data to a user who is interacting with the device (used as a client) and receiving user input from the user. The data generated at the user device, such as the result of the user interaction, can be received from the device at the server. Although this specification contains many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of the content that can be claimed, but rather as a description of the characteristics peculiar to the specific embodiments of the specific invention. Certain features described in the context of separate embodiments in this specification can also be implemented in a single embodiment in a combined manner. On the contrary, the various features described in the context of a single embodiment can also be implemented separately or in any appropriate sub-combination in multiple embodiments. In addition, although the above can be described as an action in some combinations and even initially claimed, one or more features from the claimed combination can be deleted from the combination in some cases, and the claimed combination can involve a sub-combination or a variation of the sub-combination.

[0099] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring such operations to be performed in the particular order shown or in a sequential order, or performing all of the operations shown to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can usually be integrated together in a single software product or packaged into multiple software products.

[0100] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or the sequential order to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A computer-implemented method for determining a final architecture of a neural network to perform a specific machine learning task, the computer-implemented method comprising: Receiving a baseline architecture of the neural network, wherein the baseline architecture has been trained to perform the specific machine learning task, and wherein the baseline architecture has a network width dimension, a network depth dimension, and a resolution dimension; receiving data defining a compounding factor that controls additional computing resources for scaling the baseline architecture; performing a search to determine a baseline width factor, a baseline depth factor, and a baseline resolution factor that specify how the additional computing resources are assigned to the network width dimension, the network depth dimension, and the resolution dimension, respectively, of the baseline architecture; determining a width coefficient, a depth coefficient, and a resolution coefficient based on the baseline width coefficient, the baseline depth coefficient, the baseline resolution coefficient, and the composite coefficient; and The final architecture is generated that scales the network width dimension, the network depth dimension, and the resolution dimension of the baseline architecture based on corresponding width coefficients, depth coefficients, and resolution coefficients.

2. The method according to claim 1, wherein: The baseline architecture has a plurality of network stages, and each of the plurality of network stages has a plurality of neural network layers.

3. The method according to claim 2, wherein: The multiple neural network layers in each network level of the baseline architecture share the same architecture.

4. The method according to claim 2, wherein: The network depth dimension of the baseline architecture is the set of the number of layers in the plurality of network levels of the baseline architecture.

5. The method according to claim 2, wherein: Each neural network layer in the baseline architecture is configured to receive an input tensor from a previous layer and generate an output tensor for the input tensor, which is fed as input to the next neural network layer, wherein the input tensor has a height dimension, a width dimension, and a channel dimension, and the channel dimension specifies the number of channels in the input tensor.

6. The method according to claim 5, wherein: The network width dimension of the baseline architecture is a set of numbers of input channels associated with input tensors to the plurality of neural network layers of the baseline architecture.

7. The method according to claim 5, wherein: The resolution dimension of the baseline architecture is a set of height dimensions and width dimensions of input tensors to the multiple neural network layers of the baseline architecture.

8. The method according to claim 1, wherein: Determining the width coefficient, the depth coefficient, and the resolution coefficient based on the baseline width coefficient, the baseline depth coefficient, the baseline resolution coefficient, and the composite coefficient includes: generating the width coefficient based on the composite coefficient and the baseline width coefficient; generating the depth coefficient based on the composite coefficient and the baseline depth coefficient; and The resolution coefficient is generated based on the composite coefficient and the baseline resolution coefficient.

9. The method according to claim 8, wherein: Generating the width coefficient based on the compounding coefficient and the baseline width coefficient includes summing a product of the baseline width coefficient and the compounding coefficient with a constant.

10. The method according to claim 8, wherein: Generating the depth coefficient based on the compound coefficient and the baseline depth coefficient includes summing a product of the baseline depth coefficient and the compound coefficient with a constant.

11. The method according to claim 8, wherein: Generating the resolution coefficient based on the composite coefficient and the baseline resolution coefficient includes summing a product of the baseline resolution coefficient and the composite coefficient with a constant.

12. The method according to claim 11, wherein: Generating the final architecture includes: Scaling the network width dimension of the baseline architecture by the width factor; Scaling the network depth dimension of the baseline architecture by the depth factor; and The resolution of the baseline framework is scaled by the resolution factor.

13. The method according to claim 1, wherein: Performing the search to determine the baseline width coefficient, the baseline depth coefficient, and the baseline resolution coefficient includes: A grid search is performed on the value range of each coefficient while using the composite coefficient to determine the baseline width coefficient, the baseline depth coefficient, and the baseline resolution coefficient.

14. The method according to claim 1, further comprising: Determining a performance score representing the performance of the final architecture on the specific machine learning task includes: training the final architecture on the specific machine learning task to update parameter values ​​of the final architecture, and determining the performance of the trained final architecture on the specific machine learning task.

15. The method according to any one of claims 1 to 14, wherein: The received data also includes target resource usage data, which specifies (i) a target memory size, which indicates the maximum memory size allowed for creating the final architecture, and (ii) a target number of operations, which indicates the maximum number of operations that the final architecture can perform to perform the specific machine learning task.

16. A system comprising one or more computers and one or more storage devices storing instructions which, when executed by the one or more computers, cause the one or more computers to perform the operations of the method according to any one of claims 1 to 15.

17. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 15.

Images