Composite model scaling for neural networks

Abstract

The invention discloses composite model scaling for neural networks. A method for determining a final architecture of a neural network for performing a particular machine learning task includes receiving a baseline architecture for the neural network, where the baseline architecture has a network width dimension, a network depth dimension, and a resolution dimension; receiving data defining complex coefficients that control additional computing resources for scaling the baseline architecture; performing a search to determine a baseline width coefficient, a baseline depth coefficient, and a baseline resolution coefficient; determining a width coefficient, a depth coefficient, and a resolution coefficient based on the baseline width coefficient, the baseline depth coefficient, the baseline resolution coefficient, and the composite coefficient; and generating a final architecture that scales the network width dimension, the network depth dimension, and the resolution dimension of the baseline architecture based on the corresponding width coefficients, depth coefficients, and resolution coefficients.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 202080010508.3, filed on January 23, 2020.

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS

[0004] This application claims priority to U.S. Provisional Application Serial No. 62 / 796,034, filed on January 23, 2019. The disclosure of the prior application is considered part of and incorporated by reference into the disclosure of the present application. Technical Field

[0005] This specification relates to determining the architecture of a convolutional neural network. Background Art

[0006] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output based on a given input. Some neural networks also include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input for the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from its input based on the current values of its corresponding parameter set.

[0007] Some neural networks are recurrent neural networks. A recurrent neural network is a neural network that receives an input sequence and generates an output sequence from the input sequence. In particular, a recurrent neural network can use some or all of the internal state of the network from previous time steps when computing the output at the current time step. An example of a recurrent neural network is a long short-term memory (LSTM) neural network that includes one or more LSTM memory blocks. Each LSTM memory block can include one or more cells that each include an input gate, a forget gate, and an output gate, which allows the cell to store the previous state of the cell, for example, to generate a current activation or to be provided to other components of the LSTM neural network. Summary of the Invention

[0008] This specification describes a system implemented as a computer program on one or more computers in one or more locations that determines a final architecture for a neural network based on a baseline architecture. As described below, the final architecture is determined based on a target computational resource usage for the final architecture. In this way, the system can determine a final architecture that is suitable for the low-level computational resources (e.g., number of floating point operations per second, FLOPS) available for the final architecture. Furthermore, it can do so in a more efficient manner than conventional methods that may require extensive hyperparameter tuning. The neural network is configured to perform a specific machine learning task, such as, but not limited to, image processing (e.g., image classification).

[0009] The subject matter described in this specification can be implemented in certain embodiments to achieve one or more of the following advantages. Convolutional neural networks are typically developed with a fixed computational cost and then scaled up to larger networks for better accuracy given more resources. The model scaling techniques described herein use simple but efficient compound coefficients to scale a baseline neural network model to a target model, given target resource constraints, while maintaining model efficiency. Conventional approaches arbitrarily increase network width, depth, or resolution when scaling, which results in low accuracy and efficiency, and makes scaling up neural network models inefficient and computationally expensive due to the hyperparameter adjustments required to balance these three factors. In contrast, the described techniques use compound coefficients to linearly scale the network width, depth, and resolution of the baseline model, avoiding the need to expend significant computational resources on hyperparameter adjustments. Furthermore, the model scaling techniques described herein allow baseline models to be scaled up very efficiently, resulting in larger models that exceed the accuracy of the state of the art while having fewer parameters than previous state-of-the-art models. At the same time, these larger models can perform specific machine learning tasks using fewer operations than previous state-of-the-art models, and are therefore faster to train and infer.

[0010] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An example neural architecture search system for determining a final architecture of a neural network to perform a specific machine learning task is shown.

[0012] Figure 2 is a flowchart of an example process for determining the final architecture of a neural network for performing a specific machine learning task.

[0013] Figure 3 An example baseline architecture of a neural network is shown.

[0014] Figure 4 Another example baseline architecture of a neural network is shown.

[0015] Like reference numbers and designations throughout the various drawings represent like elements. DETAILED DESCRIPTION

[0016] This specification describes a neural architecture search system implemented as a computer program on one or more computers in one or more locations that determines a final architecture for a neural network configured to perform a specific machine learning task based on a baseline architecture.

[0017] Typically, a neural network is configured to receive a network input and process the network input to generate a network output for the input.

[0018] In some cases, the neural network is a convolutional neural network that is configured to receive an input image and process the input image to generate a network output for the input image, i.e., perform some image processing task.

[0019] For example, the task may be image classification, and the output generated by the neural network for a given image may be a score for each of a set of object categories, where each score represents an estimated likelihood that the image contains an object belonging to that category.

[0020] As another example, the task can be image embedding generation, and the output generated by the neural network can be a numerical embedding of the input image. For example, the proximity in the embedding space between the embeddings generated for corresponding input images can represent the similarity between these images.

[0021] As yet another example, the task may be object detection, and the output generated by the neural network may identify locations in the input image where a particular type of object is depicted.

[0022] In some other cases, the task may be video classification, and the neural network is configured to receive as input a video or portion of a video and generate an output that determines what subject matter the input video or portion of a video relates to.

[0023] In some other cases, the task may be speech recognition, and the neural network is configured to receive audio data as input and generate an output that determines, for a given spoken utterance, one or more words that the utterance represents.

[0024] In some other cases, the task may be text classification, and the neural network is configured to receive an input text segment and generate an output that determines what topic the input text segment relates to.

[0025] Figure 1An example neural architecture search system 100 is shown that is configured to determine a final architecture for a neural network configured to perform a specific machine learning task based on a baseline architecture. The neural architecture search system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below can be implemented.

[0026] In general, a neural architecture search system 100 is a system that obtains a baseline architecture 102 of a neural network and input data 104 that defines compound coefficients 103, where the compound coefficients control additional computational resources used to scale the baseline architecture, using simple but efficient compound coefficients to uniformly scale all dimensions of the baseline architecture 102 to produce a final architecture 126 for the neural network to perform a specific machine learning task.

[0027] The system 100 can receive the baseline architecture 102 and the input data 104 in any of a variety of ways. For example, the system 100 can receive the baseline architecture 102 and the target resource usage data 104 as an upload from a remote user of the system over a data communications network, e.g., using an application programming interface (API) available to the system 100. As another example, the system 100 can receive input from the user specifying which data already maintained by the system 100 should be used as data identifying the baseline architecture 102 and which data should be used as the target resource usage data 104.

[0028] The input data 104 defines a compounding coefficient 103 that controls the additional computational resources used to scale the baseline architecture. In other words, the compounding coefficient 103 controls the amount of computational resources that can be used by the scaled final architecture compared to the resources used by the baseline architecture. In some cases, the system 100 can generate compounding coefficient values based on user constraints. For example, if the user wants to make the baseline architecture (102) N times larger, the system can generate a compounding coefficient φ = log2(N). For example, if the user wants to scale the baseline architecture by a factor of 4, then φ = 2. If the user wants to scale the baseline model by a factor of 32, then φ = 5.

[0029] The input data 104 may also specify a target amount of computing resources to be used by the final architecture. Specifically, the target resource usage data specifies (i) a target memory size, which indicates the maximum memory size allowed for creating the final architecture, i.e., the maximum memory that can be occupied by the parameters and architecture data of the final architecture, and (ii) a target number of operations (e.g., FLOPS), which indicates the maximum number of operations that the final architecture can perform to perform a specific machine learning task.

[0030] The baseline architecture 102 is an architecture that has been trained to perform a specific machine learning task. The baseline architecture 102 has a network width dimension, a network depth dimension, and a resolution dimension. The baseline architecture 102 has multiple neural network layers. In some implementations, the baseline architecture 102 has multiple network levels, and each of the multiple network levels has multiple neural network layers. The network levels include a set of neural network layers of the same type. This means that the multiple neural network layers in each network level of the baseline architecture share the same architecture, that is, perform the same type of operations on input tensors of the same size to generate output tensors of the same size.

[0031] The network depth dimension of the baseline architecture 102 is the sum of the number of layers in the multiple network levels of the baseline architecture. Each neural network layer in the baseline structure 102 is configured to receive an input tensor from the previous layer and generate an output tensor for the input tensor, which is fed as input to the next neural network layer. The input tensor of any given layer has a height dimension, a width dimension, and a channel dimension, which specifies the number of channels in the input tensor.

[0032] The network width dimension of the baseline architecture 102 is the set of the number of input channels associated with the input tensors of the multiple neural network layers of the baseline architecture 102.

[0033] The resolution size of the baseline architecture 102 is the set of the height size and the width size of the input tensors of the multiple neural network layers of the baseline architecture 102.

[0034] For example, the baseline architecture 102 may be represented as:

[0035]

[0036] in Presentation Layer F i Repeat L in level i i times, (H i , W i , C i ) represents layer F i The shape of the input tensor X is H i is the height dimension, W i It is width size, and C i is the channel size.

[0037] Figure 3 An example of a baseline architecture 102 with 5 levels is shown. In this example architecture, the spatial dimensions (height and width dimensions) gradually shrink, but the channel dimensions expand across the layers, for example, from an initial input shape of (224, 224, 3) to a final output shape of (7, 7, 512).

[0038] With the main focus on finding the best layer architecture F i (ie, find the i Unlike previous methods that perform the best type of operations), the model scaling techniques described in this specification extend the network length (L i )、Width(C i ) and / or resolution (H i ;W i ) without changing the predefined F in the baseline architecture 102 i Through the fixed layer F i The model scaling techniques described here simplify the design problem for new resource constraints relative to previous methods. However, there is still a large design space to explore different L i ; C i ;H i ;W i To further reduce the design space, system 100 can constrain all layers to scale uniformly at a constant ratio.

[0039] Given a target amount of computing resources specified by input data 104, system 100 aims to maximize model accuracy for given computing resource constraints, which can be formulated as an optimization problem as follows:

[0040]

[0041] Where w, d, and r are the width, depth, and resolution coefficients used to scale the network width, depth, and resolution size respectively; is a predefined parameter in the obtained baseline architecture 102; target_memory indicates the maximum memory size allowed for creating the final architecture, and target_flop indicates the maximum number of operations that the final architecture can perform in order to perform a specific machine learning task; represents a neural network architecture that has been created by scaling the network width, depth, and resolution dimensions of the baseline architecture 102 using depth, width, and resolution coefficients d, w, r; FLOPS (N) represents the number of operations performed by the neural network architecture N in order to perform a specific machine learning task; Memory (N) represents the memory used to create the neural network architecture N, that is, the memory occupied by the parameters and architecture data of the neural network architecture N.

[0042] Typically, to determine the final architecture based on the baseline architecture, the system 100 employs a compound scaling method that uniformly scales the network width, depth, and resolution dimensions in a principled manner using a compound coefficient φ. For example, the compound scaling method can be expressed as:

[0043]

[0044] where α, β, γ are the baseline depth coefficient, baseline width coefficient, and baseline resolution coefficient, respectively. α, β, γ can be determined, for example, by a grid search based on Equations 2 and 3 above. Intuitively, φ controls how many more resources are available for model scaling, while α, β, γ specify how these additional resources are assigned to the network depth, width, and resolution sizes, respectively. Note that the number of operations (i.e., FLOPS) of a conventional convolution operation is similar to d,w 2 ,r 2 Proportional, that is, doubling the network depth dimension will double the FLOPS, but doubling the network width or resolution dimension will quadruple the FLOPS. When the baseline architecture is a convolutional neural network, the convolution operation will dominate the computational cost in the baseline architecture, so scaling the baseline architecture using Equation 3 will approximately increase the total FLOPS by (α.β 2 γ 2 ) φ The constraint α.β 2 γ 2 ≈2 ensures that for any composite coefficient φ, the total operations (i.e. FLOPS()) will increase by approximately 2 φ .

[0045] To determine the final architecture 126, the system 100 performs a search (e.g., a grid search, a random search, or other search strategy) to search for a baseline depth coefficient α (110), a baseline width coefficient β (108), and a baseline resolution coefficient γ (112), which specify how to allocate additional computational resources to the network depth dimension, the network width dimension, and the resolution dimension, respectively, of the baseline architecture. Figure 2 Describes in detail the search performed.

[0046] System 100 generates width coefficient 114 based on the value of compound coefficient 103 and baseline width coefficient 108 , depth coefficient 116 based on the value of compound coefficient 103 and baseline depth coefficient 110 , and resolution coefficient 118 based on the value of compound coefficient 103 and baseline resolution coefficient 112 .

[0047] For example, in some implementations, system 100 may use Equation 3 above to generate width, depth, and resolution coefficients.

[0048] In some other implementations, as shown below in Equation 4, the system 100 may generate a width coefficient 114 by summing the product of the baseline width coefficient 108 and the value of the composite coefficient 103 with a constant. The system 100 may generate a depth coefficient 116 by summing the product of the baseline depth coefficient 110 and the value of the composite coefficient 103 with the constant. The system 100 may generate a resolution coefficient 118 by summing the product of the baseline resolution coefficient 112 and the value of the composite coefficient 103 with the constant.

[0049] Equation 4:

[0050] Network depth coefficient: d = 1 + αφ

[0051] Network width coefficient: w=1+β.φ

[0052] Network resolution coefficient: r = 1 + γ.φ

[0053] After generating the depth, width, and resolution coefficients (d, w, r) for the composite coefficients, the system 100 generates the final architecture 126 by scaling the network depth, network width, and resolution dimensions of the baseline architecture 102 based on the corresponding depth, width, and resolution coefficients. In particular, the system 100 scales the network width dimension of the baseline architecture 102 by the width coefficient, scales the network depth dimension of the baseline architecture 102 by the depth coefficient, and scales the resolution of the baseline architecture 102 by the resolution coefficient to generate the final architecture 126.

[0054] The system 100 may then output architecture data 150 that specifies the final architecture of the neural network, i.e., data that specifies the layers that are part of the final architecture, the connectivity between the layers, and the operations performed by the layers. For example, the system 100 may output architecture data 150 to a user who submitted the training data. In some cases, the data 150 also includes training values for parameters of the final architecture from the training of the candidate architecture identified as the final architecture.

[0055] In some implementations, instead of or in addition to outputting architecture data 150, system 100 trains an instance of a neural network having the final architecture (e.g., training from scratch or fine-tuning parameter values generated as a result of training a candidate architecture identified as the final architecture), and then uses the trained neural network to process requests received by a user, e.g., through an API provided by system 100. That is, system 100 can receive an input to be processed, process the input using the trained neural network having the final architecture, and provide output generated by the trained neural network or data derived from the generated output in response to the received input.

[0056] Figure 2is a flow chart of an example process for determining a final architecture of a neural network for performing a particular machine learning task. For convenience, process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed neural architecture search system such as Figure 1 The neural architecture search system 100 is capable of performing process 200.

[0057] The system receives a baseline architecture for a neural network (step 202). The baseline architecture has been trained to perform a specific machine learning task. The baseline architecture has a network width dimension, a network depth dimension, and a resolution dimension. The baseline architecture has multiple network levels, and each of the multiple network levels has multiple neural network layers. The multiple neural network layers in each network level of the baseline architecture share the same architecture.

[0058] The network depth dimension of the baseline architecture is the sum of the number of layers in the multiple network levels of the baseline architecture. Each neural network layer in the baseline structure is configured to receive an input tensor from the previous layer and generate an output tensor for the input tensor, which is fed as input to the next neural network layer. The input tensor has a height dimension, a width dimension, and a channel dimension, which specifies the number of channels in the input tensor.

[0059] The network width dimension of the baseline architecture is the set of the number of input channels associated with the input tensors of multiple neural network layers of the baseline architecture.

[0060] The resolution size of the baseline architecture is a set of height dimensions and width dimensions of the input tensors of the plurality of neural network layers of the baseline architecture.

[0061] The system receives input data defining a compounding coefficient that controls the additional computational resources used to scale the baseline architecture (step 204). In other words, the compounding coefficient controls the amount of computational resources that can be used by the scaled final architecture compared to the resources used by the baseline architecture. In some cases, the system can generate compounding coefficient values based on user constraints. For example, if the user wants to make the baseline architecture (102) N times larger, the system can generate a compounding coefficient φ = log2(N). For example, if the user wants to scale the baseline architecture 4 times, then φ = 2. If the user wants to scale the baseline model 32 times, then φ = 5.

[0062] The input data may also specify a target amount of computing resources to be used by the final architecture. Specifically, the target resource usage data specifies (i) a target memory size, which indicates the maximum memory size allowed for creating the final architecture, i.e., the maximum memory that the parameters and architecture data of the final architecture can occupy, and (ii) a target number of operations, which indicates the maximum number of operations that the final architecture can perform to perform a specific machine learning task.

[0063] To generate the final architecture, the system performs a search (e.g., a grid search, a random search, or other search strategy) to determine a baseline width coefficient β, a baseline depth coefficient α, and a baseline resolution coefficient γ, which specify how to allocate additional computational resources to the network width dimension, network depth dimension, and resolution dimension, respectively, of the baseline architecture (step 206). For example, assuming the composite coefficient value is 1 and twice the resources are available for the larger model, the system performs a grid search on α, βγ based on equations 2 and 3 and finds the optimal values of these baseline coefficients as follows: 2 γ 2 Under the constraint of ≈2., α=1.2, β=1.1, γ=1.15.

[0064] To perform a grid search, the system repeatedly performs the following steps:

[0065] (a) selecting a search value for a baseline width coefficient, a search value for a baseline depth coefficient, and a search value for a baseline resolution coefficient;

[0066] (b) generating a search width coefficient w', a search depth coefficient d', and a search resolution coefficient r' based on (i) the given value of the composite coefficient, and (ii) the search values of the baseline width, depth, and resolution coefficients, respectively (e.g., by using Equation 3 or 4);

[0067] (c) generating a search candidate architecture using the baseline architecture and a search width factor, a search depth factor, and a search resolution factor (e.g., by scaling the network width dimension of the baseline architecture by the search width factor; scaling the network depth dimension of the baseline architecture by the search depth factor; and scaling the resolution of the baseline architecture by the search resolution factor); and

[0068] (d) determining a performance score for the search candidate architecture, the performance score representing the performance of the search candidate architecture on a specific machine learning task. For example, the performance score may be an accuracy score representing the accuracy of the search candidate architecture on the specific machine learning task. In particular, the system may train an instance of a neural network having the search candidate architecture on the specific machine learning task to determine values for parameters of the instance of the neural network having the search candidate architecture. The system may then determine an accuracy score for the trained instance of the neural network based on the execution of the trained instance of the neural network on the specific neural network task. For example, the accuracy score may represent the accuracy of the trained instance on a validation set as measured by an appropriate accuracy metric. For example, when the output is a sequence, the accuracy score may be a perplexity measure, or when the specific neural network task is a classification task, the accuracy score may be a classification error rate. As another example, the accuracy score may be the average or maximum value of the accuracy of the instance for each of the last two, five, or ten epochs of training for the instance.

[0069] The system then selects the search value associated with the maximum performance score among the performance scores of all search candidate architectures that have been generated as the final value of the baseline depth, width, and resolution coefficients.

[0070] The system can select search values for the baseline width, depth, and resolution coefficients at step (a) by sampling the value of each coefficient (subject to constraints) from the grid of possible values to be searched. For example, the system can select search values for the baseline width, depth, and resolution coefficients at step (a). 2 γ 2 The value of each coefficient is sampled from a grid of possible values between 1 and 2, under the constraint that ≈ 2.

[0071] After obtaining the baseline width coefficient β, the baseline depth coefficient α, and the baseline resolution coefficient γ, the system determines the width coefficient, the depth coefficient, and the resolution coefficient (step 208).

[0072] In some implementations, the system can generate width, depth, and resolution coefficients based on the baseline width, depth, and resolution coefficients and the values of the composite coefficients using Equation 3.

[0073] In some other implementations, the system may generate a width coefficient by summing the product of the baseline width coefficient and the value of the composite coefficient with a constant, as shown in Equation 4. The system may generate a depth coefficient by summing the product of the baseline depth coefficient and the value of the composite coefficient with the constant. The system may generate a resolution coefficient by summing the product of the baseline resolution coefficient and the value of the composite coefficient with the constant.

[0074] The system generates a final architecture by scaling the network width, network depth, and resolution dimensions of the baseline architecture based on corresponding width, depth, and resolution factors, respectively (step 210). Specifically, the system scales the network width dimension of the baseline architecture by the width factor, scales the network depth dimension of the baseline architecture by the depth factor, and scales the resolution of the baseline architecture by the resolution factor to generate the final architecture.

[0075] Then, the neural network according to the identified final architecture can be used to perform the machine learning task. Additionally or alternatively, information characterizing the identified final architecture can be output (e.g., to a remote computer system) for use in building a neural network having the final architecture.

[0076] In some implementations, the system can generate multiple larger architectures by performing process 200 for multiple different compounding coefficients. This can allow the system or a user to select which larger architecture is most suitable (e.g., which larger architecture has the best performance score among the generated multiple larger architectures) given the amount of resources available to run the architecture at any given time (e.g., given the target amount of computing resources specified by the input data).

[0077] Figure 4 Another example baseline architecture that can be used to generate various final architectures using the above-described model scaling techniques is shown. The baseline architecture 300 includes a convolutional neural network layer 302, followed by multiple FusedConv neural network layers 304, followed by multiple MBConv neural network layers 306.

[0078] FuedConv is a type of convolutional neural network layer that includes a regular convolutional neural network sublayer followed by a point-wise convolutional neural network sublayer. No depthwise convolutional neural network sublayer is used in FusedConv.

[0079] MBConv is a type of convolutional neural network layer that includes a point-wise convolutional neural network sublayer, followed by a depth-wise convolutional neural network sublayer, followed by a point-wise convolutional neural network sublayer. Examples of MBConv can be found in Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. CVPR, 2018, and Tan, M., Chen, B., Pang, R., Vasudevan, V., Sandler, M., Howard, A., and Le, QVMnasNet: Platform-aware neural architecture search for mobile. CVPR, 2019.

[0080] The final architecture generated by scaling the baseline architecture 300 using composite coefficients can also be referred to as an "EfficientNet-Edge TPU" architecture, which is customized to run optimally on a dedicated hardware accelerator designed to run on a device (e.g., on Google's Edge TPU) rather than in the cloud. Through such model customization, the EfficientNet-Edge TPU architecture is able to provide real-time image classification performance while achieving accuracy typically seen only with much larger, computationally intensive models running in data centers.

[0081] This specification uses the term "configuration" in conjunction with system and computer program components. When referring to a system of one or more computers configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when in operation, causes the system to perform the operation or action. When referring to one or more computer programs configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0082] Embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuitry, or in tangibly embodied computer software, firmware, or in computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of these. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-volatile program carrier, the one or more modules being used for execution by a data processing apparatus or for controlling the operation of the data processing apparatus. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to an appropriate receiver apparatus for execution by the data processing apparatus.

[0083] The term "data processing apparatus" refers to data processing hardware and encompasses a variety of devices, equipment, and machines for processing data, including, for example, a programmable processor, a computer, a multiprocessor, or a computer. The apparatus may also be or further include specialized logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0084] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program can, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple collaborative files), such as a file that stores one or more modules, subroutines, or code portions. A computer program can be deployed to be executed on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a communication network. In this specification, the term "database" is used broadly to refer to any collection of data: the data does not need to be structured in any particular way or at all, and it can be stored on a storage device in one or more locations. Thus, for example, an index database may include multiple data sets, each of which may be organized and accessed differently.

[0085] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same computer or multiple computers.

[0086] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, such as an FPGA or ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0087] A computer suitable for executing a computer program can be based on a general-purpose microprocessor or a special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit (CPU) for executing instructions and one or more memory devices for storing instructions and data. The CPU and memory can be supplemented by or incorporated into a dedicated logic circuit. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to receive data from or send data to or receive and send data to the one or more mass storage devices. However, a computer need not have such a device. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few examples.

[0088] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0089] To provide for interaction with a user, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, with which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and any form of input can be received from the user, including sound, voice, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user, for example, by sending a web page to a web browser on the user's client device in response to a request received from the web browser. In addition, a computer can interact with a user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0090] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0091] Machine learning models can be implemented and deployed using a machine learning framework such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0092] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, or includes a middleware component, or includes a front-end component, or includes any combination of one or more such back-end, middleware, or front-end components, wherein the back-end component is, for example, a data server, the middleware component is, for example, an application server, and the front-end component is, for example, a client computer with a graphical user interface, a web browser, or an app, through which a user can interact with the implementation method of the subject matter described in this specification. The components of the system can be interconnected by any digital data communication form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0093] A computing system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship arises through computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data, such as HTML pages, to a user device, for example, to display data to a user interacting with the device (acting as a client) and to receive user input from the user. Data generated at the user device, such as the results of user interaction, can be received from the device at the server. Although this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of what may be claimed, but rather as describing features unique to specific embodiments of a particular invention. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable subcombination in multiple embodiments. In addition, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination may be deleted from that combination in some cases, and a claimed combination may involve subcombinations or variations of subcombinations.

[0094] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all of the illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0095] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A neural network system implemented by one or more computers, the neural network system comprising: an output neural network generated by scaling a network width dimension, a network depth dimension, and a resolution dimension of a baseline neural network architecture based on corresponding width factors, depth factors, and resolution factors, respectively, The baseline neural network architecture includes a convolutional neural network layer, followed by multiple FusedConv neural network layers, followed by multiple MBConv neural network layers.

2. The neural network system according to claim 1, wherein: The FusedConv neural network layer consists of a convolutional neural network sublayer followed by a point-wise convolutional neural network sublayer.

3. The neural network system according to claim 1, wherein: The MBConv neural network layer consists of a point-wise convolutional neural network sublayer, followed by a depth-wise convolutional neural network sublayer, followed by a point-wise convolutional neural network sublayer.

4. The neural network system according to claim 1, wherein: The convolutional neural network layer is a 3x3 convolutional neural network layer.

5. The neural network system according to claim 1, wherein: The multiple FusedConv neural network layers include seven FusedConv neural network layers.

6. The neural network system according to claim 1, wherein: The multiple FusedConv neural network layers include multiple 3x3 FusedConv neural network layers.

7. The neural network system according to claim 1, wherein: The multiple MBConv neural network layers include eleven MBConv neural network layers.

8. The neural network system according to claim 1, wherein: The multiple MBConv neural network layers include multiple 5x5 MBConv neural network layers.

9. The neural network system according to claim 1, wherein: The output neural network is configured to operate on one or more specialized hardware accelerators designed to run on the device.

10. The neural network system according to claim 1, wherein: The output neural network is configured to perform a machine learning task.

11. The neural network system according to claim 10, wherein: The machine learning task is one of the following tasks: (i) an image classification task, (ii) an image embedding generation task, (iii) an object detection task, (iv) a video classification task, (v) a speech recognition task, or (vi) a text classification task.

12. A computer-implemented method comprising: receiving a baseline neural network architecture comprising a convolutional neural network layer followed by a plurality of FusedConv neural network layers followed by a plurality of MBConv neural network layers; as well as Generates an output neural network based on the received baseline neural network architecture, including: The network width dimension, network depth dimension, and resolution dimension of the baseline neural network architecture are scaled based on the corresponding width coefficient, depth coefficient, and resolution coefficient, respectively.

13. The computer-implemented method of claim 12, wherein: The FusedConv neural network layer consists of a convolutional neural network sublayer followed by a point-wise convolutional neural network sublayer.

14. The computer-implemented method of claim 12, wherein: The MBConv neural network layer consists of a point-wise convolutional neural network sublayer, followed by a depth-wise convolutional neural network sublayer, followed by a point-wise convolutional neural network sublayer.

15. The computer-implemented method of claim 12, wherein: The convolutional neural network layer is a 3x3 convolutional neural network layer.

16. The computer-implemented method of claim 12, further comprising: Receive network input; as well as The network input is processed using the output neural network to generate a network output for the network input to perform a machine learning task.

17. The computer-implemented method of claim 16, wherein: The machine learning task is one of the following tasks: (i) an image classification task, (ii) an image embedding generation task, (iii) an object detection task, (iv) a video classification task, (v) a speech recognition task, or (vi) a text classification task.

18. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving a baseline neural network architecture comprising a convolutional neural network layer followed by a plurality of FusedConv neural network layers followed by a plurality of MBConv neural network layers; as well as Generates an output neural network based on the received baseline neural network architecture, including: The network width dimension, network depth dimension, and resolution dimension of the baseline neural network architecture are scaled based on the corresponding width coefficient, depth coefficient, and resolution coefficient, respectively.

19. The one or more non-transitory computer-readable storage media of claim 18, wherein: The operations further include: receiving network input; and The network input is processed using the output neural network to generate a network output for the network input to perform a machine learning task.

20. The one or more non-transitory computer-readable storage media of claim 19, wherein: The machine learning task is one of the following tasks: (i) an image classification task, (ii) an image embedding generation task, (iii) an object detection task, (iv) a video classification task, (v) a speech recognition task, or (vi) a text classification task.

Images