Neural architecture search for dense image prediction tasks

Abstract

Methods, systems and apparatus for determining a neural network architecture, including a computer program encoded on a computer storage medium. One method includes obtaining training data for a dense image prediction task; and determining an architecture of a neural network configured to perform the dense image prediction task, including: searching a space of candidate architectures using the training data to identify one or more best performing architectures, wherein each candidate architecture in the space of candidate architectures includes: (i) a same first neural network backbone configured to receive an input image and process the input image to generate a plurality of feature maps, and (ii) different dense prediction units configured to process the plurality of feature maps and generate outputs for the dense image prediction task; and determining the architecture of the neural network based on the best performing candidate architecture.

Description

Background Art

[0001] This specification relates to determining the architecture of a neural network that performs image processing tasks.

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer in the network (i.e., the next hidden layer or output layer). Each layer of the network generates an output from the received input based on the current values ​​of the corresponding set of parameters. Summary of the invention

[0003] This specification describes a system implemented as a computer program on one or more computers at one or more locations that determines an architecture of a neural network configured to perform a dense image prediction task.

[0004] In general, dense image prediction tasks are tasks that require a corresponding prediction for each of a large number of pixels in an input image. Many dense image prediction tasks require assigning labels or otherwise making predictions for each pixel of an image. Therefore, dense image prediction tasks often require neural networks to operate on high-resolution images.

[0005] To determine the architecture, the system obtains training data for a dense image prediction task. The system then uses the training data to determine the architecture of a neural network that is configured to perform the dense image prediction task, i.e., receive an input and generate an output that meets the requirements of the dense prediction task.

[0006] Specifically, the system searches the space of candidate architectures using the training data to identify one or more best performing architectures. Each candidate architecture in the space of candidate architectures includes (i) the same first neural network backbone configured to receive an input image and process the input image to generate a plurality of feature maps, and (ii) a different dense prediction unit configured to process the plurality of feature maps and generate an output for a dense image prediction task. Thus, each candidate architecture includes the same neural network backbone as each other candidate architecture, but has a different dense prediction unit than each other candidate architecture.

[0007] The system then determines the architecture of the neural network based on the best performing candidate architectures.

[0008] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0009] The architecture search techniques described in this specification can determine high-performance architectures for dense prediction tasks in a computationally efficient manner. Specifically, since dense prediction tasks require the generation of predictions for a large number of pixels in the input image, such tasks require the network to operate on high-resolution images. This makes existing architecture search techniques (e.g., techniques suitable for image classification or other non-dense image processing tasks) unsuitable for these tasks. This is because many of these tasks rely on low-resolution proxy tasks that are not representative of the final dense image prediction task, or the search space that needs to be searched is so large that such a search is computationally infeasible when operating on high-resolution images.

[0010] On the other hand, the described techniques effectively restrict the search space to identify the best architecture for dense prediction units, resulting in architectures that outperform the previous state of the art in multiple dense image prediction tasks.

[0011] Furthermore, by using the described techniques, the resulting architectures can be more computationally efficient than prior state-of-the-art models while outperforming them. As an example of the kinds of results that can be achieved with the described techniques, the resulting architectures can achieve state-of-the-art performance on several dense prediction tasks, including 82.7% mIOU accuracy on the Cityscapes dataset (street scene parsing), 71.3% mIOU accuracy on the PASCAL-People Parts dataset (people part segmentation), and 87.9% mIOU accuracy on the PASCAL VOC 2012 dataset (semantic image segmentation). At the same time, the resulting architectures are computationally more efficient, requiring approximately half the parameters and half the computational cost of prior state-of-the-art systems for these datasets.

[0012] Additionally, the amount of resources consumed by the search process can be reduced by using a smaller backbone during the search than will be included in the final architecture. Additionally, the amount of resources consumed by the search process can be reduced by pre-training the backbone, for example, on an object segmentation task, and then keeping the backbone fixed during the search. Additionally, the amount of resources consumed by the search can be reduced by pre-computing and then caching feature maps generated by the pre-trained backbone. As a specific example, when the backbone is smaller and the feature maps generated by the pre-trained backbone have been pre-computed and cached, the system can perform the search with less latency and higher data efficiency (i.e., using less memory) than techniques that rely on training candidate neural networks without pre-caching inputs and without reducing the size of the backbone.

[0013] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 An example neural architecture search system is shown.

[0015] Figure 2 Example candidate architectures are shown.

[0016] Figure 3 is a flow chart of an example process for determining the final architecture.

[0017] Figure 4 is a flow chart of an example process for searching a space of candidate architectures.

[0018] Like reference numbers and designations throughout the various drawings indicate like elements. DETAILED DESCRIPTION

[0019] This specification describes a system implemented as a computer program on one or more computers at one or more locations that determines an architecture of a neural network configured to perform a dense image prediction task.

[0020] In general, dense image prediction tasks are tasks that require a corresponding prediction for each pixel in a large number of pixels in an input image. Many dense image prediction tasks require assigning labels or otherwise making predictions for each pixel of an image. Therefore, dense image prediction tasks often require neural networks to operate on high-resolution images.

[0021] An example of a dense image prediction task is the image segmentation task. In a segmentation task, the input is an image and the output is a label for each pixel in the image that is used to classify what is described at that pixel in the image.

[0022] An example of an image segmentation task is the person part segmentation task, where the input is an image of one or more persons and the output is a corresponding label for each pixel in the image, such that the labels classify which pixels correspond to which person parts (e.g., head, torso, legs, etc.) and which correspond to the background (i.e., no person is depicted).

[0023] Another example of an image segmentation task is the semantic image segmentation task, in which the input is an image and the output is a corresponding label for each pixel in the image, which identifies, for example, which object category the pixel belongs to from a set of multiple foreground object categories and one or more background object categories.

[0024] Another example of an image segmentation task is the scene parsing task, in which the input is an image and the output is a corresponding label for each pixel in the image, identifying which part of the scene depicted in the image that pixel belongs to.

[0025] Another example of a dense image prediction task is an object detection task. In an object detection task, the input is an image and the output is data that specifies which pixels of the image are part of the image of an object. For example, the output may be a label for each pixel in the image that identifies whether the pixel is part of the image of an object. As another example, the output may be a score for each of a large number of bounding boxes in the image that indicates whether the bounding box is part of the image of an object.

[0026] Figure 1 An example neural architecture search system 100 is shown. Neural architecture search system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below can be implemented.

[0027] The neural architecture search system 100 is a system that obtains training data 102 for training a neural network to perform a dense image prediction task and a validation set 104 for evaluating the performance of the neural network on the dense image prediction task, and uses the training data 102 and the validation set 104 to determine the architecture of the neural network, which is configured to perform the dense image prediction task, that is, receive input and generate output that meets the requirements of the dense prediction task. The architecture defines the number of layers of the neural network, the operations performed by each layer, and the connections between the layers in the neural network, that is, which layers receive input from other layers in the neural network.

[0028] Typically, both the training data 102 and the validation set 104 include a set of neural network inputs and, for each network input, a corresponding target output that should be generated by the neural network to perform a dense image prediction task. For example, a larger training data set may have been randomly split to generate the training data 102 and the validation set 104. In the dense image prediction task setting, each neural network input is an image, and the neural network output for the neural network input identifies labels that should be assigned to some or all pixels in the neural network input.

[0029] The system 100 may receive the training data 102 and the validation set 104 in any of a variety of ways. For example, the system 100 may receive the training data as an upload from a remote user of the system over a data communications network, e.g., using an application programming interface (API) available to the system 100, and randomly partition the uploaded data into the training data 102 and the validation set 104. As another example, the system 100 may receive input from a user specifying which data already maintained by the system 100 should be used to train the neural network, and then partition the specified data into the training data 102 and the validation set 104.

[0030] Generally, system 100 determines the architecture of a neural network by searching a space of candidate architectures to identify one or more best performing architectures.

[0031] Each candidate architecture in the space of candidate architectures comprises (i) a same first neural network backbone configured to receive an input image and process the input image to generate a plurality of feature maps, and (ii) a different dense prediction unit configured to process the plurality of feature maps and generate an output for a dense image prediction task.

[0032] Therefore, each candidate architecture includes the same neural network backbone as every other candidate architecture, but has different dense prediction units than every other candidate architecture.

[0033] Figure 2 A candidate neural network architecture 200 is shown. As described above, the architecture 200 receives a neural network input 220, i.e., Figure 2 2 and generates a neural network output 250 that assigns a label to the neural network input. Figure 2 As shown, the neural network output 250 is represented as an overlay on the corresponding input image, where pixels assigned the same label are under the same shade in the overlay.

[0034] The candidate neural network architecture includes a backbone 230, which consists of multiple neural network layers, such as convolutional layers, optionally combined with other types of layers (batch normalization, pooling, etc.), and receives a neural network input 220 and generates a feature map, which is provided as an input to a dense prediction unit (DPC) 240. The dense prediction unit 240 processes the feature map to generate a neural network output 250.

[0035] As indicated above, each candidate neural network in the search space has the same backbone 230, but all candidate neural networks have different DPCs 240. Therefore, the search space for architecture search is the space of possible architectures for DPCs 240.

[0036] What follows is an example of the space of possible architectures of DPC 240 that system 100 may search.

[0037] For example, a DPC can have B branches, where B is a fixed integer greater than 1. Each of the B branches of the DPC maps an input tensor to the branch to an output tensor by applying an operation to the input tensor. The DPC can then combine (e.g., concatenate) the output tensors generated by all B branches to generate a combined tensor output, or use the output tensor generated by a specified one of the B branches (e.g., the last of the B branches in the processing order) as the combined tensor output. B can be, for example, an integer in the range of 3 to 10, inclusive, with larger values ​​of B allowing greater flexibility and a larger search space, but increasing the computational cost of the search process.

[0038] In some cases, the combined tensor output can be used as the output of DPC 240 for dense prediction tasks. In other cases, the DPC processes the combined tensor output through one or more output layers (e.g., a sigmoid output layer) to generate a final output.

[0039] Thus, different architectures in the search space each specify (i) an input tensor from a set of input tensors to be provided as input to each of the B branches, and (ii) an operation from a set of operations to be performed by each of the B branches.

[0040] The input tensors for a given branch can be selected from a set consisting of (i) feature maps generated by the backbone and (ii) output tensors generated by any branch preceding it in the processing order of the B branches. Thus, for the first branch in the processing order, the set consists only of feature maps generated by the backbone, while for the last branch in the processing order of the B branches, the set consists of (i) feature maps generated by the backbone and (ii) output tensors generated by any of the B-1 other branches.

[0041] The operator space, i.e., the space of possible operations from which the operation performed by each of the B blocks is selected, can include one or more of the following: (1) a convolution with a 1×1 kernel, (2) one or more atrous separable convolutions, each with a different sampling rate, and (3) one or more spatial pyramid pooling operations, each with a corresponding grid size.

[0042] When the operator space includes multiple atrous separable convolutions, each of the atrous separable convolutions will have a different sampling rate. For example, the sampling rate can be defined as r_h×r_w, where each of r_h and r_w is selected from the set {1, 2, 6, 9, ..., 21}.

[0043] When the operator space includes multiple spatial pyramid pooling operations, each spatial pyramid pooling operation will have a different grid size. For example, the grid size can be defined as g_h×g_w, where each of g_h and g_w is selected from the set {1, 2, 4, 8}.

[0044] Other examples of search spaces may include different operator spaces and different possible values ​​for the sampling rate and grid size of the operators in that space.

[0045] return Figure 1 As described above, the system 100 includes a candidate selection engine 130, a training engine 140, and a quality assessment engine 150.

[0046] To search the space of candidate architectures, the system repeatedly performs a set of architecture search operations.

[0047] In each iteration of operation, the candidate selection engine 130 selects one or more candidate architectures from the space of possible candidate architectures using the quality metrics for the candidate architectures that have been evaluated.

[0048] The training engine 140 then trains the selected candidate architecture on at least some of the training data 102 , and the quality assessment engine 150 evaluates the trained candidate architecture using the validation set 104 .

[0049] Reference below Figure 3 and Figure 4 Describes selecting candidate architectures, training candidate architectures, and evaluating the quality of architectures in more detail.

[0050] After the architecture search operation has been repeatedly performed, the system 100 may identify a threshold, fixed number of candidate architectures having the best quality metrics evaluated during the search as the best performing candidate architectures.

[0051] Once the system 100 identifies the best performing candidate architecture, the system 100 determines a final architecture of the neural network based on the best performing candidate architecture.

[0052] For example, the system 100 may identify a threshold number of best performing candidate architectures, generate a final architecture from each of the best performing candidates, and then train the final architecture to converge on a dense prediction task. The system may then select the best performing (e.g., determined based on accuracy on a validation set 104) trained architecture as the architecture of the neural network.

[0053] In some implementations, the candidate architectures and the corresponding final architectures are the same.

[0054] However, in other embodiments, the system 100 replaces the neural network backbone with a different, larger neural network backbone that has more parameters and allows the final neural network to perform better on dense prediction tasks.

[0055] In other words, in some cases the system uses a smaller backbone for architecture search than the backbone adopted for the final architecture.

[0056] For example, the system can use the Xception architecture for the backbone in the final architecture while using the MobileNet-v2 architecture for the candidate architecture (i.e., during the search). The MobileNet-v2 architecture requires approximately one twentieth of the computational cost of the Xception architecture and reduces the number of channels in the backbone feature map from 2048 dimensions to 320 dimensions. The Xception architecture is described in more detail in Xception: Deep learning with depthwise separable convolutions by F. Chollet in CVPR 2017, while the MobileNet-v2 architecture is described in more detail in Mobilenetv2: Inverted residuals and linear bottlenecks by M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen in CVP 2018.

[0057] Using a smaller backbone can allow the system to make architecture search more computationally efficient while still providing a quality signal about how well the final architecture with a larger backbone performs on dense image prediction tasks.

[0058] Reference below Figure 4 Describes additional techniques that the system can use to improve the computational efficiency of the search.

[0059] The neural network search system 100 may then output architecture data 150 that specifies the final architecture of the neural network, i.e., data that specifies the layers that are part of the neural network, the connections between the layers, and the operations performed by the layers. For example, the neural network search system 100 may output architecture data 150 to the user who submitted the training data.

[0060] In some embodiments, instead of or in addition to outputting architecture data 150, system 100 uses a trained neural network with a final architecture to process a request received by a user, for example, through an API provided by the system. That is, system 100 can receive an input to be processed, process the input using a trained neural network, and provide an output generated by the trained neural network or data derived from the generated output in response to the received input.

[0061] Figure 3 is a flow chart of an example process 300 for determining a final architecture for a dense image prediction task. For convenience, process 300 is described as being performed by a system of one or more computers located in one or more locations. For example, a neural architecture search system, such as a suitably programmed Figure 1 The neural architecture search system 100 can perform process 300.

[0062] The system receives training data for a dense image prediction task (step 302).

[0063] The system searches the space of candidate architectures using the training data to identify one or more best performing architectures (step 304). Searching the space of candidate architectures is described in more detail below with reference to 4. Once the search is complete, the system can select a threshold number of candidate architectures that have the best quality evaluations during the search as the best performing architectures.

[0064] The system determines an architecture for the neural network based on one or more best performing candidate architectures (step 306). For example, the system can generate a final architecture from each identified best performing candidate and then train the final architecture to converge on the dense prediction task. The system can then select the best performing (e.g., determined based on accuracy on a validation set) trained architecture as the architecture for the neural network.

[0065] In some implementations, the candidate architectures and the corresponding final architectures are the same.

[0066] However, in other embodiments, the system replaces the neural network backbone with a different, larger neural network backbone that has more parameters and allows the final neural network to perform better on dense prediction tasks.

[0067] In other words, in some cases the system uses a smaller backbone for architecture search than the backbone adopted for the final architecture.

[0068] Figure 4is a flow chart of an example process 400 for searching a space of candidate architectures. For convenience, process 400 is described as being performed by a system of one or more computers located in one or more locations. For example, a neural architecture search system, such as a suitably programmed Figure 1 The neural architecture search system 100 can perform process 400.

[0069] The system obtains data specifying a pre-trained backbone for searching a space of candidate architectures (step 402). In some cases, the system pre-trains the backbone together with the placeholder DPC on a placeholder dense image prediction task to determine training values ​​for the backbone parameters, i.e., such that the system only needs to train the backbone once for the entire search process. In some other cases, the system obtains data specifying training values ​​for the backbone parameters from another system that has pre-trained the backbone.

[0070] Optionally, the system processes at least some of the training inputs in the training data using a pre-trained backbone to generate feature maps for the training inputs (step 404). In other words, the system processes each of the training inputs using a pre-trained backbone (i.e., according to the trained values ​​of the backbone parameters) to generate a feature map, and then stores the generated feature map for use during the search process. When the system searches using a smaller backbone, because the channels of the feature map generated by the smaller backbone are much fewer than the channels of the feature map generated by the larger backbone (to be used in the final architecture), the storage space required to store the feature map is much less than the storage space required to store the feature map generated by the larger backbone.

[0071] The system then repeatedly performs steps 406 - 410 until a termination criterion for the search is met, for example, until a threshold number of candidate architectures have been evaluated, until the highest performing candidate architecture achieves a threshold accuracy, or a threshold amount of time has elapsed.

[0072] The system selects one or more candidate architectures from the space of candidate architectures (step 406). The system can use any of a variety of techniques to search the space to select candidate architectures.

[0073] For example, the system can use a random search strategy. In the random search strategy, at each iteration of process 400, the system uniformly selects one or more architectures from the space of candidate architectures at random, while also selecting one or more architectures that are close to or similar to the currently observed best architecture, i.e., the architecture that has been found to perform best on the dense prediction task and has been evaluated as part of the search. The random search strategy that the system can employ is described in more detail in Google vizier: A service for black-box optimization by D. Golovin, B. Solnik, S. Moitra, G. Kochanski, J. Karro, and D. Sculley in SIGKDD 2017 and Learning transferable architectures for scalable image recognition by B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le. in CVP 2018.

[0074] As another example, the system can use a reinforcement learning guided search strategy to select candidate actions. Specifically, in each iteration of process 400, the system can select an architecture output by a recursive neural network that is trained through reinforcement learning to output candidate architectures that perform well on the task. An example of a reinforcement learning guided search strategy that the system can employ is described in more detail in B. Zoph and QV Le., Neural architecture search with reinforcement learning, ICLR 2017.

[0075] The system trains the selected one or more candidate architectures on at least a portion of the training data (step 408). That is, for each selected candidate, the system trains a neural network having that architecture until a criterion for stopping training is met. During training, the system keeps the trained values ​​of the backbone parameters fixed while updating the parameter values ​​of the DPCs in the neural network.

[0076] Specifically, in embodiments where step 404 is performed to pre-process the training data using a pre-trained backbone, the system does not use the backbone to process the training input during step 408, but instead accesses the pre-generated feature maps from memory. This greatly reduces the processing power and time consumed by training, because the training input does not need to be processed through the backbone to generate feature maps before the DPC can process the feature maps.

[0077] The system may perform the training using an early stopping criterion, ie, the training may be performed for a fixed number of iterations rather than for convergence.

[0078] The system evaluates the performance of each of the trained candidate architectures (step 410). The system uses an evaluation metric that measures the performance of the trained neural network with the candidate architecture on the validation set to evaluate the performance.

[0079] Any evaluation metric suitable for the dense prediction task can be used. For example, the system can use the pixel-wise mean intersection-over-union (mIOU) on the validation dataset as an evaluation metric. As another example, the system can use the mean pixel accuracy on the validation dataset as an evaluation metric.

[0080] This specification uses the term "configured" with respect to systems and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0081] The embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their equivalents), or in a combination of one or more of them. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium to be executed by a data processing device or to control the operation of the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, for example, a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0082] The term "data processing apparatus" refers to data processing hardware and includes all types of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0083] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it may be deployed in any form, including as a standalone program or as a module, component, subroutine or other unit suitable for a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store portions of one or more modules, subroutines or code). A computer program may be deployed to execute on one computer, or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0084] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured or fully structured in any particular way, and it may be stored in a storage device in one or more locations. Thus, for example, an index database may include multiple data collections, each of which may be organized and accessed differently.

[0085] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same or more computers.

[0086] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit (e.g., an FPGA or ASIC), or by a combination of a special purpose logic circuit and one or more programmed computers.

[0087] A computer suitable for executing a computer program may be based on a general or special microprocessor or both, or on a central processing unit of any other type. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory may be supplemented by or incorporated into a special logic circuit. Typically, a computer will also include or be operably coupled to receive data from one or more mass storage devices (e.g., magnetic, magneto-optical disks, or optical disks) for storing data or to transfer data to one or more mass storage devices or both. However, a computer need not have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0088] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM optical disks.

[0089] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to a user) and a keyboard and pointing device (e.g., a mouse or trackball, through which a user can provide input to the computer). Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. Moreover, a computer may interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and in return receiving a response message from the user.

[0090] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0091] The machine learning model can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework).

[0092] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server) or includes a middleware component (e.g., an application server) or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an application, or a user can interact with an implementation of the subject matter described in this specification through the application) or any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0093] The computing system may include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server sends data to the user device, such as an HTML page, for example, for displaying data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device, such as the result of the user interaction, can be received from the device at the server.

[0094] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope of the claimed protection, but should be interpreted as descriptions of features specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented in multiple embodiments or in any suitable sub-combination. Moreover, although the features may be described above as working in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may be directed to a sub-combination or a variant of a sub-combination.

[0095] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be construed as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0096] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for searching a neural network architecture, comprising: obtaining training data for a dense image prediction task, wherein the dense image prediction task is a task that requires processing an image to generate an output that includes a corresponding prediction for each of a plurality of pixels in the image, and wherein the training data includes a plurality of training inputs; as well as Determining an architecture of a neural network configured to perform the dense image prediction task, including: Searching a space of candidate architectures using the training data to identify one or more best performing architectures, wherein the candidate architectures in the space of candidate architectures all include: (i) an identical first neural network backbone configured to receive an input image and process the input image to generate a plurality of feature maps, and (ii) corresponding dense prediction units configured to process the plurality of feature maps and generate outputs for the dense image prediction task, wherein each candidate architecture in the space has a corresponding dense prediction unit that is different from every other candidate architecture in the space, wherein the searching comprises: Obtaining data specifying a pre-trained first neural network backbone, wherein the data specifying the pre-trained first neural network backbone includes training values ​​of parameters of the first neural network backbone; processing each of the plurality of training inputs using the pretrained first neural network backbone and according to the trained values ​​of the parameters of the first neural network backbone to generate a corresponding plurality of feature maps for each of the plurality of training inputs; storing the corresponding plurality of feature maps for each of the plurality of training inputs in one or more memories; and Repeat the following steps: Selecting one or more candidate architectures from the space of candidate architectures, each selected candidate architecture having a same first neural network backbone and corresponding dense prediction unit; Training each of the selected one or more candidate architectures on at least a portion of the training data to update parameters of the corresponding dense prediction unit in the candidate architecture while keeping training values ​​of parameters of the first neural network backbone fixed, for each training input in a portion of the training data, the training comprising: accessing, by a pre-trained dense prediction unit, from the one or more memories, the corresponding plurality of feature maps generated for the training input; and providing the plurality of feature maps accessed from the memory as input to the corresponding dense prediction unit instead of processing the training input using the first neural network backbone; For each of the trained candidate architectures, evaluating the performance of the trained candidate architecture on the dense image prediction task; and The architecture of the neural network is determined based on the best performing candidate architecture.

2. The method according to claim 1, wherein: Training each of the selected one or more candidate architectures includes training each selected candidate architecture for a fixed number of iterations without seeking convergence.

3. The method according to claim 1, further comprising: The first neural network backbone is pre-trained to determine the training values ​​of the parameters of the first neural network backbone.

4. The method according to claim 1, wherein: Selecting one or more candidate architectures includes: The one or more candidate architectures are selected using a random search strategy.

5. The method according to claim 1, wherein: Determining the architecture of the neural network based on the best performing candidate architecture includes: For each of the one or more best performing candidate architectures, generating a final architecture, and further training the final architecture using the training data to converge on a dense prediction task; and The final architecture with the best performance is selected as the architecture of the neural network.

6. The method according to claim 5, wherein: Generating the final architecture includes replacing the first neural network backbone with a larger neural network backbone having more parameters.

7. The method according to claim 1, wherein: Each corresponding dense prediction unit includes B branches, where B is a fixed integer greater than 1, and where each of the B branches maps an input tensor to an output tensor by applying an operation to the input tensor.

8. The method according to claim 7, wherein: The dense prediction unit combines the output tensors generated by the B branches to generate a combined output tensor.

9. The method according to claim 8, wherein: The dense prediction unit concatenates the output tensors.

10. The method according to claim 8, wherein: The dense prediction unit processes the combined output tensor through one or more output layers to generate an output for a dense prediction task.

11. The method according to claim 7, wherein: Each candidate architecture specifies: (i) input tensors from an input tensor set that will be provided as input to each of the B branches in the corresponding dense prediction unit in the candidate architecture, and (ii) operations from an operation set that will be performed by each of the B branches in the corresponding dense prediction unit in the candidate architecture.

12. The method according to claim 11, wherein: The set of operations includes convolution with a 1x1 kernel.

13. The method according to claim 11, wherein: The set of operations includes one or more atrous separable convolution operations, each with a respective sampling rate.

14. The method according to claim 11, wherein: The set of operations includes one or more spatial pyramid pooling operations, each with a respective grid size.

15. The method according to claim 7, wherein: The B branches are ordered, and wherein, for each of the B branches, the set of input tensors includes: (i) a feature map generated by the first neural network backbone; and (ii) an output tensor generated by any branch preceding the branch in the order of the B branches.

16. A system for searching for a neural network architecture, the system comprising one or more computers and one or more storage devices, the storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method described in any one of claims 1-15.

17. One or more computer storage media storing instructions which, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1 to 15.

Images