Connection weight learning for guiding architecture evolution

Abstract

Methods, systems, and apparatus for determining one or more neural network architectures of a neural network for performing a video processing neural network task, including a computer program encoded on a computer storage medium. In one aspect, the method includes: at each of a plurality of iterations: selecting a parent neural network architecture from a set of neural network architectures; training a neural network having the parent neural network architecture to perform the video processing neural network task, including determining training values ​​for connection weight parameters of the parent neural network architecture; generating a new neural network architecture based at least in part on the training values ​​for the connection weight parameters of the parent neural network architecture; and adding the new neural network architecture to the set of neural network architectures.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is an international application and claims the benefit of U.S. Application No. 62 / 852,217 filed on May 23, 2019. The disclosure of the above application is incorporated herein by reference in its entirety. Background Art

[0003] This specification relates to using machine learning models to process data.

[0004] A machine learning model receives an input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate an output based on the received input and the values ​​of the model parameters.

[0005] Some machine learning models are deep models that use multiple layers to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. Summary of the invention

[0006] This specification describes a system for determining a neural network architecture for performing a machine learning task, the system being implemented as a computer program on one or more computers in one or more locations.

[0007] According to a first aspect, there is provided a method for determining a neural network architecture of a neural network for performing a video processing neural network task, performed by one or more data processing devices, the method comprising: maintaining data defining a set of neural network architectures, wherein for each neural network architecture: the neural network architecture comprises a plurality of blocks, wherein each block is a space-time convolution block, the space-time convolution block comprising one or more neural network layers configured to process a block input to generate a block output; and for each of one or more given blocks: (i) the block input of the given block comprises the block output from each of the plurality of other blocks, (ii) the given block has a corresponding connection weight parameter corresponding to each of the plurality of other blocks, (iii) processing block inputs includes combining other block outputs using connection weight parameters corresponding to the other blocks; at each of multiple iterations: selecting a parent neural network architecture from a set of neural network architectures; training a neural network having the parent neural network architecture to perform a video processing neural network task, including determining training values ​​for the connection weight parameters of the parent neural network architecture; generating a new neural network architecture based at least in part on the training values ​​for the connection weight parameters of the parent neural network architecture; and adding the new neural network architecture to the set of neural network architectures; and after a final iteration of the multiple iterations, selecting a final neural network architecture from the set of neural network architectures based on a performance metric of the final neural network architecture on the video processing neural network task.

[0008] In some embodiments, each neural network architecture is configured to process input comprising (i) a plurality of video frames, and / or (ii) a plurality of optical flow frames corresponding to the plurality of video frames.

[0009] In some implementations, each block processes a block input at a corresponding time resolution to generate a block output having a corresponding number of channels.

[0010] In some embodiments, each block includes one or more dilated temporal convolutional layers having a temporal dilation rate corresponding to the temporal resolution of the block.

[0011] In some implementations, each neural network architecture includes blocks with different temporal resolutions.

[0012] In some embodiments, each block includes one or more residual modules.

[0013] In some embodiments, combining the other block outputs using connection weight parameters corresponding to the other blocks includes: for each other block output, scaling the other block output by the connection weight parameter corresponding to the other block; and generating a combined input by summing the scaled other block outputs.

[0014] In some implementations, processing a block input further comprises processing the combined input according to a plurality of block parameters to generate the block output.

[0015] In some implementations, generating a new neural network architecture based at least in part on trained values ​​of connection weight parameters of a parent neural network architecture includes determining which blocks in the new neural network architecture should receive block outputs from which other blocks in the new neural network architecture based at least in part on trained values ​​of connection weight parameters of the parent neural network architecture.

[0016] In some implementations, determining which blocks in a new neural network architecture should receive block outputs from which other blocks in the new neural network architecture based at least in part on trained values ​​of connection weight parameters of a parent neural network architecture includes, for each given block in the parent neural network architecture that (i) receives block outputs from other blocks and (ii) has connection weight parameters corresponding to other blocks having trained values ​​exceeding a threshold, determining that a block in the new neural network architecture corresponding to the given block should receive block outputs from blocks corresponding to other blocks in the new neural network architecture.

[0017] In some embodiments, the threshold is a predetermined threshold.

[0018] In some implementations, the thresholds are sampled according to a predetermined probability distribution.

[0019] In some embodiments, the method further includes, for each pair of one or more pairs of blocks in the new neural network architecture including a first block and a second block, randomly determining whether the second block should receive a block output from the first block.

[0020] In some embodiments, generating a new neural network architecture includes: for each of one or more blocks in the new neural network architecture that corresponds to a corresponding block in a parent neural network architecture, applying one or more mutation operations to the block, wherein the mutation operation includes: splitting the block, merging the block with a different block, and adjusting the temporal resolution of the block.

[0021] In some implementations, selecting a parent neural network architecture from a set of neural network architectures includes: for each of a plurality of specific neural network architectures from the set of neural network architectures, determining a performance metric of a neural network having the specific neural network architecture trained to perform a video processing neural network task; and selecting a parent neural network architecture from the plurality of specific neural network architectures based on the performance metric.

[0022] In some implementations, selecting a parent neural network architecture from a plurality of specific neural network architectures based on a performance metric includes selecting the parent neural network architecture as the specific neural network architecture having the highest performance metric on the video processing neural network task.

[0023] In some embodiments, the method further includes removing from the set of neural network architectures a particular neural network architecture having a lowest performance metric on the video processing neural network task.

[0024] In some embodiments, selecting a final neural network architecture from the set of neural network architectures includes: for each neural network architecture from the set of neural network architectures, determining a performance metric of a neural network having the neural network architecture trained to perform a video processing neural network task; and selecting the final neural network architecture as the neural network architecture having the highest performance metric on the video processing neural network task.

[0025] In some embodiments, the method further includes providing a neural network that (i) has a final neural network architecture and (ii) has been trained to perform a video processing neural network task.

[0026] In some embodiments, for each neural network architecture: each block in the neural network architecture is associated with a corresponding level in the sequence of levels; and for each given block associated with a given level after the first level in the sequence of levels, the given block receives block outputs only from other blocks associated with levels before the given level.

[0027] According to another aspect, a system is provided, comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform the operations of the aforementioned method.

[0028] According to another aspect, one or more computer storage media are provided that store instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the aforementioned method.

[0029] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0030] The system described in this specification can automatically select a neural network architecture that enables a neural network having the architecture to effectively perform machine learning tasks, such as video processing tasks. The system can select an architecture from a space of possible architectures, each of which includes multiple neural network layer blocks (e.g., spatial-temporal convolutional layers) that process block inputs, which block inputs can be derived from different input streams (e.g., video frame streams or optical flows) at corresponding temporal resolutions. As part of selecting an architecture, the system selects how the blocks in the architecture are connected to each other, that is, which blocks receive inputs from which other blocks, thereby being able to control how data flows through the architecture and when and where features that encode different information at various levels of abstraction are combined together. In contrast, some traditional architecture selection systems select architectures from a less flexible space of possible architectures, such as a space in which each architecture includes the same neural network layer module repeated multiple times in sequence. By searching a complex and flexible space of possible architectures, the system described in this specification can select an architecture that enables a neural network having the architecture to more effectively perform machine learning tasks, such as with higher prediction accuracy.

[0031] To select an architecture, the system evolves (i.e., updates) a population (i.e., a collection) of neural network architectures in multiple evolutionary iterations. In each evolutionary iteration, the system is able to generate a new neural network architecture from an existing "parent" neural network architecture based on the trained values ​​of connection weight parameters that define the strength of connections between blocks in the parent neural network architecture. For example, the system is able to determine that strongly connected blocks in the parent architecture should still be connected in the new architecture, while connections between other blocks may be randomly reconfigured in the new architecture. Guiding the evolution of a population of neural network architectures based on the trained values ​​of the connection weight parameters can enable the system to select an architecture that achieves acceptable performance on a machine learning task with fewer evolutionary iterations, thereby reducing the consumption of computing resources, such as memory and computing power.

[0032] The details of one or more embodiments of the subject matter of the present specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter of the present specification will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 Examples of possible neural network architectures for performing video processing tasks generated by the architecture selection system are shown.

[0034] Figure 2 An example architecture selection system is shown.

[0035] Figure 3 is a flow chart of an example process for determining a neural network architecture for a neural network for performing a video processing neural network task.

[0036] Like reference numbers and designations throughout the various drawings represent like elements. DETAILED DESCRIPTION

[0037] This specification describes an architecture selection system 200 for selecting an architecture for a neural network configured to perform a machine learning task.

[0038] The machine learning task may be a video processing task, i.e., a neural network processes a video (or features derived from a video) to generate an output that represents the video. "Video" may refer to a sequence of video frames, where each video frame may be represented as an array of values ​​and corresponds to a corresponding point in time. For example, a video frame may be represented as an array of RBG or CIELAB values.

[0039] Inputs to the neural network may include, for example, a sequence of video frames, a sequence of optical flow frames corresponding to the video frames, or both. Each optical flow frame may characterize motion between a corresponding pair of video frames, i.e., motion between a first video frame and a subsequent video frame. For example, each optical flow frame may specify a corresponding displacement vector corresponding to each pixel in the first video frame, where the displacement vector estimates the displacement of the pixel between the first video frame and the subsequent video frame. The optical flow frames may be derived from the video frames using any of a variety of techniques (e.g., the Lucas-Kanade method).

[0040] The neural network can generate any of a variety of outputs that characterize the input video. For example, the neural network can generate a classification output that includes a corresponding score for each of a plurality of categories, wherein the score for a category defines the likelihood that the video is included in that category. In a specific example, the neural network can generate a classification output that classifies an action being performed by a person depicted in the video, and the classification output can specify a corresponding score for each action in a set of possible actions.

[0041] Reference Figure 2 The architecture selection system 200 selects the architecture of a neural network from a space of possible architectures in more detail. An example architecture 100 selected by the architecture selection system 200 for a neural network configured to perform a video processing task is illustrated in FIG. Figure 1 Each possible architecture contains multiple "blocks", including one or more input blocks and one or more intermediate blocks. As used throughout this specification, a block refers to a set of neural network layers, for example, which can be arranged in sequence.

[0042] Each input block (e.g., input block 102) can be configured to process a sequence of video frames (e.g., video frame 104), a sequence of optical flow frames (e.g., optical flow frame 106), or both. Each intermediate block can be configured to process an input including the output of one or more other blocks (e.g., block 108 can be configured to receive input from blocks 102 and 110). The neural network can generate an output (e.g., output 112) by processing the output of one or more blocks (e.g., block 114) of the neural network by one or more neural network layers, such as a sequence of layers including pooling layers, fully connected layers, and soft-max layers.

[0043] Each block can be a space-time convolutional block, i.e., a block that includes one or more convolutional neural network layers and is configured to process space-time input to generate space-time output. The space-time data refers to an ordered collection of values, such as a tensor of values, which includes multiple spatial dimensions, time dimensions, and optionally, channel dimensions. Each block can generate an output with a corresponding number of channels, and Figure 1 In the illustrated neural network, the number of channels in the output of each block is denoted by C, for example, where block 102 generates a block output having C=32 channels.

[0044] Each block may include, for example, a spatial convolution layer (i.e., having convolution kernels defined in the spatial dimension), a spatial-temporal convolution layer (i.e., having convolution kernels defined across spatial and temporal dimensions), and a temporal convolution layer (i.e., having convolution kernels defined in the temporal dimension).

[0045] A temporal convolutional layer can be a "dilated" temporal convolutional layer, i.e., the layer output is generated by convolving the layer input with a kernel defined in the time dimension, where the convolution jumps over the input (i.e., along the time dimension) according to a stride size called the dilation rate. A dilated temporal convolutional layer can be said to process the spatio-temporal input at a "temporal resolution" defined by the dilation rate, e.g., such that a higher dilation rate can correspond to a lower temporal resolution. In some cases, each temporal convolutional layer in a block can be configured to process the spatio-temporal input at the same temporal resolution, which can be referred to as the temporal resolution of the block, and different blocks can have different temporal resolutions. In reference to Figure 1 In the illustrated neural network, the dilation rate of each block is represented by r, for example, where the dilation rate of block 102 is r = 4. The temporal resolution of a block may affect the nature of the information encoded in the output generated by the block, and blocks with different temporal resolutions may generate block outputs that encode complementary information, which may improve the performance of the neural network.

[0046] A spatial convolutional layer can be a "strided" convolutional layer, i.e., the layer output is generated by convolving the layer input with a kernel defined in the spatial dimension, where the convolution window traverses the input (i.e., along the spatial dimension) according to a step size called the stride.

[0047] In some embodiments, the neural network layers in each block can be arranged into "residual modules". Each residual module includes one or more neural network layers, and the output of the residual module can be the sum of: (i) the input of the first layer of the residual module, and (ii) the output of the last layer of the residual module. Using residual modules can stabilize the training of the neural network and thereby improve the performance of the trained neural network.

[0048] A given block configured to receive inputs from other blocks may maintain corresponding "connection weight" parameters corresponding to each of these other blocks. The given block may combine inputs from other blocks according to the connection weight parameters to generate a combined input, and then process the combined input (e.g., through one or more convolutional neural network layers) to generate a block output. The given block may generate a combined input F in , for example, as follows:

[0049]

[0050] where i indexes the n other blocks that provide input to the given block, and w i represents the connection weight parameter corresponding to block i, and F i out Represents the input received from block i. For example, combining inputs from other blocks according to equation (1) may require that the inputs from other blocks have the same spatial dimension and channel dimension. If the inputs from other blocks have different spatial dimensions, the given block may apply a pooling operation (e.g., a max pooling operation) to the inputs from other blocks to make their spatial dimensions match. If the inputs from other blocks have different channel dimensions, the given block may perform a projection operation (e.g., implemented by a spatial convolutional layer with a 1×1 dimensional kernel) to make their channel dimensions match.

[0051] In some embodiments, each block in a neural network can be associated with a corresponding level, e.g., levels 116-124, and each block in a given level receives input only from blocks in lower levels. For example, each block in level 3 (120) can receive input from blocks in level 1 (116) and level 2 (118), but not from blocks in level 3 (120), level 4 (122), or level 5 (124).

[0052] As part of selecting a neural network architecture, the architecture selection system can select: the number of blocks in each level of the architecture, the temporal resolution of each block, the number of channels in the output generated by each block, and which blocks receive input from which other blocks. To do this, the architecture selection system performs an automatic search through the space of possible architectures to select an architecture that enables the neural network to effectively perform the machine learning task, such as referring to Figure 2 Described in more detail.

[0053] Although the present specification primarily describes the architecture selection system as selecting a neural network architecture for performing a video processing task, more generally, the architecture selection system can be used to select a neural network architecture for performing any of a variety of neural network tasks. Examples of other neural network tasks include, for example: semantic segmentation tasks, image classification tasks, object detection tasks, action selection tasks, natural language processing tasks, and speech recognition tasks. In addition, although the present specification primarily describes the blocks as space-time convolution blocks, more generally, the blocks may include any suitable type of neural network layer. In addition, although the present specification primarily describes selecting a neural network architecture configured to process video frames and / or optical flow frames, the architecture selection system can be used to select a neural network architecture that processes any suitable type of data, for example, data comprising a sequence of point clouds (e.g., generated by a lidar sensor) and / or data comprising a sequence of hyperspectral images (e.g., generated by a hyperspectral sensor).

[0054] Figure 2 An example architecture selection system 200 is shown. Architecture selection system 200 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0055] The system 200 is configured to select a "final" neural network architecture 202 that enables a neural network having the architecture to effectively perform a machine learning task, such as a video processing task, as described above. To this end, the system 200 maintains a population (i.e., a set) of possible neural network architectures 204, and updates the population 204 at each of one or more iterations, which will be referred to herein as an "evolutionary" iteration. In particular, the system 200 updates the population of architectures 204 to increase the likelihood that the population 204 includes an architecture having a superior (e.g., higher) performance metric. The performance metric of an architecture can characterize the performance (e.g., prediction accuracy) of a neural network having the architecture on a machine learning task. After the final evolutionary iteration, the system 200 can select an architecture from the population 204 as the final architecture 202, as will be described in more detail below.

[0056] System 200 includes a population of possible architectures 204 , a selection engine 206 , and an architecture generation engine 208 .

[0057] The population 204 of possible architectures includes a plurality of possible architectures and is maintained by the system 200 in evolutionary iterations. Each possible architecture may be, for example, a video processing architecture including a plurality of spatiotemporal convolutional blocks, as shown in FIG. Figure 1 as described.

[0058] Prior to a first evolutionary iteration, system 200 may initialize population 204 of architectures by randomly generating a predefined number (eg, 20 or any other suitable number) of architectures and adding the generated architectures to population 204 .

[0059] To randomly generate possible architectures, system 200 can initialize the architecture with a predefined number of blocks at each of a predefined number of levels in the architecture (e.g., 5 levels, or any other suitable number of levels). The architecture of each block (e.g., the number, type, and configuration of neural network layers within the block) can be randomly selected from a predefined set of possible block architectures. Each possible block architecture can include dilated temporal convolutional neural network layers with a dilation rate from a set of possible dilation rates, e.g., r∈{1,2,4,8}. Each possible block architecture can include strided convolutional neural network layers having a stride from a set of possible strides, e.g., s∈{1,2,4,8}. Each possible block architecture can be configured to generate a block output with multiple channels from a set of possible channel dimensions, e.g., C∈{32,64,128,256}.

[0060] After initializing the blocks of each layer in the randomly generated neural network architecture, the system 200 can apply a random number of "splitting" or "merging" operations to the randomly selected blocks in the architecture. Applying a splitting operation to a block can refer to replacing the block with two new blocks, for example, where each new block can be configured to generate an output with only half the channels of the original block, but with the same temporal resolution as the original block. Applying a merging operation to a pair of blocks can refer to replacing the pair of blocks with a single new block. The new block can be configured to generate an output with a number of channels equal to the sum of the corresponding number of channels in the output of each original block in the merged block pair. The temporal resolution of the new block can be randomly selected from between the temporal resolutions of each original block.

[0061] As part of randomly generating possible architectures, the system 200 also determines which blocks should provide inputs to which other blocks, i.e., the system determines which blocks should be "connected". As used throughout this specification, a first block is said to be connected to a second block if the second block is configured to receive input from the first block. The system 200 can determine which blocks should be connected in the randomly generated architecture, for example, by adding each possible connection to the architecture with a predefined probability, e.g., p=0.5. A connection between blocks is called "possible" only when it is specified that a higher-level block should receive input from a lower-level block.

[0062] In each evolutionary iteration, selection engine 206 selects a “parent” neural network architecture from the population of architectures 204. To select parent architecture 210, selection engine 206 may randomly sample a predefined number of “candidate” architectures from the population of architectures 204 and determine a corresponding performance metric for each candidate architecture.

[0063] To determine the performance metric of the candidate architecture, the selection engine 206 can train a neural network having the candidate architecture to perform a machine learning task by training the neural network on a set of training data to determine training values ​​for neural network parameters. Specifically, the selection engine trains the neural network to determine training values ​​for connection weight parameters corresponding to connections between blocks in the candidate architecture, and to determine training values ​​for other neural network parameters, such as parameters of a neural network layer within each block.

[0064] The training data set may include a plurality of training examples, wherein each training example specifies: (i) training inputs to the neural network, and (ii) target outputs that should be generated by the neural network by processing the training inputs. For example, each training example may include training inputs specifying a sequence of video frames and / or a corresponding sequence of optical flow frames, and, for example, a target classification output indicating an action being performed by a person depicted in the video frames. The selection engine 206 may train the neural network using any suitable machine learning training technique, such as stochastic gradient descent, wherein the gradient of an objective function is back-propagated through the neural network at each of one or more training iterations. The objective function may be, for example, a cross entropy objective function or any other suitable objective function.

[0065] It will be appreciated that by appropriate selection of training data and / or loss functions, neural networks can be trained for video processing tasks other than classification tasks. For example, a neural network can be trained for super-resolution (in the spatial and / or temporal domain) using a training set comprising downsampled videos and corresponding higher resolution real videos, using a loss function, e.g., L1 or L2, that compares the output of the neural network to the higher resolution real video corresponding to the downsampled video input of the neural network. As a further example, a neural network can be trained to remove one or more types of image / video artifacts from a video, such as blocking artifacts that may be introduced during video encoding. In this example, the training data set can include a set of real videos, each video having one or more corresponding "degraded" videos (i.e., having one or more types of artifacts introduced), and a loss function, e.g., L1 or L2, that compares the output of the neural network to the real video corresponding to the degraded video input of the neural network.

[0066] After training the neural network with the candidate architecture on the training data set, the selection engine 206 can determine a performance metric for the candidate architecture by evaluating the performance of the trained neural network on the validation data set. The validation data can also include multiple training examples, as described above, but is typically separate from the training data, i.e., such that the neural network is not trained on the validation data. The selection engine 206 can evaluate the performance of the trained neural network on the validation data set using an objective function that measures the prediction accuracy of the trained neural network, such as a cross entropy objective function. The selection engine 206 can determine the performance metric for the candidate architecture, for example, as an average of the objective function values ​​across the validation data set when the training inputs of the validation data are processed using the trained neural network with the candidate architecture.

[0067] The selection engine 206 can identify a parent architecture 210 for the current evolutionary iteration based on the performance metrics of the candidate architectures selected in the current evolutionary iteration from the population of architectures 204. For example, the selection engine 206 can identify the candidate architecture with the “best” (e.g., highest) performance metric as the parent architecture 210 for the current evolutionary iteration. In addition, the selection engine 206 can also remove one or more candidate architectures from the population of architectures 204. For example, the selection engine 206 can remove the candidate architecture with the “worst” (e.g., lowest) performance metric from the population of architectures 204.

[0068] In some cases, some or all of the candidate architectures may have been trained to perform machine learning tasks at previous evolutionary iterations, and the architecture selection system 200 may have stored previously generated performance metrics for these architectures. In these cases, the selection engine 206 can reuse the previously generated performance metrics for these architectures rather than generating them again at each evolutionary iteration.

[0069] In each evolutionary iteration, architecture generation engine 208 is configured to generate a “new” neural network architecture based on: (i) parent architecture 210, and (ii) trained values ​​of connection weight parameters 212 of parent architecture 210. In particular, architecture generation engine 208 determines which blocks in the new architecture should be connected (i.e., which blocks should receive inputs from which other blocks) based at least in part on the trained values ​​of connection weight parameters 212 of parent architecture 210.

[0070] For example, the architecture generation engine 208 can initialize the new architecture 214 by generating a copy of the parent architecture 210 and setting the new architecture 214 equal to the copy of the parent architecture 210. The architecture generation engine 208 can then modify the connections between the blocks of the new architecture 214 based on the connection weight parameter values ​​212 of the parent architecture 210. For example, the architecture generation engine 208 can prune (remove) each connection from the new architecture 214 that corresponds to a connection weight parameter value 212 below a threshold. That is, the architecture generation engine 208 can only keep the connections in the new architecture 214 that correspond to connection weight parameter values ​​212 that exceed the threshold, while removing other connections. The threshold can be a predefined threshold (e.g., 0.5), or a threshold that is dynamically determined for each connection by sampling from a predefined probability distribution, such as a uniform probability distribution on the interval [0,1].

[0071] The architecture generation engine 208 can also add new connections to the new architecture 214. For example, for each possible connection that is not included in the parent architecture 210, the architecture generation engine 208 can add the possible connection to the new architecture with a predefined probability. The predefined probability can be a ratio of: (i) the number of connections pruned from the new architecture to (ii) the number of possible edges that are not included in the parent architecture 210. Adding new connections to the new architecture 214 with such a probability can result in the new architecture 214 having, on average, the same total number of connections as the parent architecture 210.

[0072] After modifying the connections between the blocks of the new architecture 214, the architecture generation engine 208 may apply one or more "mutation" operations to the randomly selected blocks of the new architecture 214. The mutation operations may include, for example, block splitting operations, block merging operations, and operations that change the dilation rate of the temporal convolutional layer in the block. When applying a split operation to a block to generate two new blocks, the architecture generation engine 208 may determine that each new block has the same connections as the original block. That is, each new block may be configured to receive input from the same source as the original block and provide its output to the same destination as the original block. When applying a merge operation to a pair of blocks, the architecture generation engine 208 may determine that the new block has all the connections of the two original blocks. That is, the new block may be configured to receive input from the same source as each original block and provide its output to the same destination as each original block. For example, the architecture generation engine 208 may change the dilation rate of the temporal convolutional layer in the block by randomly sampling a new dilation rate for the temporal convolutional layer in the block from a set of possible temporal dilation rates.

[0073] System 200 can add new architecture 214 generated by architecture generation engine 208 to the population of architectures 204 and then proceed to the next evolutionary iteration.

[0074] Generating a new architecture 214 based on the learned connection weight parameter values ​​212 of the parent architecture 210 can enable the system 200 to generate a new architecture 214 that maintains some of the properties that made the parent architecture 210 effective in performing machine learning tasks. At the same time, new connections are randomly added to the new architecture 214, thereby enabling the system 200 to explore variations of the parent architecture 210 that may be more effective in performing machine learning tasks. Guiding the evolution of the architecture population based on the trained values ​​of the connection weight parameters can increase the likelihood that the system will generate a new architecture 214 with a better (e.g., higher) performance metric.

[0075] After the final evolution iteration, the system 200 can select one or more architectures from the population of architectures 204 as the final architectures 202. For example, the system 200 can identify the final architectures 202 as the architectures having the best (e.g., highest) performance metrics in the population of architectures 204. As another example, the system 200 can identify a predefined number of architectures having the best performance metrics in the population of architectures 204 as the final architectures.

[0076] The final architecture(s) 202 can be used in any of a variety of ways. For example, a neural network having the final architecture 202 can be deployed to perform a machine learning task. As another example, the final architecture(s) 202 can be used to initialize a subsequent architecture refinement process, e.g., modifying the final architecture(s) 202 to determine other architectures for performing a machine learning task. As another example, multiple final architectures can be combined to form an ensemble model for performing a machine learning task, e.g., such that the output of each final architecture is combined (e.g., averaged) to determine an ensemble output.

[0077] Figure 3 is a flow chart of an example process 300 for determining a neural network architecture for a neural network for performing a video processing neural network task. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a computer appropriately programmed according to the present specification Figure 2 The architecture selection system 200 can perform the process 300 .

[0078] The system maintains data defining a set of neural network architectures (302). Each neural network architecture includes a plurality of blocks, wherein each block is a spatio-temporal convolutional block that includes one or more neural network layers configured to process a block input to generate a block output. For each of one or more given blocks: (i) the block input of the given block includes a block output from each of one or more other blocks, (ii) the given block has a corresponding connection weight parameter corresponding to each of the one or more other blocks, and (iii) processing the block input includes combining the other block outputs using the connection weight parameters corresponding to the other blocks.

[0079] The system performs steps 304 - 310 in each of one or more evolutionary iterations.

[0080] The system selects a parent neural network architecture from the set of neural network architectures (304).

[0081] The system trains a neural network having a parent neural network architecture to perform a video processing neural network task, including determining training values ​​for connection weight parameters of the parent neural network architecture (306).

[0082] The system generates a new neural network architecture based at least in part on the trained values ​​of the connection weight parameters of the parent neural network architecture (308).

[0083] The system adds the new neural network architecture to the collection of neural network architectures (310).

[0084] If the current evolution iteration is not the final evolution iteration, the system returns to step 304. If the current evolution iteration is the final evolution iteration, the system selects a final neural network architecture from the set of neural network architectures based on a performance metric of the final neural network architecture on the video processing neural network task (312).

[0085] This specification uses the term "configuration" in conjunction with system and computer program components. For a system where one or more computers are configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when run, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0086] The embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more computer program instructions encoded on a tangible non-temporary storage medium for execution by a data processing device or for controlling its operation. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal, for example, a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0087] The term "data processing apparatus" refers to data processing hardware, and includes various devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus can also be, or further include, dedicated logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus can also optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0088] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other suitable unit for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file containing other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to related programs, or in multiple coordinated files, such as files storing one or more modules, subroutines or portions of code. A computer program can be deployed on one computer for execution, or it can be executed on multiple computers located at one site or distributed at multiple sites and interconnected by a data communications network.

[0089] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0090] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit, such as an FPGA or ASIC, or by a combination of a special purpose logic circuit and one or more programmed computers.

[0091] A computer suitable for executing a computer program can be based on a general or special microprocessor or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or merged into the special logic circuit. Typically, the computer will also include, or be operably coupled to receive data from one or more large-capacity storage devices (e.g., magneto-optical disks, magneto-optical disks, or optical disks) for storing data or to transfer data to these devices or both. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, for example, a universal serial bus (USB) flash drive, to name a few.

[0092] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CDROM and DVD-ROM disks.

[0093] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, as well as a keyboard and a pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and then receiving a response message from the user.

[0094] The data processing apparatus for implementing the machine learning model can also include, for example, dedicated hardware accelerator units for processing the general and computationally intensive parts of machine learning training or production, i.e., inference, workloads.

[0095] Machine learning models can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, Microsoft Cognitive Toolkit framework, Apache Singa framework, or Apache MXNet framework.

[0096] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, for example, as a data server, or includes a middleware component, for example, an application server, or includes a front-end component, for example, a client computer with a graphical user interface, a web browser, or an application, through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by digital data communication of any form or medium, for example, a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), for example, the Internet.

[0097] The computing system can include a client and a server. The client and the server are usually remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of a computer program running on a corresponding computer, and has a client-server relationship with each other. In some embodiments, the server transmits data, such as an HTML page, to a user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from the user interacting with the device. Data generated at the user device, such as the result of the user interaction, can be received from the device at the server.

[0098] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination. In addition, although the above-mentioned features may be described as working in certain combinations in certain combinations, or even initially claimed as such, in some cases, one or more features from the claimed combination may be removed from the combination, and the claimed combination may be directed to a sub-combination or a variant of a sub-combination.

[0099] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0100] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes described in the accompanying drawings do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for determining a neural network architecture of a neural network for performing a video processing neural network task, performed by one or more data processing devices, the method comprising: Maintain data defining a collection of neural network architectures, where, for each neural network architecture: The neural network architecture comprises a plurality of blocks, wherein each block is a space-time convolution block, the space-time convolution block comprising one or more neural network layers, the space-time convolution block being configured to process a block input to generate a block output; and For each of one or more given blocks: (i) a block input of the given block includes a block output from each of a plurality of other blocks, (ii) the given block has a respective connection weight parameter corresponding to each of the plurality of other blocks, and (iii) processing the block input includes combining the other block outputs using the connection weight parameters corresponding to the other blocks; At each of the multiple iterations: Selecting a parent neural network architecture from the set of neural network architectures; Training a neural network having the parent neural network architecture to perform a video processing neural network task, including determining training values ​​for connection weight parameters of the parent neural network architecture; generating a new neural network architecture based at least in part on the trained values ​​of connection weight parameters of the parent neural network architecture; and adding the new neural network architecture to the set of neural network architectures; and After a final iteration of the plurality of iterations, a final neural network architecture is selected from the set of neural network architectures based on a performance metric of the final neural network architecture on the video processing neural network task.

2. The method according to claim 1, wherein: Each neural network architecture is configured to process input comprising: (i) a plurality of video frames, and / or (ii) a plurality of optical flow frames corresponding to the plurality of video frames.

3. The method according to claim 1, wherein: Each block processes a block input at a corresponding temporal resolution to generate a block output with a corresponding number of channels.

4. The method according to claim 3, wherein: Each block includes one or more dilated temporal convolutional layers having a temporal dilation rate corresponding to the temporal resolution of the block.

5. The method according to claim 3, wherein: Each neural network architecture includes blocks with different temporal resolutions.

6. The method according to claim 1, wherein: Each block includes one or more residual modules.

7. The method according to claim 1, wherein: Combining the outputs of the other blocks using connection weight parameters corresponding to the other blocks comprises: For each other block output, scaling the other block output by the connection weight parameter corresponding to the other block; and The combined input is generated by summing the scaled outputs of the other blocks.

8. The method according to claim 7, wherein: Processing the block input further includes: The combined input is processed according to a plurality of block parameters to generate the block output.

9. The method according to claim 1, wherein: Generating a new neural network architecture based at least in part on the trained values ​​of the connection weight parameters of the parent neural network architecture comprises: Determining which blocks in the new neural network architecture should receive block outputs from which other blocks in the new neural network architecture based at least in part on the trained values ​​of the connection weight parameters of the parent neural network architecture.

10. The method according to claim 9, wherein: Determining which blocks in the new neural network architecture should receive block outputs from which other blocks in the new neural network architecture based at least in part on the trained values ​​of the connection weight parameters of the parent neural network architecture comprises: For each given block in the parent neural network architecture that (i) receives block outputs from other blocks and (ii) has connection weight parameters corresponding to the other blocks having training values ​​exceeding a threshold, it is determined that the block corresponding to the given block in the new neural network architecture should receive block outputs from the block corresponding to the other block in the new neural network architecture.

11. The method according to claim 10, wherein: The threshold is a predetermined threshold.

12. The method according to claim 10, wherein: The threshold is sampled according to a predetermined probability distribution.

13. The method according to claim 9, further comprising: For each pair of one or more pairs of blocks in the new neural network architecture comprising a first block and a second block, randomly determining whether the second block should receive a block output from the first block.

14. The method according to claim 1, wherein: Generating the new neural network architecture includes: For each of the one or more blocks in the new neural network architecture that correspond to a corresponding block in the parent neural network architecture, one or more mutation operations are applied to the block, wherein the mutation operations include: splitting the block, merging the block with a different block, and adjusting the temporal resolution of the block.

15. The method according to claim 1, wherein: Selecting a parent neural network architecture from the set of neural network architectures includes: For each of a plurality of specific neural network architectures from the set of neural network architectures, determining a performance metric for a neural network having the specific neural network architecture trained to perform the video processing neural network task; and The parent neural network architecture is selected from the plurality of specific neural network architectures based on the performance metric.

16. The method according to claim 15, wherein: Selecting the parent neural network architecture from the plurality of specific neural network architectures based on the performance metric comprises: The parent neural network architecture is selected as the specific neural network architecture having the highest performance metric on the video processing neural network task.

17. The method according to claim 15, further comprising: A particular neural network architecture having a lowest performance metric on the video processing neural network task is removed from the set of neural network architectures.

18. The method according to claim 1, wherein: Selecting the final neural network architecture from the set of neural network architectures includes: For each neural network architecture from the set of neural network architectures, determining a performance metric for a neural network having the neural network architecture trained to perform the video processing neural network task; and The final neural network architecture is selected as the neural network architecture having the highest performance metric on the video processing neural network task.

19. The method of claim 1, further comprising: A neural network is provided that (i) has the final neural network architecture and (ii) has been trained to perform the video processing neural network task.

20. The method according to any one of claims 1 to 19, wherein: For each neural network architecture: Each block in the neural network architecture is associated with a corresponding level in the level sequence; and For each given block associated with a given level after a first level in the sequence of levels, the given block receives block outputs only from other blocks associated with levels preceding the given level.

21. A system for determining a neural network architecture for a neural network for performing a video processing neural network task, comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform the operations of the method described in any one of claims 1 to 20.

22. One or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1 to 20.

Images