Image classification neural network

Abstract

The present disclosure relates to image classification neural networks. A neural network system includes a subnetwork that includes a first subnetwork that includes a plurality of first modules, each first module including: a pass-through convolutional layer that processes a subnetwork input to generate a pass-through output; an average pooling stack of neural network layers that processes the subnetwork input of the first subnetwork to generate an average pooling output; a first stack of convolutional neural network layers that processes the subnetwork input to generate a first stack output; a second stack of convolutional neural network layers that processes the subnetwork input to generate a second stack output; and a concatenation layer that concatenates the generated outputs to generate a first module output. By including module subnetworks in a deep neural network, the deep neural network can perform image processing tasks better and can be trained faster and more efficiently while maintaining improved performance on the image processing tasks.

Description

[0001] Divisional Statement

[0002] This application is a divisional application of Chinese Patent Application No. 201680084514.7, filed December 29, 2016, which is based on and claims priority to U.S. Provisional Application Serial No. 62 / 297,101, filed February 18, 2016. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated in the disclosure of this application by reference.

[0003] Cross Reference to Related Applications

[0004] This application claims priority to U.S. Provisional Application Serial No. 62 / 297,101, filed February 18, 2016. The disclosure of the prior application is considered part of the disclosure of this application and is incorporated in the disclosure of this application by reference. TECHNICAL FIELD

[0005] This specification relates to processing images using deep neural networks, such as convolutional neural networks. BACKGROUND

[0006] Convolutional neural networks typically include at least two types of neural network layers, convolutional neural network layers and fully connected neural network layers. Convolutional neural network layers have sparse connectivity in which each node in a convolutional layer receives input from only a subset of nodes in the next lowest neural network layer. Some convolutional neural network layers have nodes that share weights with other nodes in the layer. However, nodes in a fully connected layer receive input from every node in the next lowest neural network layer. SUMMARY

[0007] In general, one innovative aspect of the subject matter described in this specification can be embodied in a first neural network system configured to receive an image and generate a classification output for the input image. The first neural network system can be implemented as a computer program on one or more computers in one or more locations. The first neural network system can include a plurality of subnetworks arranged in a stack on top of each other, where each subnetwork is configured to process a subnetwork input to generate a subnetwork output and provide the subnetwork output as input to another subnetwork above the subnetwork in the stack, and where the plurality of subnetworks includes a first subnetwork including a plurality of first modules, each first module including a pass-through convolutional layer configured to process a subnetwork input of the first subnetwork to generate a pass-through output, an average pooling stack of neural network layers, where layers in the average pooling stack are configured to collectively process the subnetwork input of the first subnetwork to generate an average pooling output, a first stack of convolutional neural network layers, where layers in the first stack are configured to collectively process the subnetwork input of the first subnetwork to generate a first stack output, a second stack of convolutional neural network layers, where layers in the second stack are configured to collectively process the subnetwork input of the first subnetwork to generate a second stack output, and a concatenation layer configured to concatenate the pass-through output, the average pooling output, the first stack output, and the second stack output to generate a first module output of the first module.

[0008] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. The first subnetwork includes four first modules. The pass-through convolutional layer is a 1x1 convolutional layer. The average pooling stack includes an average pooling layer followed by a 1x1 convolutional layer. The first stack includes a 1x1 convolutional layer followed by a 3x3 convolutional layer. The second stack includes a 1x1 convolutional layer followed by a 3x3 convolutional layer followed by a 3x3 convolutional layer. The first subnetwork is configured to combine the first module outputs generated by the plurality of first subnetworks to generate a first subnetwork output of the first subnetwork. The first subnetwork receives an input of 35x 35x 384, and each first module generates an output of 35x 35x 384.

[0009] Another innovative aspect of the subject matter described in this specification can be embodied in a second neural network system configured to receive an image and generate a classification output for the input image. The second neural network system can be implemented by one or more computers and configured to receive an image and generate a classification output for the input image. The second neural network system can include: a plurality of subnetworks arranged in a stack on top of each other, where each subnetwork is configured to process a subnetwork input to generate a subnetwork output and provide the subnetwork output as input to another subnetwork in the stack above the subnetwork, and where the plurality of subnetworks includes: a second subnetwork including a plurality of second modules, each second module including: a pass-through convolutional layer configured to process the subnetwork input for the second subnetwork to generate a pass-through output; an average pooling stack of neural network layers, where layers in the average pooling stack are configured to collectively process the subnetwork input for the second subnetwork to generate an average pooling output; a third stack of convolutional neural network layers, where layers in the third stack are configured to collectively process the subnetwork input for the second subnetwork to generate a third stack output; a fourth stack of convolutional neural network layers, where layers in the fourth stack are configured to collectively process the subnetwork input for the second subnetwork to generate a fourth stack output; and a concatenation layer configured to concatenate the pass-through output, the average pooling output, the third stack output, and the fourth stack output to generate a second module output for the second module.

[0010] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. The second subnetwork can include seven second modules. The pass-through convolutional layer can be a 1x1 convolutional layer. The average pooling stack can include an average pooling layer followed by a 1x1 convolutional layer. The third stack can include a 1x1 convolutional layer followed by a 1x7 convolutional layer followed by a 1x7 convolutional layer. The fourth stack can include a 1x1 convolutional layer followed by a 1x7 convolutional layer followed by a 7x1 convolutional layer followed by a 1x7 convolutional layer followed by a 7x1 convolutional layer. The second subnetwork can be configured to combine the second module outputs generated by the plurality of second modules to generate a second subnetwork output for the second subnetwork. The second subnetwork can receive an input of 17x 17x 1024 and each first module generates an output of 17x 17x 1024.

[0011] Another innovative aspect of the subject matter described in this specification can be embodied in a third neural network system implemented by one or more computers and configured to receive an image and generate a classification output for the input image. The third neural network system can include: a plurality of subnetworks arranged in a stack on top of each other, where each subnetwork is configured to process a subnetwork input to generate a subnetwork output and provide the subnetwork output as input to another subnetwork above it in the stack, and where the plurality of subnetworks includes: a third subnetwork that includes a plurality of third modules, each third module including: a pass-through convolutional layer configured to process a subnetwork input of the third subnetwork to generate a pass-through output; an average pooling stack of neural network layers, where layers in the average pooling stack are configured to collectively process the subnetwork input of the third subnetwork to generate an average pooling output; a first group of convolutional neural network layers, where layers in the first group are configured to collectively process the subnetwork input of the third subnetwork to generate a first group output; a second group of convolutional neural network layers, where layers in the second group are configured to collectively process the subnetwork input of the third subnetwork to generate a second group output; and a concatenation layer configured to concatenate the pass-through output, the average pooling output, the first group output, and the second group output to generate a third module output of the third module.

[0012] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. The second subnetwork includes three second modules. The pass-through convolutional layer can be a 1x1 convolutional layer. The average pooling stack can include an average pooling layer followed by a 1x1 convolutional layer. The first group can include: a 1x1 convolutional layer configured to process the subnetwork input of the third subnetwork to generate a first intermediate output; a 1x3 convolutional layer configured to process the first intermediate output to generate a second intermediate output; a 3x1 convolutional layer configured to process the first intermediate output to generate a third intermediate output; and a first group concatenation layer configured to concatenate the second intermediate output and the third intermediate output to generate the first group output. The second group can include: a fifth stack of convolutional layers configured to process the subnetwork input of the third subnetwork to generate a fifth stack output; a 1x3 convolutional layer configured to process the fifth stack output to generate a fourth intermediate output; a 3x1 convolutional layer configured to process the fifth stack output to generate a fifth intermediate output; and a second group concatenation layer configured to concatenate the fourth intermediate output and the fifth intermediate output to generate the second group output. The fifth group can include a 1x1 convolutional layer followed by a 1x3 convolutional layer followed by a 3x1 convolutional layer. The third subnetwork can be configured to combine the third module outputs generated by the plurality of third modules to generate a third subnetwork output of the third subnetwork. The third subnetwork can receive an input of 8x 8x 1536 and each third module generates an output of 8x 8x 1536.

[0013] Another innovative aspect of the subject matter described in this specification can be embodied in a fourth neural network system implemented by one or more computers and configured to receive an image and generate a classification output for the input image. The fourth neural network system can include: a plurality of subnetworks arranged in a stack on top of each other, wherein each subnetwork is configured to process a subnetwork input to generate a subnetwork output and provide the subnetwork output as input to another subnetwork above it in the stack, and wherein the plurality of subnetworks includes: the first subnetwork, the second subnetwork, and the third subnetwork.

[0014] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. The fourth neural network system can further include: a trunk subnetwork below the first subnetwork, the second subnetwork, and the third subnetwork in the stack, wherein the trunk subnetwork is configured to: receive the image; and process the image to generate a trunk subnetwork output. The fourth neural network system can further include: a first reduction subnetwork between the first subnetwork and the second subnetwork in the stack. The fourth neural network system can further include: a second reduction subnetwork between the second subnetwork and the third subnetwork in the stack.

[0015] Another innovative aspect of the subject matter described in this specification can be embodied in one or more non-transitory storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to implement one of the first neural network system, the second neural network system, the third neural network system, or the fourth neural network system.

[0016] Another innovative aspect of the subject matter described in this specification can be embodied in a fifth neural network system implemented by one or more computers and configured to receive an image and generate a classification output for the input image. The neural network system can include: a plurality of subnetworks arranged in a stack on top of each other, wherein each subnetwork is configured to process a subnetwork input to generate a subnetwork output and provide the subnetwork output as input to another subnetwork above it in the stack, and wherein the plurality of subnetworks includes: a first residual subnetwork including a plurality of first residual modules, each first residual module including: a first sub-module including: a pass-through convolutional layer configured to process a subnetwork input of the first residual subnetwork to generate a pass-through output; one or more sets of neural network layers, each of the one or more sets of neural network layers configured to process the subnetwork input of the first residual subnetwork to generate a respective set output; and a filter expansion layer configured to generate an expanded output by upsizing dimensions of the pass-through output and each of the set outputs; a summation layer configured to generate a summation output from the subnetwork input of the first residual subnetwork and the expanded output; and an activation function layer configured to apply an activation function to the summation output to generate a first residual module output of the first residual module.

[0017] The foregoing and other embodiments can each optionally include one or more of the following features, alone or in combination. The pass-through convolutional layer can be a 1x1 convolutional layer. The filter expansion layer can be configured to receive the pass-through output and the group output, and apply a 1x1 convolution to the pass-through output and the group output to generate the expanded output. The summing layer can be configured to sum the subnetwork input of the first subnetwork and the expanded output to generate the summed output. The summing layer can be configured to scale the expanded output to generate a scaled expanded output, and sum the subnetwork input of the first subnetwork and the scaled expanded output to generate the summed output. The activation function can be a rectified linear unit (ReLU) activation function. The one or more groups of neural network layers can include a first group that is a stack of multiple convolutional neural network layers. The one or more groups of neural network layers can also include a second group that is a different stack of multiple convolutional neural network layers. The first residual subnetwork can be configured to combine the first residual module outputs generated by the multiple first residual modules to generate a first residual subnetwork output of the first residual subnetwork.

[0018] Another innovative aspect of the subject matter described in this specification can be embodied in one or more non-transitory storage media that store instructions that, when implemented by one or more computers, cause the one or more computers to implement the fifth neural network system.

[0019] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. By including subnetworks, and in particular module subnetworks, in a deep neural network, the deep neural network can perform image processing tasks such as object recognition or image classification better. Moreover, a deep neural network that includes module subnetworks can be trained faster and more efficiently than a deep neural network that does not include module subnetworks, while maintaining improved performance on image processing tasks.

[0020] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 An example of a neural network system is shown.

[0022] Figure 2 An example of a first subnetwork is shown.

[0023] Figure 3 An example of a second subnetwork is shown.

[0024] Figure 4 An example of a third subnetwork is shown.

[0025] Figure 5 An example of a residual subnetwork is shown.

[0026] Figure 6 is a flowchart of an example process for generating an output from an input image.

[0027] Figure 7 is a flowchart of an example process for processing an input using a deep neural network.

[0028] The same reference numbers and designations in the various drawings indicate the same elements. DETAILED DESCRIPTION

[0029] Figure 1 An example neural network system 100 is shown. The neural network system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0030] The neural network system 100 receives data characterizing an input image, such as pixel information for the input image, or other information characterizing the input image. For example, the neural network system 100 can receive input image data 102. The neural network system 100 processes the received data using a deep neural network 150 and an output layer 112 to generate an output for the input image, such as output 114 from the input image data 102.

[0031] The neural network system 100 can be configured to receive input image data and generate any kind of score or classification output based on the input image, i.e., can be configured to perform any type of image processing task. The score or classification output generated by the system depends on the task the neural network system has been configured to recognize. For example, for an image classification or recognition task, the output generated by the neural network system for a given image can be a score for each of a set of object classes, where each score represents the likelihood that the image contains an object belonging to the class. As another example, for an object detection task, the output generated by the neural network system can identify the location, size, or both, of an object of interest in the input image.

[0032] Generally, the deep neural network 150 includes a plurality of subnetworks arranged on top of each other in a stack, where each subnetwork is configured to process a subnetwork input to generate a subnetwork output. Each subnetwork then provides the subnetwork output as an input to another subnetwork above it in the stack, or as an output of the deep neural network 150 if there is no subnetwork above it in the stack. The output layer 112 then processes the output of the deep neural network 150 to generate the output 114 of the neural network system 100. As described above, the type of output generated by the output layer 112 depends on the image classification task that the neural network system 100 has been configured to confirm. Similarly, the type of output layer 112 used to generate the output 114 also depends on the task. Specifically, the output layer 112 is the output layer appropriate for the task, i.e., produces the type of output required for the image processing task. For example, for an image classification task, the output layer can be a softmax output layer that generates a respective score for each of the set of object classes.

[0033] The subnetworks in the deep neural network 150 include a plurality of module subnetworks and one or more other subnetworks. Each of the other subnetworks is composed of one or more regular neural network layers, such as max pooling layers, average pooling layers, convolutional layers, fully connected layers, regularization layers, output layers, such as softmax output layers or linear regression output layers, and the like.

[0034] For example, in some implementations, the deep neural network 150 includes the subnetwork A 202, the subnetwork B 302, or the subnetwork C 402, or a combination thereof. The following references the subnetwork A 202, the subnetwork B 302, and the subnetwork C 402. Figures 2-4 Examples of the subnetwork A 202, the subnetwork B 302, and the subnetwork C 402 are provided in detail.

[0035] In various implementations, the module subnetworks can also include a trunk subnetwork, which is the lowest subnetwork in the stack and is configured to receive an image and process the image to generate a trunk output that is an input to the next higher subnetwork in the stack. For example, as shown in FIG. 1, the trunk subnetwork 104 is configured to receive the input image data 102 and process the input image data 102 to generate a trunk output that is an input to the subnetwork A 202. Figure 1

[0036] In various implementations, the module subnetworks can also include one or more reduction subnetworks that receive a subnetwork output and process the subnetwork output to reduce the dimensionality of the subnetwork output. For example, as shown in FIG. 1, the reduction subnetwork 106 is configured to receive the subnetwork A output 204 and process the subnetwork A output 204 to generate a reduced subnetwork A output that is an input to the subnetwork B 302. Figure 1 ​A simplified subnetwork X106 between subnetwork A 202 and subnetwork B 302, and a simplified subnetwork Y108 between subnetwork B 302 and subnetwork C 402 are illustrated. Simplified subnetwork X106 is configured to receive the output of subnetwork A 202 and process the output to reduce its dimensionality. Simplified subnetwork Y108 is configured to receive the output of subnetwork B 302 and process the output to reduce its dimensionality.

[0037] In some implementations, the deep neural network may include an average pooling subnetwork (e.g., average pooling subnetwork 110), which is the highest subnetwork in the stack and is configured to average pool the outputs of the aforementioned subnetworks to generate the output of the deep neural network 150.

[0038] In some implementations, a module subnetwork comprises one or more residual subnetworks. A residual subnetwork comprises multiple residual modules. Each residual module comprises one or more residual submodules. The following references... Figure 5 Provide a detailed description of an example of a residual submodule.

[0039] Figure 2 An example of subnetwork A 202 is shown. Subnetwork A 202 is depicted as a modular subnetwork including the first module. Although in Figure 2 The example shown only depicts a single module, but module subnetworks typically include multiple first modules. For instance, module subnetwork A 202 could include four first modules. Figure 2 As shown, the first module includes a pass-through convolutional layer, such as pass-through convolutional layer 210; an average stack of neural network layers, such as the average stack of neural network layer 224; one or more stacks of neural network layers, such as the stack of neural network layer 226 and another stack of neural network layer 118; and cascaded layers, such as cascaded layer 222. Module subnetwork A 202 receives input from a previous subnetwork (such as previous subnetwork 204) and generates an output representation from the received input.

[0040] Pass-through convolutional layers are configured to process the subnetwork input of subnetwork A 202 obtained from the previous subnetwork 204 to generate a pass-through output. In some implementations, the pass-through convolutional layer is a 1x1 convolutional layer. Generally, a kxk convolutional layer is a convolutional layer that uses a kxk filter. That is, kxk represents the size of the patch in the previous layer to which the convolutional layer is connected. In these implementations, 1x1 pass-through convolutional layers are often used as dimensionality reduction modules to reduce the dimension of the previous output representation and eliminate computational bottlenecks that might otherwise limit the size of the deep neural network.

[0041] The average pooling stack of neural network layers includes a stack of neural network layers configured to collectively process the subnetwork input of subnetwork A 202 to generate the average pooling output. For example, in the example of Figure 2

[0042] Each of the one or more stacks of neural network layers in the module subnetwork includes two or more neural network layers, where an initial neural network layer is followed by one or more other neural network layers. For example, subnetwork A 202 includes a first stack 226 that includes a 1x1 convolutional layer 212 followed by a 3x3 convolutional layer 214, and a second stack 228 that includes a 1x1 convolutional layer 216 followed by a 3x3 convolutional layer 218 followed by a 3x3 convolutional layer 220. However, other combinations of convolutional layer sizes are possible. The layers in the first stack 226 are configured to collectively process the subnetwork input of subnetwork A 202 to generate a first stack output, and the layers in the second stack 228 are configured to collectively process the subnetwork input of subnetwork A 202 to generate a second stack output.

[0043] The concatenation layer 222 is configured to concatenate the pass-through output, the average pooling output, the first stack output, and the second stack output to generate a first module output of the first module. For example, the concatenation layer 222 is configured to concatenate tensors generated by the pass-through convolutional layer, the average pooling stack of neural network layers, and the stacks of convolutional neural network layers along the depth dimension to generate a single tensor, i.e., an output representation. The output representation of the first module can be used as input to the next module in subnetwork A 202. The next module in subnetwork A 202 can process the input, e.g., the previous output representation, in the manner described in more detail below with reference to Figure 7

[0044] In some implementations, subnetwork A 202 can receive an input of 35x35x384, and each first module can generate an output of 35x35x384. However, other input and output sizes are possible.

[0045] Figure 3 An example of subnetwork B 302 is shown. Subnetwork B 302 is depicted as including a module subnetwork of second modules. While Figure 3 ​​Only a single module is shown, but the subnetwork B 302 can include multiple second modules. For example, the subnetwork B 302 can include seven second modules. Similar to the first modules, the second modules include a pass-through convolutional layer, such as the 1x1 pass-through convolutional layer 310; an average pooling stack of neural network layers, such as the average pooling stack of neural network layers 330; one or more stacks of neural network layers, such as the stack of neural network layers 332 and the stack of neural network layers 334; and a concatenation layer, such as the concatenation layer 328. The module subnetwork B 302 receives input from a previous subnetwork, such as the previous subnetwork 304, and generates an output representation from the received input.

[0046] The pass-through convolutional layer 310 is configured to process the subnetwork input for the subnetwork B 302 obtained from the previous subnetwork 304 to generate a pass-through output. The average pooling stack of neural network layers includes a stack of neural network layers that are configured to collectively process the subnetwork input for the subnetwork B 302 to generate an average pooling output. For example, in the instance of the average pooling stack of neural network layers 330, the average pooling stack includes the average pooling layer 306 followed by the 1x1 convolutional layer 308. Figure 3

[0047] Each of the one or more stacks of neural network layers in the module subnetwork B 302 includes two or more neural network layers, where an initial neural network layer is followed by one or more other neural network layers. For example, the subnetwork B 302 includes a third stack 332 that includes the 1x1 convolutional layer 312 followed by the 1x7 convolutional layer 314 followed by the 1x7 convolutional layer 316, and a fourth stack 334 that includes the 1x1 convolutional layer 318 followed by the 1x7 convolutional layer 320 followed by the 7x1 convolutional layer 322 followed by the 1x7 convolutional layer 324 followed by the 7x1 convolutional layer 326. However, other combinations of convolutional layer sizes are possible. The layers in the third stack 332 are configured to collectively process the subnetwork input for the subnetwork B 302 to generate a third stack output, and the layers in the fourth stack 334 are configured to collectively process the subnetwork input for the subnetwork B 302 to generate a fourth stack output.

[0048] The concatenation layer 328 is configured to concatenate the pass-through output, the average pooling output, the third stack output, and the fourth stack output to generate a second module output for the second module. For example, the concatenation layer 328 is configured to concatenate the tensors generated by the pass-through convolutional layer 310, the average pooling stack of neural network layers 330, and the stacks of convolutional neural network layers 332 and 334 along the depth dimension to generate a single tensor, which is the output representation for the second module. The output representation for the second module can be used as input for the next module in the subnetwork B 302. The next module in the subnetwork B 302 can be described below with reference to FIG. 3B. Figure 7 ​The input is processed in more detail, e.g., the previous output representation.

[0049] In some implementations, the subnetwork B 302 can receive an input of 17x17x1024 and each second module can generate an output of 17x17x1024. However, other input and output sizes are also possible.

[0050] Figure 4 An example of a subnetwork C 402 is shown. The subnetwork C 402 is depicted as including a module subnetwork of third modules. While Figure 4 Only a single module is shown, but the subnetwork C 402 can include multiple third modules. For example, the subnetwork C 402 can include three third modules. The third modules include a pass-through convolutional layer, e.g., a 1x1 pass-through convolutional layer 410; an average pooling stack of neural network layers, e.g., an average pooling stack of neural network layers 432; one or more groups of neural network layers, e.g., a group of neural network layers 434 and another group of neural network layers 436; and a concatenation layer, e.g., a concatenation layer 430. The module subnetwork C 402 receives an input from a previous subnetwork, e.g., a previous subnetwork 404, and generates an output representation from the received input.

[0051] The pass-through convolutional layer is configured to process a subnetwork input for the subnetwork C 402 obtained from the previous subnetwork 404 to generate a pass-through output. The average pooling stack of neural network layers includes a stack of neural network layers configured to collectively process the subnetwork input for the subnetwork C 402 to generate an average pooling output. For example, as shown, the average pooling stack of neural network layers 432 includes an average pooling layer 406 followed by a 1x1 convolutional layer 408. Figure 4

[0052] Each of the one or more groups of neural network layers in the module subnetwork C 402 includes two or more neural network layers, where an initial neural network layer is followed by one or more other neural network layers. As an example, as shown, the subnetwork C 402 includes a first group of neural network layers 434 that includes a 1x1 convolutional layer 412, a 1x3 convolutional layer 414, and a 3x1 convolutional layer 416. However, other combinations of convolutional layer sizes are also possible. The layer 412 is configured to process the subnetwork input for the subnetwork C 402 to generate a first intermediate output. The layer 414 and the layer 416 are each configured to process the first intermediate output to generate a second intermediate output and a third intermediate output, respectively. The first group can include a first group of concatenation layers (not shown) configured to concatenate the second intermediate output and the third intermediate output to generate a first group output. Figure 4

[0053] ​​In another example, subnetwork C 402 includes a second set of neural network layers 436 that includes a fifth stack of neural network layers 438 configured to process a subnetwork input of subnetwork C 402 to generate a fifth stack output. The second set also includes a 1x3 convolutional layer 428 configured to process the fifth stack output to generate a fourth intermediate output and a 3x1 convolutional layer 426 configured to process the fifth stack output to generate a fifth intermediate output. However, other combinations of convolutional layer dimensions are possible. The second set 436 can include a second concatenation layer (not shown) configured to concatenate the fourth intermediate output and the fifth intermediate output to generate a second set output.

[0054] The concatenation layer 430 is configured to concatenate the pass-through output, the average pooling output, the first set output, and the second set output to generate a third module output of the third module. For example, the concatenation layer 430 is configured to concatenate tensors generated by the pass-through convolutional layer 410, the average pooling stack of neural network layers 432, and the sets 434 and 436 of convolutional neural network layers along the depth dimension to generate a single tensor, i.e., an output representation of the second module. The output representation of the third module can be used as an input to a next module in subnetwork C 402. The next module in subnetwork C 402 can process the input, e.g., the previous output representation, in the manner described in more detail below with reference to FIG. 5. Figure 7

[0055] In some implementations, subnetwork C 402 can receive an input of 8x8x1536 and each third module can generate an output of 8x8x1536. However, other input and output dimensions are possible.

[0056] Figure 5 An example of a residual submodule 550 of a residual module of residual subnetwork 502 is shown. Although only one residual submodule is depicted, residual subnetwork 502 can include multiple residual modules and each residual module can include multiple residual submodules. Residual submodule 550 includes a pass-through convolutional layer, e.g., pass-through convolutional layer 506, one or more sets of neural network layers, e.g., a set of neural network layers 524 and another set of neural network layers 526, a filter expansion layer, e.g., filter expansion layer 512, a summation layer, e.g., summation layer 520, and an activation function layer, e.g., activation function layer 522. Residual submodule 550 receives an input from a previous subnetwork, e.g., previous subnetwork 504, and generates an output representation from the received input.

[0057] The pass-through convolutional layer is configured to process a subnetwork input of residual subnetwork 502 to generate a pass-through output. For example, pass-through convolutional layer 506 is a 1x1 convolutional layer that processes an input from previous subnetwork 504 to generate a pass-through output.

[0058] ​Each of the one or more groups of neural network layers is configured to process the subnetwork input of the residual subnetwork to generate a respective group output. In some implementations, the one or more groups of neural network layers include a first group that is a stack of multiple convolutional neural network layers and a second group that is another stack of multiple convolutional neural network layers. For example, the residual subnetwork 550 includes a stack of neural network layers 524 that includes the 1x1 convolutional layer 508 followed by the 1x1 convolutional layer 510, and another stack 526 that includes the 1x1 convolutional layer 514 followed by the 3x3 convolutional layer 516 followed by the 3x3 convolutional layer 518. Each of these stacks receives the subnetwork input from the previous subnetwork 504 and processes the subnetwork input to generate a respective group output.

[0059] The sum layer 520 is configured to generate the summed output from the subnetwork input of the residual subnetwork, the pass-through output, and the group outputs. However, after processing the subnetwork input (received from the previous subnetwork 504) through the pass-through convolutional layer 506, the stack of neural network layers 524, and the stack of neural network layers 526, the dimensions of the pass-through output and the group outputs can not match the dimensions of the original subnetwork input (e.g., the dimensions of the subnetwork input can be reduced by these neural network layers).

[0060] The filter expansion layer 512 is configured to generate the expanded output by upsizing the dimensions of the pass-through output and each of the group outputs so that the dimensions of the expanded output match the dimensions of the original subnetwork input. For example, as shown in FIG. 5, the filter expansion layer 512 is configured to receive the pass-through output from the pass-through convolutional layer 506 and the respective group outputs, and apply a 1x1 convolution to these outputs to generate the expanded output. Figure 5

[0061] Subsequently, the sum layer 520 can be configured to sum the subnetwork input of the residual subnetwork 502 and the expanded output to generate the summed output.

[0062] The activation function layer 522 is configured to apply an activation function to the summed output to generate the residual module output of the residual module. In some implementations, the activation function can be a rectified linear unit (ReLU) activation function.

[0063] After the residual module output is generated by the residual module, the residual subnetwork is configured to combine the residual module outputs to generate a residual subnetwork output for the residual subnetwork.

[0064] In implementations in which multiple residual subnetworks are included in the neural network system, the groups of neural networks in different ones of the residual subnetworks can have different configurations, such as different numbers of groups, different configurations of neural network layers within the groups, or both.

[0065] Figure 6 ​is a flowchart of an example process 600 for generating an output from a received input. For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, an image processing system (e.g., the image processing system 100 of Figure 1 FIG. 1, programmed appropriately in accordance with this specification, can perform process 600.

[0066] The system receives data characterizing an input image (step 602).

[0067] The system processes the data using a deep neural network that includes a sequence of subnetworks (e.g., the deep neural network 150 of Figure 1 FIG. 1) to generate a surrogate representation (step 604). The deep neural network includes a series of subnetworks arranged from a lowest subnetwork in the sequence to a highest subnetwork in the sequence. The system processes the data through each subnetwork in the sequence to generate the surrogate representation. The subnetworks in the sequence include a plurality of module subnetworks, and optionally one or more subnetworks that include one or more regular neural network layers, such as a straight-through convolutional layer, an average pooling layer, a convolutional layer, a concatenation layer, and so on. Processing an input through a module subnetwork is described below with reference to Figure 7 FIG. 2.

[0068] The system processes the surrogate representation through an output layer to generate an output for the input image (step 606). In general, the output generated by the system depends on the image processing task that the system has been configured to perform. For example, if the system is configured to perform an image classification or recognition task, then the output generated by the output layer can be respective scores for each of a predetermined set of object classes, where a score for a given object class indicates a likelihood that the input image contains an image of an object belonging to the object class.

[0069] Figure 7 is a flowchart of an example process 700 for processing an input using a module subnetwork. For convenience, the process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, Figure 1 the neural network system 100 of FIG. 1, programmed appropriately in accordance with this specification, can perform process 700.

[0070] The system receives an input (step 702). In particular, the input is a previous output representation, i.e., an output representation generated by a previous subnetwork in a sequence of subnetworks, or an output representation generated by a previous module in a sequence of modules of a module subnetwork.

[0071] The system processes the previous output representation through a straight-through convolutional layer to generate a straight-through output (step 704). In some implementations, the straight-through convolutional layer is a 1x1 convolutional layer.

[0072] The system processes the previous output representation through an average pooling stack of neural network layers to generate an average pooling output (step 706). For example, the average pooling stack of neural network layers can include an average pooling layer that averages the subnetwork input, followed by a 1x1 convolutional layer.

[0073] The system processes the previous output representation through one or more groups of neural network layers (step 708). Each group of neural network layers includes an initial neural network layer followed by one or more additional neural network layers. The system processes the previous output representation through a given group by processing the previous output representation through each neural network layer in the group to generate a group output for the group.

[0074] In some implementations, one or more of the groups includes one convolutional layer followed by another convolutional layer. For example, one group can include a 1x1 convolutional layer followed by a 3x3 convolutional layer. As another example, another group can include a 1x1 convolutional layer followed by a 3x3 convolutional layer followed by a 3x3 convolutional layer. As described above, the 1x1 convolutional layer can be used as a dimension reduction module to reduce the dimensionality of the previous output representation prior to processing by the other convolutional layer following the 1x1 convolutional layer. However, other combinations of convolutional layer sizes are possible.

[0075] The system concatenates the pass-through output, the average pooling output, and the group outputs through a concatenation layer to generate an output representation (step 710). For example, the system can concatenate the tensors generated by the pass-through convolutional layer, the average pooling stack of neural network layers, and the groups to generate a single tensor, i.e., the output representation. The system can then use the output representation as input to the next module in the sequence of modules of the subnetwork, or as input to the next subnetwork in the sequence of subnetworks, or as input to an output layer of the system.

[0076] The processes 600 and 700 can be performed to generate classification data for images for which the desired classification, i.e., the output that the system should generate for the images, is unknown. The processes 600 and 700 can also be performed on documents in a set of training images, i.e., for which the output that should be predicted by the system is known, in order to train the deep neural network, i.e., to determine the training values for the parameters of the layers in the deep neural network, i.e., the training values for the parameters of the layers in the module subnetwork and the other subnetworks. In particular, the processes 600 and 700 can be repeatedly performed for images selected from the set of training images as part of a backpropagation training technique to determine the training values for the parameters of the layers of the deep neural network.

[0077] In some implementations, during training, the deep neural network is augmented with one or more other training subnetworks that are removed after the deep neural network has been trained. Each other training subnetwork (also referred to as a "side tower") includes one or more conventional neural network layers, e.g., can include one or more of an average pooling layer, a fully connected layer, a dropout layer, etc., and an output layer configured to generate an output of the same classification as the output layer of the system. Each other training subnetwork is configured to receive an output generated by one of the subnetworks of the deep neural network, i.e., in parallel with the subnetwork that has received the subnetwork output, and to process the subnetwork output to generate a training subnetwork output for the training image. As part of a backpropagation training technique, the training subnetwork output is also used to adjust parameter values of layers in the deep neural network. As described above, once the deep neural network has been trained, the training subnetworks are removed.

[0078] A system that is configured to perform a particular operation or action means that the system has installed on it software, firmware, hardware, or a combination thereof that in operation cause the system to perform the operation or action. A computer program that is configured to perform a particular operation or action means that the one or more programs include instructions that when executed by a data processing apparatus cause the apparatus to perform the operation or action.

[0079] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not, however, a propagated signal.

[0080] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0081] A computer program (which can also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and are

[0082] As used in this specification, an "engine" or "software engine" refers to a software implemented input / output system that provides an output that is different from the input. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any appropriate type of computing device, such as a server, mobile phone, tablet computer, notebook computer, music player, e-book reader, laptop or desktop computer, PDA, smart phone, or other stationary or portable device that includes one or more processors and computer readable media. Moreover, two or more engines can be implemented on the same computing device or on different computing devices.

[0083] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit), and the

[0084] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.

[0085] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0086] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a user’s client device in response to requests received from the web browser.

[0087] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

[0088] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0089] Although this specification contains many specific implementation details, these should not be construed as limitations on any implementations, but rather as descriptions of features that can be specific to particular embodiments. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features can be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination and the claimed combination can be directed to a subcombination or variation of a subcombination.

[0090] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring such an order, nor that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing can be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0091] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the acts recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageous.

Claims

1. A neural network system, wherein the neural network system is configured to receive an image and generate an output of an image processing task, the neural network system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to implement a neural network, the neural network comprising: a residual subnetwork comprising a residual module, the residual module comprising: a first stack of convolutional neural network layers and a second stack of convolutional neural network layers, the first stack of convolutional neural network layers configured to process a subnetwork input of the residual subnetwork having a first dimension to generate a first set of outputs, the second stack of convolutional neural network layers configured to process the same subnetwork input having the first dimension to generate a second set of outputs, wherein each of the first set of outputs and the second set of outputs has a dimension that is less than the first dimension of the subnetwork input, and wherein each of the first stack of convolutional neural network layers and the second stack of convolutional neural network layers is connected to a filter expansion layer; the filter expansion layer configured to generate an expanded output by upsizing a dimension of each of the first set of outputs and the second set of outputs generated by processing the subnetwork input using the first stack of convolutional neural network layers and the second stack of convolutional neural network layers; and a summing layer configured to sum (i) the same subnetwork input having the first dimension and (ii) a second output generated by scaling the expanded output generated by upsizing the dimension of each of the first set of outputs and the second set of outputs to generate a summed output.

2. The neural network system of claim 1, wherein, the residual module further comprises a pass-through convolutional layer configured to process the subnetwork input to generate a pass-through output.

3. The neural network system of claim 2, wherein, the filter expansion layer is configured to generate the expanded output by upsizing a dimension of each of the first set of outputs, the second set of outputs, and the pass-through output.

4. The neural network system of claim 2, wherein, the pass-through convolutional layer is a 1x1 convolutional layer.

5. The neural network system of claim 2, wherein, the filter expansion layer is configured to receive the pass-through output, the first set of outputs, and the second set of outputs and apply a 1x1 convolution to the pass-through output, the first set of outputs, and the second set of outputs to generate the expanded output.

6. The neural network system of claim 1, wherein, the second output is the expanded output.

7. The neural network system of claim 1, wherein, the summing layer is configured to: scale the expanded output to generate the second output.

8. The neural network system of claim 1, wherein, the residual module further comprises an activation function layer configured to apply an activation function to the summed output to generate a residual module output of the residual module.

9. The neural network system of claim 8, wherein, the activation function is a rectified linear unit (Relu) activation function.

10. The neural network system of claim 1, wherein, the first stack of convolutional neural network layers comprises a 1x1 convolutional layer followed by a 1x1 convolutional layer.

11. The neural network system of claim 1, wherein, the second stack of convolutional neural network layers comprises a 1x1 convolutional layer followed by a 3x3 convolutional layer followed by a 3x3 convolutional layer.

12. The neural network system of claim 8, wherein, The residual subnetwork includes a plurality of other residual modules and is configured to: combine the residual module output of the residual module and other residual module outputs of the other residual modules to generate a residual subnetwork output of the residual subnetwork.

13. The neural network system of claim 1, wherein, The expansion output has a second dimension that matches the first dimension of the subnetwork input.

14. The neural network system of claim 2, wherein, The subnetwork input is a representation of an image.

15. One or more non-transitory storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to implement a neural network system configured to receive an input image and generate a classification output for the input image, wherein the neural network system comprises: a residual subnetwork including a residual module including: a first stack of convolutional neural network layers configured to process a subnetwork input of the residual subnetwork having a first dimension to generate a first set of outputs, and a second stack of convolutional neural network layers configured to process the same subnetwork input having the first dimension to generate a second set of outputs, wherein each of the first set of outputs and the second set of outputs has a dimension that is less than the first dimension of the subnetwork input, and wherein each of the first stack of convolutional neural network layers and the second stack of convolutional neural network layers is connected to a filter expansion layer; the filter expansion layer configured to generate an expansion output by upsizing a dimension of each of the first set of outputs and the second set of outputs generated by processing the subnetwork input using the first stack of convolutional neural network layers and the second stack of convolutional neural network layers; and a summing layer configured to sum (i) the same subnetwork input having the first dimension and (ii) a second output generated by scaling the expansion output generated by upsizing the dimension of each of the first set of outputs and the second set of outputs to generate a summed output.

16. The one or more non-transitory storage media of claim 15, wherein, The residual module further includes a pass-through convolutional layer configured to process the subnetwork input to generate a pass-through output.

17. The one or more non-transitory storage media of claim 16, wherein, The filter expansion layer is configured to generate the expansion output by upsizing a dimension of each of the first set of outputs, the second set of outputs, and the pass-through output.

18. The one or more non-transitory storage media of claim 15, wherein, The residual module further includes an activation function layer configured to apply an activation function to the summed output to generate a residual module output of the residual module.

19. The one or more non-transitory storage media of claim 18, wherein, The residual subnetwork includes a plurality of other residual modules and is configured to: combine the residual module output of the residual module and other residual module outputs of the other residual modules to generate a residual subnetwork output of the residual subnetwork.

20. The one or more non-transitory storage media of claim 15, wherein, The subnetwork input is a representation of an image.

Images