Batch Normalization Layer

Abstract

The present invention provides methods, systems and apparatus for processing inputs using a neural network system including a batch normalization layer, including a computer program encoded on a computer storage medium. One method in the method includes: receiving a corresponding first layer output for each training example in the batch; calculating a plurality of normalization statistics for the batch based on the first layer output; normalizing each component of each first layer output using the normalization statistics to generate a corresponding normalized layer output for each training example in the batch; generating a corresponding batch normalization layer output for each training example in the training examples from the normalized layer output; and providing the batch normalization layer output as an input to the second neural network layer.

Description

Technical Field

[0001] This specification relates to processing inputs through neural network layers to generate outputs. Background Art

[0002] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict an output for a received input. In addition to an output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer in the network generates an output from the received input based on the current values ​​of the corresponding set of parameters. Summary of the invention

[0003] In general, one innovative aspect of the subject matter described in this specification can be embodied as a neural network system implemented by one or more computers, the neural network system comprising: a batch normalization layer, the batch normalization layer between a first neural network layer and a second neural network layer, wherein the first neural network layer generates a first layer output having multiple components, wherein the batch normalization layer is configured to, during training of the neural network system based on a batch of training examples: receive a corresponding first layer output for each training example in the batch; calculate a plurality of normalization statistics for the batch based on the first layer output; normalize each component of each first layer output using the normalization statistics to generate a corresponding normalized layer output for each training example in the batch; generate a corresponding batch normalization layer output for each training example in the training examples from the normalized layer output; and provide the batch normalization layer output as an input to the second neural network layer.

[0004] For a system of one or more computers to be configured to perform a specific operation or action, it means that the system has software, firmware, hardware, or a combination thereof installed thereon that, when operated, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a specific operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0005] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. A neural network system including one or more batch normalization layers can be trained more quickly than the same neural network that does not include any batch normalization layers. For example, by including one or more batch normalization layers in the neural network system, problems caused by the distribution of inputs to a given layer that changes during training can be alleviated. This can allow higher learning rates to be used effectively during training and can reduce the impact of how parameters are initialized on the training process. In addition, during training, the batch normalization layer can act as a regularization matrix and can reduce the need for other regularization techniques (e.g., dropout) employed during training. Once trained, a neural network system including a normalization layer can generate a neural network output that is as accurate as, if not more accurate than, a neural network output generated by the same neural network system.

[0006] The details of one or more embodiments of the subject matter of this specification are set forth in the drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 An example neural network system is shown.

[0008] Figure 2 is a flowchart of an example process for using a batch normalization layer to process input during training of a neural network system.

[0009] Figure 3 is a flow chart of an example process for using batch normalization to process input after a neural network system has been trained.

[0010] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION

[0011] This specification describes a neural network system including a batch normalization layer, which is implemented as a computer program on one or more computers at one or more locations.

[0012] Figure 1 An example neural network system 100 is shown. The neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations in which the systems, components, and techniques described below may be implemented.

[0013] The neural network system 100 includes a plurality of neural network layers arranged in a sequence: arranged in order from the lowest layer in the sequence to the highest layer in the sequence. The neural network system generates a neural network output through the neural network input by processing the neural network input through each layer in the sequence.

[0014] Neural network system 100 may be configured to receive any kind of digital data input and generate any kind of score or classification output based on that input.

[0015] For example, if the input to neural network system 100 is an image or features that have been extracted from an image, the output generated by neural network system 100 for a given image may be a score for each category in a set of object categories, where each score represents an estimated likelihood that the image contains an object belonging to that category.

[0016] As another example, if the input to neural network system 100 is an Internet resource (e.g., a web page), a document, or a portion of a document, or features extracted from an Internet resource, a document, or a portion of a document, the output generated by neural network system 100 for a given Internet resource, document, or portion of a document may be a score for each topic in a topic set, where each score represents an estimated likelihood that the Internet resource, document, or portion of a document is related to the topic.

[0017] As another example, if the input to neural network system 100 is a characteristic of the flash context of a particular advertisement, the output generated by neural network system 100 may be a score representing the estimated likelihood that the particular advertisement will be clicked.

[0018] As another example, if the input to neural network system 100 is features of personalized recommendations for a user, e.g., features characterizing the context of the recommendation, e.g., features characterizing actions previously taken by the user, the output generated by neural network system 100 may be a score for each content item in the set of content items, where each score represents an estimated likelihood that the user will respond positively to the recommended content item.

[0019] As another example, if the input to neural network system 100 is text in one language, the output generated by neural network system 100 may be a score for each text segment in a set of text segments in another language, where each score represents an estimated likelihood that the text segment in the other language is a suitable translation of the input text into the other language.

[0020] As another example, if the input to neural network system 100 is a spoken utterance, a sequence of spoken utterances, or features derived from one of the former two, the output generated by neural network system 100 can be a score for each text segment in the set of text segments, each score representing an estimated likelihood that the text segment is a correct transcription of the utterance or sequence of utterances.

[0021] As another example, neural network system 100 may be part of an auto-completion system or part of a text processing system.

[0022] As another example, neural network system 100 may be part of a reinforcement learning system and may generate outputs for selecting actions to be performed by an agent interacting with an environment.

[0023] Specifically, each of the layers of the neural network is configured to receive an input and generate an output from the input, and the neural network layers collectively process the neural network inputs received by the neural network system 100 to generate a corresponding neural network output for each received neural network input. Some or all of the neural network layers in the sequence generate outputs from the inputs based on current values ​​of the parameter sets of the neural network layers. For example, some layers may multiply the received inputs by a matrix of current parameter values ​​as part of generating outputs from the received inputs.

[0024] The neural network system 100 also includes a batch normalization layer 108 between neural network layer A 104 and neural network layer B 112 in the sequence of neural network layers. The batch normalization layer 108 is configured to perform a set of operations on inputs received from the neural network layer A 104 during training of the neural network system 100 and to perform another set of operations on inputs received from the neural network layer A 104 after the neural network system 100 has been trained.

[0025] Specifically, neural network system 100 may be trained based on a plurality of training example batches to determine training values ​​for parameters of neural network layers. A training example batch is a plurality of training example sets. For example, during training, neural network system 100 may process training example batch 102 and generate a corresponding neural network output for each training example in batch 102. The neural network outputs are then used to adjust the values ​​of the parameters of the neural network layers in the sequence, for example, by conventional gradient descent and back-propagation neural network training techniques.

[0026] During training of the neural network system 100 based on a given batch of training examples, the batch normalization layer 108 is configured to receive the layer A output 106 generated by the neural network layer A 104 for the training examples in the batch, process the layer A output 106 to generate a corresponding batch normalization layer output 110 for each training example in the batch, and then provide the batch normalization layer output 110 as input to the neural network layer B 112. The layer A output 106 includes the corresponding output generated by the neural network layer A 104 for each training example in the batch. Similarly, the batch normalization layer output 110 includes the corresponding output generated by the batch normalization layer 108 for each training example in the batch.

[0027] Typically, batch normalization layer 108 computes a set of normalized statistics for the batch based on layer A outputs 106, normalizes layer A outputs 106 to generate corresponding normalized outputs for each training example in the batch, and, optionally, transforms each of the normalized outputs before providing the output as input to neural network layer B 112.

[0028] The normalization statistics computed by the batch normalization layer 108 and the manner in which the batch normalization layer 108 normalizes the layer A output 106 during training depend on the properties of the neural network layer A 104 that generates the layer A output 106.

[0029] In some cases, neural network layer A 104 is a layer that generates an output that includes multiple components indexed by a dimension. For example, neural network layer A 104 can be a fully connected neural network layer. However, in some other cases, neural network layer A 104 is a convolutional layer or other type of neural network layer that generates an output that includes multiple components indexed by both feature indices and spatial position indices. Figure 2 Generating batch normalization layer outputs during training of neural network system 100 in each of these two cases is described in more detail.

[0030] Once the neural network system 100 has been trained, the neural network system 100 can receive new neural network inputs for processing and process the neural network inputs through the neural network layers to generate new neural network outputs for the inputs according to the trained values ​​of the parameters of the components of the neural network system 100. The operations performed by the batch normalization layer 108 during the processing of the new neural network inputs also depend on the properties of the neural network layer A 104. Figure 3 The processing of new neural network inputs after the neural network system 100 has been trained is described in more detail.

[0031] Batch normalization layers 108 can be included at various locations in the sequence of neural network layers, and, in some implementations, multiple batch normalization layers can be included in the sequence.

[0032] exist Figure 1 In the example of FIG. 1 , in some embodiments, neural network layer A 104 generates an output by modifying the input to the first neural network layer according to the current values ​​of the parameter set of the first neural network layer (e.g., by multiplying the input to the layer by a matrix of the current parameter values). In these embodiments, neural network layer B 112 can receive the output from batch normalization layer 108 and generate an output by applying a non-linear operation (i.e., a non-linear activation function) to the batch normalization layer output. Thus, in these embodiments, batch normalization layer 108 is inserted into the traditional neural network layer, and the operation of the traditional neural network layer is divided between neural network layer A 104 and neural network layer B 112.

[0033] In some other embodiments, the neural network layer A 104 generates an output by modifying the layer input according to the current values ​​of the parameter set to generate a modified first layer input and then applying a non-linear operation to the modified first layer input before providing the output to the batch normalization layer 108. Therefore, in these embodiments, the batch normalization layer 108 is inserted after the traditional neural network layer in the sequence.

[0034] Figure 2 is a flow chart of an example process 200 for generating a batch normalization layer output during training of a neural network based on a batch of training examples. For convenience, process 200 is described as being performed by a system of one or more computers located in one or more locations. For example, a batch normalization layer (e.g., included in a neural network system) Figure 1 The batch normalization layer 108 in the neural network system 100 can perform process 200 after being appropriately programmed.

[0035] The batch normalization layer receives the lower layer outputs of the batch of training examples (step 202). The lower layer outputs include the corresponding outputs generated by the layers below the batch normalization layer in the sequence of neural network layers for each training example in the batch.

[0036] The batch normalization layer generates a corresponding normalized output for each training example in the batch (step 204). That is, the batch normalization layer generates a corresponding normalized output from each received lower layer output.

[0037] In some cases, the layer below the batch normalization layer is a layer that generates an output that includes multiple components indexed by a dimension.

[0038] In these cases, the batch normalization layer computes, for each dimension, the mean and standard deviation of the component of the lower layer output corresponding to that dimension. The batch normalization layer then normalizes each component of each of the lower layer outputs using the mean and standard deviation to generate a corresponding normalized output for each training example in the batch. Specifically, for a given component of a given output, the batch normalization layer normalizes the component using the mean and standard deviation computed for the dimension corresponding to the component. For example, in some embodiments, for the component x corresponding to the kth dimension of the ith lower layer output from the batch β, k,i , the normalized output satisfy:

[0039]

[0040] Among them, μ Β is the mean of the components corresponding to the kth dimension of the lower layer outputs in batch β, and σ B is the standard deviation of the component corresponding to the kth dimension of the lower layer output in batch β. In some embodiments, the standard deviation is equal to (σ B 2 +ε) 1 / 2 The numerically stable standard deviation of B 2 is the variance of the component corresponding to the kth dimension of the lower layer output in batch β.

[0041] However, in some other cases, the neural network layer below the batch normalization layer is a convolutional layer or other kind of neural network layer that generates an output including multiple components indexed by both feature indices and spatial position indices, respectively.

[0042] In some of these cases, the batch normalization layer computes, for each possible feature index and spatial position index combination, the mean and variance of the components of the lower layer output having the feature index and the spatial position index. The batch normalization layer then computes, for each feature index, the average of the means of the feature index and spatial position index combination including the feature index. The batch normalization layer also computes, for each feature index, the average of the variances of the feature index and spatial position index combination including the feature index. Thus, after computing the average, the batch normalization layer has computed the mean statistic for each feature across all spatial positions and the variance statistic for each feature across all spatial positions.

[0043] The batch normalization layer then normalizes each component of each of the lower layer outputs using the averaged mean and the averaged variance to generate a corresponding normalized output for each training example in the batch. Specifically, for a given component of a given output, the batch normalization layer normalizes the component using the averaged mean and the averaged variance of the feature indexes corresponding to the component (e.g., in the same manner as described above when the layers below the batch normalization layer generate outputs indexed by a dimension).

[0044] In other of these cases, the batch normalization layer computes, for each feature index, the mean and variance of the components of the lower layer output corresponding to the feature index (ie, the lower layer output with the feature index).

[0045] The batch normalization layer then normalizes each component of each lower layer output in the lower layer outputs using the mean and variance of the feature index to generate a corresponding normalized output for each training example in the batch. Specifically, for a given component of a given output, the batch normalization layer then normalizes the component using the mean and variance of the feature index corresponding to the component (e.g., in the same manner as described above when the layers below the batch normalization layer generate outputs indexed by a dimension).

[0046] Optionally, a batch normalization layer transforms each component of each normalized output (step 206).

[0047] In the case where the layer below the batch normalization layer is a layer that generates an output including multiple components indexed by a dimension, the batch normalization layer transforms, for each dimension, each component of the normalized output in that dimension according to the current value of the parameter set for that dimension. That is, the batch normalization layer maintains a corresponding parameter set for each dimension and uses these parameters to apply the transformation to the components of the normalized output in that dimension. The values ​​of the parameter set are adjusted as part of the training of the neural network system. For example, in some embodiments, the normalized output The resulting transformed and normalized output y k,i satisfy:

[0048]

[0049] Among them, γ k and A k is the parameter for the kth dimension.

[0050] In the case where the layer below the batch normalization layer is a convolutional layer, the batch normalization layer transforms each component of each normalized output according to the current value of the parameter set for the feature index corresponding to the component. That is, the batch normalization layer maintains a corresponding parameter set for each feature index and uses these parameters to apply a transformation to the component of the normalized output having the feature index, e.g., in the same manner as described above when the layer below the batch normalization layer generates outputs indexed by a dimension. The values ​​of the parameter set are adjusted as part of training the neural network system.

[0051] The batch normalization layer provides the normalized output or the transformed normalized output as input to the layer above the batch normalization layer in the sequence (step 208).

[0052] The normalization statistics are back-propagated after the neural network has generated neural network outputs for the training examples in the batch as part of adjusting the values ​​of the parameters of the neural network, i.e., as part of performing the back-propagation training technique.

[0053] Figure 3 is a flow chart of an example process 300 for generating a batch normalization layer output for a new neural network input after the neural network has been trained. For convenience, process 300 is described as being performed by a system of one or more computers located in one or more locations. For example, a batch normalization layer included in a neural network system (e.g., included in Figure 1 The batch normalization layer 108 in the neural network system 100 can perform process 300 after being appropriately programmed.

[0054] The batch normalization layer receives the lower layer outputs for the new neural network input (step 302). The lower layer outputs are the outputs generated by the layer below the batch normalization layer in the sequence of neural network layers for the new neural network input.

[0055] The batch normalization layer generates normalized outputs for new neural network inputs (step 304).

[0056] If the output generated by the layer below the batch normalization layer is indexed by a dimension, the batch normalization layer normalizes each component in the lower layer output using a pre-computed mean and standard deviation for each of the dimensions to generate a normalized output. In some cases, the mean and standard deviation for a given dimension are calculated from the components in the dimension of all outputs generated by the layer below the batch normalization layer during training of the neural network system.

[0057] However, in some other cases, the mean and standard deviation of a given dimension are calculated based on components in the dimension of lower layer outputs generated by the layer below the batch normalization layer after training (e.g., based on lower layer outputs generated during a recent time window of a specified duration or from a specified number of lower layer outputs recently generated by the layer below the batch normalization layer).

[0058] Specifically, in some cases, the distribution of network inputs may change between the training examples used during training and the new neural network inputs used after the neural network system has been trained, and therefore, the distribution of lower layer outputs may change between them, for example, if the new neural network inputs are inputs of a different kind than the training examples. For example, a neural network system has been trained based on user images and the neural network system can now be used to process video frames. User images and video frames may have different distributions in terms of classes pictured, image attributes, composition, etc. Therefore, using statistics from training to normalize lower layer inputs may not accurately capture the statistics of lower layer outputs generated for new inputs. Therefore, in these cases, the batch normalization layer can use normalization statistics calculated based on the lower layer outputs generated by the layers below the batch normalization layer after training.

[0059] If the output generated by the layer below the batch normalization layer is indexed by feature index and spatial position index, the batch normalization layer normalizes each component of the lower layer output using the pre-computed average mean and average variance of each feature index in the feature index to generate a normalized output. In some cases, as described above, the average mean and average variance of the given feature index are calculated based on the outputs generated by the layer below the batch normalization layer for all training examples used during training. In some other cases, as described above, the mean and standard deviation of the given feature index are calculated based on the lower layer outputs generated by the layer below the batch normalization layer after training.

[0060] Optionally, a batch normalization layer transforms each component of the normalized output (step 306).

[0061] If the output generated by the layer below the batch normalization layer is indexed by dimension, the batch normalization layer transforms the components of the normalized output in each dimension according to the training values ​​of the parameter set for the dimension. If the output generated by the layer below the batch normalization layer is indexed by feature index and spatial position index, the batch normalization layer transforms each component of the normalized output according to the training values ​​of the parameter set for the feature index corresponding to the component. The batch normalization layer provides the normalized output or the transformed normalized output as input to the layer above the batch normalization layer in the sequence (step 308).

[0062] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier that are executed by a data processing device or control the operation of the data processing device. Alternatively or in addition, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. Computer storage media may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0063] The term "data processing apparatus" covers all kinds of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may include special-purpose logic circuits, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include: code that creates an execution environment for the computer program in question, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0064] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script or code) may be written in any form of programming language (including compiled or interpreted languages, declarative or procedural languages), and the computer program may be deployed in any form (including independent programs or modules, components, subroutines, or other units suitable for use in a computing environment). A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that stores other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborative files (e.g., files storing one or more modules, subroutines or partial codes). A computer program may be deployed to be executed on a computer or multiple computers located in one location or distributed across multiple locations and interconnected by a communication network.

[0065] The processes and logic flows described in this specification may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by a dedicated logic circuit (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)), and the apparatus may also be implemented as a dedicated logic circuit.

[0066] The computer that is suitable for executing computer program can be based on general or special microprocessor or both or any other kind of central processing unit.Generally speaking, central processing unit will receive instruction and data from read-only memory or random access memory or both.The basic element of computer is the central processing unit for carrying out or executing instruction and one or more memory devices for storing instruction and data.Generally speaking, computer also comprises one or more mass storage devices (for example, magnetic disk, magneto-optical disk or optical disk) for storing data, or can be operatively coupled to receive data from this mass storage device or data is transferred to this mass storage device or carry out both.However, computer does not need to have such equipment.In addition, computer can be embedded in another device, for example, mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, global positioning system (GPS) receiver or portable storage device (for example, universal serial bus (USB) flash drive), just to mention a few. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, CD-ROM disks and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0067] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, a keyboard, and a pointing device (such as a mouse or trackball), through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by a user (e.g., by sending a web page to a web browser on a user's client device in response to a request received from the web browser).

[0068] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification), or a computing system that includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected via any form of digital data communication medium, such as a communication network. Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0069] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other.

[0070] Although this specification contains many specific implementation details, these details should not be viewed as limitations on the scope of any invention or what may be claimed, but rather as descriptions of features for specific embodiments of specific inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in a single embodiment in combination. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. Moreover, although features may be described above as functioning in certain combinations or even initially claimed as such, in some cases one or more features from a claimed combination may be deleted from the combination, and a claimed combination may be directed to a sub-combination or a variant of a sub-combination.

[0071] Likewise, although operations are depicted in a particular order in the accompanying drawings, it should not be understood that such operations need to be performed in the particular order shown or in a sequential order, or that all illustrated operations need to be performed to achieve the desired result. In some environments, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0072] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions referenced in the claims can be performed in a different order and still achieve the desired results. As an example, the processes shown in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing may be beneficial.

Claims

1. A neural network system for processing image data input, the neural network system being configured to receive the input of image data and generate a corresponding output for the image data, the neural network system comprising: a batch normalization layer between a first neural network layer and a second neural network layer, wherein the first neural network layer is a convolutional layer, wherein the first neural network layer generates a first layer output having a plurality of components, the plurality of components being indexed by feature indices and spatial position indices, and wherein the batch normalization layer is configured to, during training of the neural network system based on batches of training examples of image data: receiving a corresponding first layer output for each training example of image data in the batch; A plurality of normalized statistics of the batch are calculated according to the first layer output, the calculation comprising, for each feature index: Calculating an average value of the component of the first layer output corresponding to the feature index; and Calculating the variance of the component of the first layer output corresponding to the feature index; normalizing each component of each first layer output using the normalization statistic to generate a corresponding normalized layer output for each training example of image data in the batch; generating, from the normalized layer outputs, a corresponding batch normalization layer output for each of the training examples of image data; and The batch normalization layer output is provided as input to the second neural network layer.

2. The neural network system according to claim 1, wherein: The plurality of components of the first layer output are indexed by dimension rather than by feature index and spatial position index, and wherein computing a plurality of normalized statistics from the batch of the first layer outputs comprises: For each dimension, calculating the average value of the component of the first layer output in the dimension; and For each dimension, a standard deviation of the component of the first layer output in the dimension is calculated.

3. The neural network system according to claim 2, wherein: Normalizing each component of each first layer output includes: The components are normalized using the calculated mean and the calculated standard deviation for the dimensions corresponding to the components.

4. The neural network system according to claim 2, wherein: Generating the corresponding batch normalization layer output for each training example of image data in the training examples of image data from the normalized layer outputs comprises: For each dimension, transform the component of the normalized layer output of the training example in the dimension according to the current value of the parameter set for the dimension.

5. The neural network system according to claim 4, wherein: The batch normalization layer is configured to, after the neural network system has been trained to determine training values ​​for the parameters for each dimension: Receiving a new first layer output generated by the first neural network layer for a new neural network input; Normalizing each component of the new first layer output using pre-computed mean and standard deviation statistics for the corresponding dimension to generate a new normalized layer output; generating a new batch normalized layer output by transforming, for each dimension, the components of the new normalized layer output for the training examples in the dimension according to the training values ​​of the parameter set for the dimension; as well as The batch normalization layer output is provided as a new layer input to the second neural network layer.

6. The neural network system according to claim 5, wherein: The pre-computed mean and standard deviation statistics for the dimension are computed based on first layer outputs generated by the first neural network layer during training of the neural network system.

7. The neural network system according to claim 5, wherein: The pre-computed mean and standard deviation statistics for the respective dimensions are computed based on new first layer outputs generated by the first neural network layer after the neural network system has been trained.

8. The neural network system according to claim 7, wherein: New neural network inputs processed by the neural network system after the neural network system has been trained are inputs of a different type than the training examples used to train the neural network system.

9. The neural network system according to claim 1, wherein: Calculating a plurality of normalized statistics for the batch from the first layer output comprises: For each combination of a feature index and a spatial position index, calculating an average value of the components of the first layer output having the feature index and the spatial position index; For each feature index, calculating an average of the average values ​​of the combinations comprising the feature index; For each combination of a feature index and a spatial position index, calculating the variance of the component of the first layer output having the feature index and the spatial position index; and For each feature index, an average of the variances of the combination comprising the feature index is calculated.

10. The neural network system according to claim 9, wherein: Normalizing each component of each layer output involves: The components are normalized using the average of the means and the average of the variances of the feature indices corresponding to the components.

11. The neural network system according to claim 8, wherein: Generating the corresponding batch normalization layer output for each training example of image data in the training examples of image data from the normalized layer outputs comprises: Each component of the normalized layer output is transformed according to a current value of the parameter set for the feature index corresponding to the component.

12. The neural network system according to claim 11, wherein: The batch normalization layer is configured to, after the neural network system has been trained to determine training values ​​for the parameters for each dimension: receiving a new first layer input generated from the new neural network input; Normalizing each component of the new first layer output using pre-computed mean and standard deviation statistics of the feature indices to generate a new normalized layer output; generating a new batch normalized layer output by transforming each component of the normalized layer output according to the training values ​​of the parameter set for the feature index corresponding to the component; as well as The new batch normalization layer output is provided as a new layer input to the second neural network layer.

13. The neural network system according to claim 1, wherein: Normalizing each component of each layer output involves: The components are normalized using the mean and the variance of the feature indexes corresponding to the components.

14. The neural network system according to claim 1, wherein: Generating the corresponding batch normalization layer output for each training example of image data in the training examples of image data from the normalized layer outputs comprises: Each component of the normalization layer output is transformed according to a current value of the parameter set of the feature index corresponding to the component.

15. The neural network system according to claim 14, wherein: The batch normalization layer is configured to, after the neural network system has been trained to determine training values ​​for the parameters for each dimension: receiving a new first layer input generated from the new neural network input; Normalizing each component of the new first layer output using pre-computed mean and standard deviation statistics of the feature indices to generate a new normalized layer output; generating a new batch normalized layer output by transforming each component of the normalized layer output according to the training values ​​of the parameter set for the feature index corresponding to the component; as well as The new batch normalization layer output is provided as a new layer input to the second neural network layer.

16. The neural network system according to any one of claims 1 to 15, wherein: The first neural network layer generates the first layer output by modifying the first layer input according to the current values ​​of the set of parameters of the first neural network layer.

17. The neural network system according to claim 16, wherein: The second neural network layer generates a second layer output by applying a non-linear operation to the batch normalization layer output.

18. The neural network system according to any one of claims 1 to 15, wherein: The first neural network layer generates the first layer output by modifying the first layer input according to the current values ​​of the parameter set to generate a modified first layer input and then applying a non-linear operation to the modified first layer input.

19. The neural network system according to any one of claims 1 to 15, wherein: During training of the neural network system, the neural network system is configured to back-propagate the normalized statistics as part of adjusting parameter values ​​of the neural network system.

20. A method comprising the operation performed by a batch normalization layer in a neural network system according to any one of claims 1 to 19.

21. One or more non-transitory computer storage media encoded with a computer program, the computer program comprising instructions that, when executed by one or more computers, cause the one or more computers to implement the neural network system of any one of claims 1 to 19.

Images