Batch normalization layer
By introducing a batch normalization layer into the neural network, the inefficiency caused by changes in input distribution during training is solved, resulting in faster training and a higher learning rate. This reduces the reliance on parameter initialization and other regularization techniques, and generates stable output.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GOOGLE LLC
- Filing Date
- 2016-01-28
- Publication Date
- 2026-06-09
AI Technical Summary
Existing neural networks are easily affected by changes in input distribution during training, resulting in low training efficiency. They also rely heavily on parameter initialization and require additional regularization techniques to stabilize the training process.
Introducing batch normalization layers into neural network systems reduces the impact of input distribution variations by normalizing the layer outputs during training by calculating and applying normalization statistics, and allows the normalization layers to continue processing inputs after training.
It improves the speed and stability of the training process, allows for higher learning rates, reduces reliance on other regularization techniques, and generates outputs with the same accuracy as neural networks without batch normalization layers.
Smart Images
Figure CN120068935B_ABST
Abstract
Description
[0001] Case Analysis
[0002] This application is a divisional application of Chinese invention patent application 201680012517.X, filed on January 28, 2016. Technical Field
[0003] This specification relates to processing inputs through neural network layers to generate outputs. Background Technology
[0004] A neural network is a machine learning model that uses one or more layers of non-linear units to predict the output from a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer in the network generates an output from the received input based on the current values of its corresponding parameter set. Summary of the Invention
[0005] Generally, an innovative aspect of the subject matter described in this specification can be embodied in a neural network system implemented by one or more computers, the neural network system comprising: a batch normalization layer between a first neural network layer and a second neural network layer, wherein the first neural network layer generates a first layer output having multiple components, wherein the batch normalization layer is configured to, during training of the neural network system based on batches of training examples: receive a corresponding first layer output for each training example in the batch; calculate multiple normalization statistics for the batch based on the first layer outputs; normalize each component of each first layer output using the normalization statistics to generate a corresponding normalized layer output for each training example in the batch; generate a corresponding batch normalization layer output for each training example from the normalized layer outputs; and provide the batch normalization layer output as input to the second neural network layer.
[0006] For a system of one or more computers to be configured to perform a specific operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon that causes the system to perform the operation or action during operation. For one or more computer programs to be configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.
[0007] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. A neural network system including one or more batch normalization layers can be trained much faster than the same neural network without any batch normalization layers. For example, by including one or more batch normalization layers in the neural network system, problems caused by the distribution of the input to a given layer changing during training can be mitigated. This allows for the efficient use of higher learning rates during training and reduces the impact of how parameters are initialized on the training process. Additionally, during training, the batch normalization layer can act as a regularization matrix and reduce the need for other regularization techniques (e.g., dropout) applied during training. Once trained, a neural network system including a normalization layer can generate a neural network output that is as accurate as, if not more accurate than, the output generated by the same neural network system.
[0008] Details of one or more embodiments of the subject matter in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of this subject matter will become apparent from the description, drawings, and claims. Attached Figure Description
[0009] Figure 1 An example neural network system is shown.
[0010] Figure 2 This is a flowchart illustrating an example process of using batch normalization layers to process input during the training of a neural network system.
[0011] Figure 3 This is a flowchart illustrating an example process of using batch normalization to process inputs after a neural network system has been trained.
[0012] In the various figures, the same reference numerals and symbols indicate the same elements. Detailed Implementation
[0013] This specification describes a neural network system including a batch normalization layer, which is implemented as a computer program on one or more computers at one or more locations.
[0014] Figure 1 An example neural network system 100 is shown. This neural network system 100 is an example of a system implemented as a computer program on one or more computers at one or more locations, wherein the systems, components and techniques described below can be implemented.
[0015] The neural network system 100 includes multiple neural network layers arranged in a sequence from the lowest layer to the highest layer in the sequence. The neural network system generates neural network outputs by processing the neural network inputs that pass through each layer in the sequence.
[0016] The neural network system 100 can be configured to receive any kind of digital data input and generate any kind of score or classification output based on that input.
[0017] For example, if the input to the neural network system 100 is an image or features already extracted from the image, the output generated by the neural network system 100 for a given image can be a score for each category in the set of object categories, where each score represents the estimated likelihood that the image contains an object belonging to that category.
[0018] As another example, if the input to the neural network system 100 is an internet resource (e.g., a webpage), a document, or a portion of a document, or features extracted from an internet resource, document, or portion of a document, then the output generated by the neural network system 100 for a given internet resource, document, or portion of a document can be a score for each topic in a topic set, where each score represents an estimated likelihood of the internet resource, document, or portion of a document relating to that topic.
[0019] As another example, if the input to the neural network system 100 is the features of a flash scene of a particular advertisement, the output generated by the neural network system 100 can be a score representing the estimated likelihood of clicking that particular advertisement.
[0020] As another example, if the input to the neural network system 100 is features of a user’s personalized recommendation, such as features characterizing the context of the recommendation, such as features characterizing the user’s previous actions, then the output generated by the neural network system 100 can be a score for each content item in the content item set, where each score represents an estimated likelihood that the user will respond positively to the recommended content item.
[0021] As another example, if the input to the neural network system 100 is text in one language, the output generated by the neural network system 100 can be a score for each text segment in a set of text segments in another language, where each score represents an estimated likelihood that the text segment in the other language is a suitable translation of the input text into the other language.
[0022] As another example, if the input to the neural network system 100 is spoken utterances, a sequence of spoken utterances, or features derived from either of the former two, the output generated by the neural network system 100 can be a score for each text segment in the set of text segments, each score representing an estimated likelihood that the text segment is a correct transcription of a utterance or a sequence of utterances.
[0023] As another example, the neural network system 100 may be part of an auto-completion system or a text processing system.
[0024] As another example, neural network system 100 may be part of a reinforcement learning system and may generate outputs for selecting actions to be performed by an agent that interacts with the environment.
[0025] Specifically, each layer in the neural network is configured to receive input and generate an output from said input, and the neural network layers collectively process the neural network input received by the neural network system 100 to generate a corresponding neural network output for each received neural network input. Some or all of the neural network layers in the sequence generate outputs from the input based on the current values of the parameter set of the neural network layer. For example, some layers may multiply the received input by a matrix of current parameter values as part of generating the output from the received input.
[0026] The neural network system 100 also includes a batch normalization layer 108 located between neural network layer A 104 and neural network layer B 112 in the neural network layer sequence. The batch normalization layer 108 is configured to perform a set of operations on inputs received from neural network layer A 104 during training of the neural network system 100, and to perform another set of operations on inputs received from neural network layer A 104 after training of the neural network system 100 has been completed.
[0027] Specifically, the neural network system 100 can be trained based on multiple training example batches to determine the training values of the parameters of the neural network layers. A training example batch is a set of multiple training examples. For example, during training, the neural network system 100 can process training example batch 102 and generate a corresponding neural network output for each training example in batch 102. The neural network outputs are then used to adjust the values of the parameters of the neural network layers in the sequence, for example, through conventional gradient descent and backpropagation neural network training techniques.
[0028] During training of the neural network system 100 based on a given batch of training examples, the batch normalization layer 108 is configured to receive layer A output 106 generated by neural network layer A 104 for each training example in the batch, process layer A output 106 to generate a corresponding batch normalization layer output 110 for each training example in the batch, and then provide the batch normalization layer output 110 as input to neural network layer B 112. Layer A output 106 includes the corresponding output generated by neural network layer A 104 for each training example in the batch. Similarly, batch normalization layer output 110 includes the corresponding output generated by batch normalization layer 108 for each training example in the batch.
[0029] Typically, the batch normalization layer 108 calculates a set of normalized statistics for the batch based on the output 106 of layer A, normalizes the output 106 of layer A to generate a corresponding normalized output for each training example in the batch, and optionally, transforms each of the normalized outputs before providing the output as input to the neural network layer B 112.
[0030] The normalized statistics calculated by batch normalization layer 108 and the way batch normalization layer 108 normalizes the output 106 of layer A during training depend on the nature of neural network layer A 104 that generates the output 106 of layer A.
[0031] In some cases, neural network layer A104 is a layer that generates an output that includes multiple components indexed by dimension. For example, neural network layer A104 could be a fully connected neural network layer. However, in other cases, neural network layer A104 is a convolutional layer or other type of neural network layer that generates an output that includes multiple components indexed by both feature indexing and spatial location indexing. The following will refer to... Figure 2 The generation of batch normalized layer outputs during the training of the neural network system 100 is described in more detail in each of these two cases.
[0032] Once the neural network system 100 has been trained, it can receive new neural network inputs for processing, and process these inputs through neural network layers to generate new neural network outputs based on the training values of the parameters of the components of the neural network system 100. The operations performed by the batch normalization layer 108 during the processing of new neural network inputs also depend on the properties of neural network layer A 104. The following will refer to... Figure 3 The process of processing new neural network inputs after the neural network system 100 has been trained is described in more detail.
[0033] Batch normalization layer 108 may be included at various locations in the neural network layer sequence, and in some embodiments, multiple batch normalization layers may be included in the sequence.
[0034] exist Figure 1 In some implementations, neural network layer A 104 generates output by modifying the input to that layer according to the current values of the parameter set of the first neural network layer (e.g., by multiplying the input to that layer by a matrix of the current parameter values). In these implementations, neural network layer B 112 may receive the output from batch normalization layer 108 and generate output by applying a nonlinear operation (i.e., a nonlinear activation function) to the batch normalization layer output. Therefore, in these implementations, the batch normalization layer 108 is inserted within a conventional neural network layer, and the operation of dividing a conventional neural network layer is performed between neural network layer A 104 and neural network layer B 112.
[0035] In some other embodiments, neural network layer A 104 generates an output by modifying the layer input according to the current values of the parameter set to generate a modified first layer input and then applying a nonlinear operation to the modified first layer input before providing the output to the batch normalization layer 108. Therefore, in these embodiments, the batch normalization layer 108 is inserted after the conventional neural network layers in the sequence.
[0036] Figure 2 This is a flowchart of an example process 200 for generating batch normalized layer outputs during training of a neural network based on training example batches. For convenience, process 200 is described as being executed by a system of one or more computers located in one or more locations. For example, a batch normalized layer included in the neural network system (e.g., included in...) Figure 1 The batch normalization layer 108 in the neural network system 100 can execute process 200 after being properly programmed.
[0037] The batch normalization layer receives the lower layer outputs of the training example batches (step 202). The lower layer outputs include the corresponding outputs generated by the layers below the batch normalization layer in the neural network layer sequence for each training example in the batch.
[0038] The batch normalization layer generates the corresponding normalized output for each training example in the batch (step 204). That is, the batch normalization layer generates the corresponding normalized output from each received lower layer output.
[0039] In some cases, the layer below the batch normalization layer is the layer that generates the output, which includes multiple components indexed by dimension.
[0040] In these cases, the batch normalization layer computes the mean and standard deviation of the component of the lower-level output corresponding to that dimension for each dimension. The batch normalization layer then normalizes each component of each lower-level output in the lower-level output using the mean and standard deviation to generate the corresponding normalized output for each training example in the batch. Specifically, for a given component of a given output, the batch normalization layer normalizes the component using the mean and standard deviation computed for the dimension corresponding to the component. For example, in some implementations, for the component x corresponding to the k-th dimension of the i-th lower-level output from batch β... k,i Normalized output satisfy:
[0041]
[0042] Where, μ Β It is the average of the components corresponding to the k-th dimension of the lower-level output in batch β, and σ B It is the standard deviation of the component corresponding to the k-th dimension of the lower-level output in batch β. In some implementations, the standard deviation is equal to (σ B 2 +ε) 1 / 2 The numerically stable standard deviation, where ε is a constant value and σ B 2 It is the variance of the component corresponding to the k-th dimension of the lower-level output in batch β.
[0043] However, in some other cases, the neural network layer below the batch normalization layer is a convolutional layer or other type of neural network layer that generates an output comprising multiple components indexed by both feature indexing and spatial location indexing.
[0044] In some of these cases, the batch normalization layer computes the mean and variance of the components of the lower layer output with that feature index and spatial location index for each possible combination of feature index and spatial location index. The batch normalization layer then computes the mean of the combination of feature index and spatial location index, including that feature index, for each feature index. The batch normalization layer also computes the mean of the variance of the combination of feature index and spatial location index, including that feature index, for each feature index. Therefore, after computed the mean, the batch normalization layer has computed the mean statistic and the variance statistic for each feature across all spatial locations.
[0045] The batch normalization layer then normalizes each component of each lower-level output in the lower-level output using the average mean and average variance to generate the corresponding normalized output for each training example in the batch. Specifically, for a given component of a given output, the batch normalization layer normalizes the component using the average mean and average variance of the feature indices corresponding to the component (e.g., in the same way described above when the layers below the batch normalization layer generate outputs indexed by dimension).
[0046] In other cases, the batch normalization layer computes the mean and variance of the components of the lower layer output (i.e., the lower layer output with that feature index) for each feature index.
[0047] The batch normalization layer then normalizes each component of each lower-level output in the lower-level output using the mean and variance of the feature indices to generate the corresponding normalized output for each training example in the batch. Specifically, for a given component of a given output, the batch normalization layer then normalizes the component using the mean and variance of the feature indices corresponding to the component (e.g., in the same way described above when the layers below the batch normalization layer generate outputs indexed by dimension).
[0048] Optionally, the batch normalization layer transforms each component of each normalized output (step 206).
[0049] In cases where the layer below the batch normalization layer generates an output comprising multiple components indexed by a dimension, the batch normalization layer transforms each component of the normalized output in that dimension based on the current values of the parameter set for that dimension. That is, the batch normalization layer maintains a corresponding parameter set for each dimension and uses these parameters to apply the transformation to the components of the normalized output in that dimension. The values of this parameter set are adjusted as part of the training of the neural network system. For example, in some implementations, the normalized output... The generated normalized output y after transformation k,i satisfy:
[0050]
[0051] Where, γ k and A k It refers to the parameter for the k-th dimension.
[0052] When the layer below the batch normalization layer is a convolutional layer, the batch normalization layer transforms each component of the normalized output for each normalized output based on the current values of the parameter set corresponding to that component's feature index. That is, the batch normalization layer maintains a corresponding parameter set for each feature index and uses these parameters to apply the transformation to the components of the normalized output with the feature index, for example, in the same way described above when the layer below the batch normalization layer generates the output indexed by dimension. The values of this parameter set are adjusted as part of the training of the neural network system.
[0053] The batch normalization layer provides normalized output or transformed normalized output as input to the layers on the batch normalization layer in the sequence (step 208).
[0054] After the neural network has generated the output of the training examples in the batch, the normalized statistics are backpropagated as part of adjusting the values of the neural network parameters, that is, as part of performing the backpropagation training technique.
[0055] Figure 3 This is a flowchart of an example process 300 for generating batch normalization layer outputs of new neural network inputs after a neural network has been trained. For convenience, process 300 is described as being executed by a system of one or more computers located in one or more locations. For example, a batch normalization layer included in a neural network system (e.g., included in...) Figure 1 The batch normalization layer 108 in the neural network system 100 can execute process 300 after being properly programmed.
[0056] The batch normalization layer receives the lower layer outputs of the new neural network input (step 302). The lower layer outputs are generated by the layers below the batch normalization layer in the neural network layer sequence in response to the new neural network input.
[0057] The batch normalization layer generates the normalized output of the new neural network input (step 304).
[0058] If the output generated by the layers below the batch normalization layer is indexed by dimension, the batch normalization layer normalizes each component of the lower layer's output using pre-computed mean and standard deviation for each dimension to generate a normalized output. In some cases, the mean and standard deviation for a given dimension are computed from the components of all outputs generated by the layers below the batch normalization layer during the training of the neural network system.
[0059] However, in some other cases, the mean and standard deviation of a given dimension are calculated based on the components in the lower layer outputs generated by the layers below the batch normalization layer after training (e.g., based on the lower layer outputs generated during the most recent time window of a specified duration or from a specified number of lower layer outputs recently generated by the layers below the batch normalization layer).
[0060] Specifically, in some cases, the distribution of network inputs can change between the training examples used during training and the new neural network inputs used after the neural network system has been trained. Therefore, the distribution of lower-level outputs can also change between these two types, for example, if the new neural network input is of a different kind than the training examples. For instance, a neural network system has been trained based on user images and can now be used to process video frames. User images and video frames may have different distributions in terms of the class pictured, image attributes, composition, etc. Therefore, using statistics from training to normalize the lower-level inputs may not accurately capture the statistics of the lower-level outputs generated for the new inputs. Therefore, in these cases, the batch normalization layer can use normalized statistics calculated from the lower-level outputs generated after training by the layers below the batch normalization layer.
[0061] If the output generated by the layers below the batch normalization layer is indexed by feature indexes and spatial location indexes, the batch normalization layer normalizes each component of the lower layer output using the pre-computed mean and mean variance of the average of each feature index in the feature indexes to generate a normalized output. In some cases, as described above, the mean and mean variance of a given feature index are calculated based on the output generated by the layers below the batch normalization layer for all training examples used during training. In other cases, as described above, the mean and standard deviation of a given feature index are calculated based on the lower layer output generated by the layers below the batch normalization layer after training.
[0062] Optionally, the batch normalization layer transforms each component of the normalized output (step 306).
[0063] If the output generated by the layer below the batch normalization layer is indexed by dimension, the batch normalization layer transforms the components of the normalized output in that dimension for each dimension based on the training values of the parameter set for that dimension. If the output generated by the layer below the batch normalization layer is indexed by feature index and spatial location index, the batch normalization layer transforms each component of the normalized output based on the training values of the parameter set corresponding to the feature index of the component. The batch normalization layer provides the normalized output or the transformed normalized output as input to the layer above the batch normalization layer in the sequence (step 308).
[0064] Embodiments of the subject matter and functional operation described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware including the structures disclosed herein and their equivalents, or in one or more combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, which are executed by or control the operation of a data processing device. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated for encoding information to be transmitted to a suitable receiver device for execution by the data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof.
[0065] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. Apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.
[0066] Computer programs (also referred to or described as programs, software, software applications, modules, software modules, scripts, or code) can be written in any programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including standalone programs or modules, components, subroutines, or other units suitable for a computing environment. Computer programs may, but do not necessarily, correspond to files in a file system. Programs may be stored as a portion of a file containing other programs or data (e.g., one or more scripts in a markup language document), a single file dedicated to the program in discussion, or multiple collaborating files (e.g., a file storing one or more modules, subroutines, or portions of code). Computer programs can be deployed to execute on one or more computers located in one location or distributed across multiple locations and interconnected via a communication network.
[0067] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit)), and the apparatus can also be implemented as special-purpose logic circuitry.
[0068] A computer suitable for executing computer programs can be, for example, based on a general-purpose or special-purpose microprocessor or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory or random access memory or both. The basic components of a computer are the central processing unit for making or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or operatively coupled to receive data from or transfer data to or both from such mass storage devices. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device (e.g., a Universal Serial Bus (USB) flash drive), to name just a few. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, CD-ROMs, and DVD-ROMs. The processor and memory may be supplemented by dedicated logic circuitry or may be incorporated into that dedicated logic circuitry.
[0069] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor), a keyboard, and pointing devices (such as a mouse or trackball), through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user (e.g., by sending a webpage to a web browser on the user's client device in response to a request received from a web browser).
[0070] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface or web browser through which a user can interact with embodiments of the subject matter described in this specification), or the computing system includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected via any form of digital data communication medium, such as a communication network. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.
[0071] A computing system may include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is established through computer programs running on the respective computers and having a client-server relationship with each other.
[0072] While this specification contains numerous details of specific implementation, these details should not be construed as limiting the scope of any invention or potentially claimed content, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in this specification within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while features may be described above as functioning in certain combinations or even initially claimed in this way, in some cases one or more features from a claimed combination may be removed from the combination, and a claimed combination may refer to a sub-combination or a variation of a sub-combination.
[0073] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the indicated specific order or in a sequential order, or as requiring all illustrated operations to achieve the desired result. In some environments, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
[0074] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions referenced in the claims can be performed in a different order and still achieve the desired result. As an example, the processes shown in the figures do not necessarily require the specific order or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.
Claims
1. A system for processing image data input, the system comprising: User computer; as well as A computer system comprising one or more computers and one or more storage devices storing instructions, the instructions, when executed by the one or more computers, causing the one or more computers to perform operations, the operations including: Receive network input, including an image or image features of the image, from the user's computer; The network input is processed using a convolutional neural network, the convolutional neural network being configured to receive the network input and generate a network output representing the image, the convolutional neural network comprising: Multiple neural network layers, including a first convolutional neural network layer and a second neural network layer; and A batch normalization layer, situated between a first convolutional neural network layer and a second neural network layer, wherein the first convolutional neural network layer generates a first layer output having multiple components indexed by feature indexing and spatial location indexing, and wherein the batch normalization layer is configured to, during training of the convolutional neural network on training example batches: Receive the corresponding first-layer output for each training example in the batch; Calculate multiple normalized statistics for the batch based on the output of the first layer, wherein calculating the multiple normalized statistics for the output of the first layer includes, for each feature index in the feature index: Calculate the average value of the components of the first layer output corresponding to the feature index; and Calculate the variance of the component of the first layer output corresponding to the feature index; The normalization statistics are used to normalize each component of each first layer output to generate the corresponding normalized layer output for each training example in the batch. Generate the corresponding batch normalization layer output for each training example from the normalization layer output; and The batch normalization layer output is provided as the input to the second neural network layer; and The network output is provided as the output of the computing system.
2. The system according to claim 1, wherein, Normalizing each component of the output from each layer includes: The component is normalized using the mean and variance of the feature index corresponding to the component.
3. The system according to claim 1, wherein, Generating the corresponding batch normalization layer output for each training example from the normalization layer output includes: The component is transformed based on the current value of the parameter set of the feature index corresponding to each component of the normalization layer output.
4. The system according to claim 3, wherein, The batch normalization layer is configured to be applied after the neural network has been trained to determine the training values of the parameters for each feature index in the feature indices: Receive the new first-layer input generated from the new neural network input; A new normalized layer output is generated by normalizing each component of the new first layer output using pre-computed average and standard deviation statistics for the feature index. A new batch normalization layer output is generated by transforming the components according to the training values of the parameter set of the feature index corresponding to each component of the normalization layer output. as well as The new batch normalized layer output is provided as a new layer input to the second neural network layer.
5. The system according to claim 1, wherein, The first convolutional neural network layer generates the first layer output by applying convolution to the first layer input based on the current values of the parameter set of the first convolutional neural network layer.
6. The system according to claim 5, wherein, The second neural network layer generates the second layer output by applying a nonlinear operation to the batch normalization layer output.
7. The system according to claim 1, wherein, The first neural network layer generates the first layer output by applying convolution to the first layer input based on the current values of the parameter set of the first convolutional neural network layer to generate a modified first layer input, and then applying a nonlinear operation to the modified first layer input.
8. The system according to claim 1, wherein, During the training of the neural network, the neural network system is configured to backpropagate the normalized statistics as part of adjusting the parameter values of the neural network.
9. The system according to claim 1, wherein, The network output includes a score for each of the multiple categories.
10. One or more non-transitory computer-readable storage media storing instructions, said instructions, when executed by one or more computers, causing said one or more computers to perform operations for processing image data input, said operations including: Receive network input from the user's computer, including an image or image features of the image; The network input is processed using a convolutional neural network, the convolutional neural network being configured to receive the network input and generate a network output representing the image, the convolutional neural network comprising: Multiple neural network layers, including a first convolutional neural network layer and a second neural network layer; and A batch normalization layer, situated between a first convolutional neural network layer and a second neural network layer, wherein the first convolutional neural network layer generates a first layer output having multiple components indexed by feature indexing and spatial location indexing, and wherein the batch normalization layer is configured to, during training of the convolutional neural network on training example batches: Receive the corresponding first-layer output for each training example in the batch; Calculate multiple normalized statistics for the batch based on the output of the first layer, wherein calculating the multiple normalized statistics for the output of the first layer includes, for each feature index in the feature index: Calculate the average value of the components of the first layer output corresponding to the feature index; and Calculate the variance of the component of the first layer output corresponding to the feature index; The normalization statistics are used to normalize each component of each first layer output to generate the corresponding normalized layer output for each training example in the batch. Generate the corresponding batch normalization layer output for each training example from the normalization layer output; and The batch normalization layer output is provided as the input to the second neural network layer; and The network output is provided as the output of the one or more computers.
11. The computer-readable storage medium according to claim 10, wherein, Normalizing each component of the output from each layer includes: The component is normalized using the mean and variance of the feature index corresponding to the component.
12. The computer-readable storage medium according to claim 10, wherein, Generating the corresponding batch normalization layer output for each training example from the normalization layer output includes: The component is transformed based on the current value of the parameter set of the feature index corresponding to each component of the normalization layer output.
13. The computer-readable storage medium according to claim 12, wherein, The batch normalization layer is configured to be applied after the neural network has been trained to determine the training values of the parameters for each feature index in the feature indices: Receive the new first-layer input generated from the new neural network input; A new normalized layer output is generated by normalizing each component of the new first layer output using pre-computed average and standard deviation statistics for the feature index. A new batch normalization layer output is generated by transforming the components according to the training values of the parameter set of the feature index corresponding to each component of the normalization layer output. as well as The new batch normalized layer output is provided as a new layer input to the second neural network layer.
14. The computer-readable storage medium according to claim 10, wherein, The first convolutional neural network layer generates the first layer output by applying convolution to the first layer input based on the current values of the parameter set of the first convolutional neural network layer.
15. The computer-readable storage medium according to claim 10, wherein, The second neural network layer generates the second layer output by applying a nonlinear operation to the batch normalization layer output.
16. The computer-readable storage medium of claim 10, wherein, The first neural network layer generates the first layer output by applying convolution to the first layer input based on the current values of the parameter set of the first convolutional neural network layer to generate a modified first layer input, and then applying a nonlinear operation to the modified first layer input.
17. The computer-readable storage medium of claim 10, wherein, During the training of the neural network, the neural network system is configured to backpropagate the normalized statistics as part of adjusting the parameter values of the neural network.
18. A method for processing image data input, the method being performed by one or more computers and comprising: Receive network input from the user's computer, including an image or image features of the image; The network input is processed using a convolutional neural network, the convolutional neural network being configured to receive the network input and generate a network output representing the image, the convolutional neural network comprising: Multiple neural network layers, including a first convolutional neural network layer and a second neural network layer; and A batch normalization layer, situated between a first convolutional neural network layer and a second neural network layer, wherein the first convolutional neural network layer generates a first layer output having multiple components indexed by feature indexing and spatial location indexing, and wherein the batch normalization layer is configured to, during training of the convolutional neural network on training example batches: Receive the corresponding first-layer output for each training example in the batch; Calculate multiple normalized statistics for the batch based on the output of the first layer, wherein calculating the multiple normalized statistics for the output of the first layer includes, for each feature index in the feature index: Calculate the average value of the components of the first layer output corresponding to the feature index; and Calculate the variance of the component of the first layer output corresponding to the feature index; The normalization statistics are used to normalize each component of each first layer output to generate the corresponding normalized layer output for each training example in the batch. Generate the corresponding batch normalization layer output for each training example from the normalization layer output; and The batch normalization layer output is provided as the input to the second neural network layer; and The network output is provided as the output of the one or more computers.
19. The method according to claim 18, wherein, Normalizing each component of the output from each layer includes: The component is normalized using the mean and variance of the feature index corresponding to the component.
20. The method according to claim 18, wherein, Generating the corresponding batch normalization layer output for each training example from the normalization layer output includes: The component is transformed based on the current value of the parameter set of the feature index corresponding to each component of the normalization layer output.