Methods, systems, and storage media for training neural networks

Abstract

This disclosure relates to methods, systems, and storage media for training neural networks. Methods, systems, and apparatus for training neural networks, including computer programs encoded on a computer storage medium, are provided, wherein the neural network is configured to receive input data items and process the input data items to generate corresponding scores. The method includes the actions of: obtaining a set of training data comprising a plurality of training items, wherein each training item is associated with a corresponding label from a predetermined set of a plurality of labels; and modifying the training data to generate regularized training data, including: for each training item, determining whether to modify the label associated with the training item, and changing the label associated with the training item to a different label from a predetermined set of labels, and training the neural network on the regularized data. Aspects of this disclosure have the technical effect of improving the performance of trained neural networks by reducing overfitting.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201680060065.2, filed on November 4, 2016.

[0003] Cross-reference to related applications

[0004] This application claims the benefit of U.S. Provisional Application No. 62 / 252,374, filed November 6, 2015, entitled “Regularizing Machine Learning Models,” which is incorporated herein by reference in its entirety. Technical Field

[0005] This manual relates to training neural networks. Background Technology

[0006] Neural networks are machine learning models that use one or more layers to generate outputs from received inputs, such as for classification. Some neural networks contain one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input to the next layer, which is either the next hidden layer or the network's output layer. Each layer of the network generates an output from the received input based on the current values ​​of its corresponding set of parameters.

[0007] Neural networks can overfit their training data. Overfitting can be described as a neural network becoming overconfident due to a particular set of training data. When a neural network is overfitted, it may start to make poor generalizations about terms that are not in the training data. Summary of the Invention

[0008] The aspects disclosed herein have the technical effect of improving the performance of trained neural networks by reducing overfitting.

[0009] According to one innovative aspect of this disclosure, a method for training a neural network is described, wherein the neural network is configured to receive input data items and process the input data items to generate a corresponding score for each label in a predetermined set of multiple labels. The method may include the following actions: obtaining a set of training data comprising multiple training items, wherein each training item is associated with a corresponding label from the predetermined set of multiple labels; and modifying the training data to generate regularized training data for regularizing the training of the neural network. The modification actions may include: for each training item, determining whether to modify the label associated with the training item; and in response to determining that the label associated with the training item should be modified, changing the label associated with the training item to a different label from the predetermined set of labels; and training the neural network on the regularized data.

[0010] According to various innovative aspects of this disclosure, the performance of a trained neural network can be improved by using a regularized dataset to reduce overfitting. The regularized dataset can be generated by introducing a predetermined amount of noise into the labels of the training dataset, for example, by modifying one or more labels associated with training data items in the training dataset. Training data items may include training images.

[0011] Other aspects include corresponding systems, apparatuses, and computer programs for performing actions coded on a computer storage device. For a system of one or more computers configured to perform a specific operation or action, this means that software, firmware, hardware, or a combination thereof are installed on the system, which, in operation, causes the system to perform said operation or action. For one or more computer programs configured to perform a specific operation or action, this means that one or more programs include instructions that, when executed by a data processing apparatus, cause the apparatus to perform said operation or action.

[0012] These and other aspects may optionally include one or more of the following features. For example, in some implementations, changing the label associated with the training item to a different label from the predetermined set of labels includes changing the label from one that correctly describes the training item to one that does not. In some implementations, the different label is randomly selected from the predetermined set of labels.

[0013] In some implementations, the labels may include a training label distribution comprising scores for the training item for each label in a predetermined set of labels associated with a set of training images. In such an implementation, changing the label associated with the training item to a different label from the predetermined set of labels may include changing the distribution of scores in the training label distribution of the training data item from a distribution representing scores of correct labels to a distribution representing scores of incorrect labels.

[0014] In some implementations, determining whether to modify the label associated with the training item can be based on a predetermined probability. In some such implementations, the predetermined probability can be 10%.

[0015] According to another innovating aspect of this disclosure, a method for training a neural network is described, wherein the neural network is configured to receive input data items and process the input data items to generate a corresponding score for each label in a predetermined set of multiple labels. The method may include the following actions: obtaining a set of training data comprising multiple training items, wherein each training item is associated with an initial target label distribution that assigns a corresponding target score to each label in the predetermined set of labels; modifying the training data to generate regularized training data for training the regularized neural network, comprising, for each training item, modifying the initial target label distribution by combining the initial target label distribution with a smoothed label distribution to generate a modified target label distribution; and training the neural network on the regularized training data.

[0016] Other aspects include corresponding systems, apparatuses, and computer programs for performing actions coded on computer storage devices.

[0017] These and other aspects may optionally include one or more of the following features. For example, in some implementations, combining the initial target label distribution with the smooth label distribution may include: calculating a weighted sum of the initial target label distribution and the smooth label distribution.

[0018] In some implementations, the target score for the known label of the training item can be assigned a predetermined positive value in the initial target distribution of the training item, and the target score for each label other than the known label can be set to 0 in the initial target distribution.

[0019] In some implementations, the smoothed label distribution may include a corresponding smoothing score for each label in the set of predetermined labels, and each smoothing score may be the same predetermined value.

[0020] In some implementations, the smoothed label distribution may include a corresponding smoothing score for each label in the predetermined set of labels, and the smoothing score may be non-uniform.

[0021] According to another innovating aspect of this disclosure, a method for training a neural network is provided, wherein the neural network is configured to receive input data items and process the input data items to generate a corresponding score for each label in a predetermined set of multiple labels. The method may include the actions of: receiving a request to train the neural network to optimize a loss function including a first error term; and training the neural network to optimize a regularized loss function, the regularized loss function including the first error term and a regularized error term, the regularized error term penalizing the neural network based on an error between a predicted distribution and a smooth distribution.

[0022] Other aspects include corresponding systems, apparatuses, and computer programs for performing actions coded on computer storage devices.

[0023] These and other aspects may optionally include one or more of the following features. For example, in some implementations, the smooth distribution may be a uniform distribution. Alternatively, the smooth distribution may be a distribution used prior to the predicted distribution. In some implementations, the predicted distribution may include a set of scores generated by a neural network for a particular input.

[0024] In some embodiments, a computer program includes instructions that, when executed by a computing device, cause it to perform any of the methods disclosed herein.

[0025] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. The accuracy of inferences generated by the trained neural network can be improved. Specifically, by modifying the labels associated with the training data used to train the neural network, it is possible to prevent the neural network from overfitting the training data when making inferences based on information learned by the neural network during training, i.e., to prevent it from becoming overly reliant and overconfident, thereby improving the performance of the neural network on new inputs after training. Attached Figure Description

[0026] Figure 1 This is a block diagram of an example system used to train a neural network.

[0027] Figure 2 This is a context graph of an example system for generating a set of regularized training data.

[0028] Figure 3 This is a flowchart of an example process for generating a set of regularized training data.

[0029] Figure 4 This is a flowchart of another example process used to generate a set of regularized training data.

[0030] Figure 5 This is a flowchart of an example process that uses regularized error terms to optimize the loss function.

[0031] Details of these and other implementations are set forth in the accompanying drawings and the description below. Other features and advantages will become apparent from the specification, drawings, and claims. Detailed Implementation

[0032] This specification describes how a system, implemented as a computer program on one or more computers at one or more locations, can regularize the training of a neural network by generating a set of regularized training data. The neural network is configured to receive input data items and process the input data items to generate a corresponding score for each label in a predetermined set of multiple labels.

[0033] The system can receive a training data set comprising a set of multiple training items for training a neural network. Each training item in the set of multiple training items can be associated with a corresponding training label distribution that associates the corresponding score with each label in the set of labels. The neural network can then be trained iteratively by optimizing a specified objective function that takes the neural network output generated by the neural network in response to the neural network input and the target output of the neural network input as inputs.

[0034] Figure 1 This is a block diagram of an example of a neural network training system 100 for training neural network 110. The neural network training system 100 is an example of a system in which the systems, components, and techniques described below can be implemented, and which can be implemented as a computer program on one or more computers at one or more locations. The neural network training system 100 includes a neural network 110 and a database 120 of training data items.

[0035] The neural network 110 is configured to receive input data items and process the input data items to generate a corresponding score for each label in a predetermined set of multiple labels.

[0036] The neural network 110 can be configured to receive any kind of digital data input and generate any kind of score or classification output based on the input. For example, if the input to the neural network 110 is an image or features already extracted from an image, the output generated by the neural network 110 for a given image can be a score for each of a set of object categories, where each score represents an estimated probability that the image contains an object belonging to that category.

[0037] As another example, if the input to the neural network 110 is an Internet resource (e.g., a webpage), a document or a portion of a document, or features extracted from an Internet resource, document or portion of a document, then the output generated by the neural network 110 for a given Internet resource, document or portion of a document can be a score for each of a set of topics, where each score represents the probability that the Internet resource, document or portion of a document is an estimate about that topic.

[0038] As another example, if the input to neural network 110 is features for personalized recommendations to a user, such as features characterizing the context for the recommendation, or features characterizing the user's previous actions, then the output generated by neural network 110 could be a score for each of the set of content items, where each score represents an estimate of the likelihood that the user will respond favorably to the recommended content item. In some of these examples, neural network 110 is part of a reinforcement learning system that provides content recommendations to a user.

[0039] As another example, if the input to the neural network 110 is text in one language, the output generated by the neural network 110 can be a score for each of a set of text pieces in another language, where each score represents an estimate of the probability that the text piece in the other language is an appropriate translation of the input text into that other language.

[0040] As another example, if the input to the neural network 110 is features of spoken speech, the output generated by the neural network 110 can be a score for each of the set of text segments, each score representing an estimate of the probability that the text segment is a correct transcription of the spoken speech.

[0041] In order to allow the neural network 110 to generate accurate output for the received data items, the neural network training system 100 trains the neural network 110 to adjust the values ​​of the parameters of the neural network 110, for example, to determine the values ​​of the trained parameters from the initial values.

[0042] When training neural network 110, neural network training system 100 uses training items from a database 120 of labeled training items. Database 120 stores a set of multiple training items, each associated with a corresponding label. Typically, the labels used for training items identify one or more correct labels for the training item, i.e., one or more labels that should be identified as one or more labels for the training item by the scores generated by neural network 110. In some implementations, the label data for a given training item is a score distribution containing the corresponding scores for each label in the set of labels, where the scores reflect the correct one or more labels for the training item. For example, training data item 122 may be associated with training label 122a.

[0043] Specifically, the neural network training system 100 trains the neural network 110 to minimize the loss function 130. Typically, the loss function 130 depends on: (i) the network output generated by the neural network 110 by processing a given training term; and (ii) the label for the training term, i.e., the target output that the neural network 110 should generate by processing the training term.

[0044] The neural network training system 100 is able to iteratively adjust the values ​​of the parameters of the neural network 110 by performing multiple iterations of conventional neural network training techniques (e.g., stochastic gradient descent with backpropagation) on training items from the database 120, thereby training the neural network 110 to minimize the loss function 130.

[0045] To reduce overfitting and improve the performance of the trained neural network, the neural network training system 100 regularizes the training of the neural network 110 by (i) modifying the label data for the training terms before using the training terms to train the neural network 110, or (ii) modifying the loss function 130 used to train the neural network 110. See below for reference. Figures 2-4 Describe the modification of tag data, and refer to the following: Figure 5 Describe the modification of the loss function.

[0046] Figure 2 This is a context graph of an example of a system 200 that generates a set of regularized training data. The system includes a neural network training system 210 and a database 220 of training data items.

[0047] At stage A, the set of training data items stored in database 220 comprises a set of n training images 222A-1, 222A-2, 222A-3, 222A-n, where n is any positive integer. Each image in this set of training images is associated with a label 223A-1, 223A-2, 223A-3, 223A-n describing the classification associated with each corresponding training image 222A-1, 222A-2, 222A-3, 222A-n. For example, training image 222A-1 is labeled "cat" 223A-1, training image 222A-2 is labeled "dog" 223A-2, training image 222A-3 is labeled "snake" 223A-3, and training image 222A-n is labeled "bear" 223A-n. For ease of description, the labels 223A-1, 223A-2, 223A-3, 223A-n associated with each training image 222A-1, 222A-2, 222A-3, 222A-n are shown and described as English words. However, in implementation, each label could be a distribution of scores for recognizing English words.

[0048] For example, the set of predetermined labels associated with the set of training data images at stage A may include “cat”, “dog”, “snake”, and “bear”. Thus, for example, training images such as training image 222A-1 may have training label distributions of “.6”, “.2”, “.1”, and “.2” for the set of labels “cat”, “dog”, “snake”, and “bear”, respectively.

[0049] In some implementations, the training label distributions of ".6", ".2", ".1", and ".2" for the sets of labels "cat", "dog", "snake", and "bear" respectively can indicate that the training image 222A-1 is labeled as "cat", since the highest score in the training label distribution corresponds to the category "cat". Alternatively, the training label distribution can be a one-hot distribution. In such a distribution, the value assigned to the correct label is a positive value, such as "1", and all other labels are assigned values ​​such as "0". Therefore, a one-hot training label distribution of "1", "0", "0", and "0" for the sets of labels "cat", "dog", "snake", and "bear" respectively can indicate that image 222A-1 is classified as a cat.

[0050] In some implementations, a neural network trained using the training data provided at stage A may tend to overfit. In this case, the neural network may begin to process the training data precisely and become overconfident. However, when the neural network is overfitted, it may begin to make poor generalizations about images that are not in the training data. For example, even after the neural network has been trained to classify an image as a cat even though it has accurately classified the entire set of labeled training data items, it may still begin to classify an image as a cat. Aspects of this disclosure attempt to reduce overfitting by training the neural network using a regularized training data set. The regularized training data set can be generated by modifying one or more labels associated with the training data items in the training data set used to train the neural network.

[0051] Typically, a regularized training dataset can be generated by introducing a predetermined amount of noise into the labels of the training dataset. For example, a neural network training system 210 can process the training dataset at stage A and determine whether to modify the labels 223A-1, 223A-2, 223A-3, 223A-n associated with specific training data items such as training images 222A-1, 222A-2, 222A-3, 222A-n. In some implementations, the neural network training system 210 can randomly determine whether to modify the labels 223A-1, 223A-2, 223A-3, 223A-n associated with specific training data items such as training images 222A-1, 222A-2, 222A-3, 222A-n. Alternatively, the neural network training system 210 may determine the time for modifying the labels 223A-1, 223A-2, 223A-3, 223A-n associated with specific training data items such as training images 222A-1, 222A-2, 222A-3, 222A-n. For example, the neural network training system 210 may determine the time for modifying the labels 223A-1, 223A-2, 223A-3, 223A-n associated with specific training data items such as training images 222A-1, 222A-2, 222A-3, 222A-n, etc., at 10%, 20%, 30%, etc.

[0052] In some implementations, modifying the labels 223A-1, 223A-2, 223A-3, 223A-n associated with specific training data items such as training images 222A-1, 222A-2, 222A-3, 222A-n may include changing the labels 223A-1, 223A-2, 223A-3, 223A-n associated with specific training data items such as training images 222A-1, 222A-2, 222A-3, 222A-n from correct labels to incorrect labels. Modifying the labels may, for example, change the distribution of scores in the training label distribution of the training data items from a distribution representing scores of correct labels to a distribution representing scores of incorrect labels. Alternatively, modifying the labels may, for example, include changing the distribution of scores in the training label distribution of the training data items to reduce the recognition of labels with a particular class. For example, changing the distribution of scores in the training label distribution of the training data items may include, for example, reducing the highest score in the training label distribution by a predetermined amount. Similarly, changing the distribution of scores in the training label distribution of training data items can include, for example, increasing one or more of the lowest scores in the training label distribution by a predetermined amount. The resulting training data set can be referred to as the regularized data set.

[0053] refer to Figure 2 For example, neural network training system 210 can obtain the set of training images shown at stage A. Neural network training system 210 can process the set of training images shown at stage A and determine whether to modify the labels 223A-1, 223A-2, 223A-n associated with each corresponding training image 222A-1, 222A-2, 222A-3, 222A-n. Figure 2 In the example, the neural network training system 210 may randomly determine to change the label associated with training image 222A-2 from the label "dog" to the label "snake" 223B-2. Although training image 222A-2 is correctly labeled as "dog", the neural network training system 210 modifies the label 223A-2 such that images 222A-2 and 222B-2 are now incorrectly labeled as "snake" 223B-2 at stage B. This modification results in the set of regularized training images shown at stage B.

[0054] The neural network can then be trained using the regularized training image shown at stage B. Using the regularized training image shown at stage B to train the neural network helps the neural network become less dependent on the training data and produce better generalizations based on the processed training data.

[0055] Alternatively, a label modification process known as label smoothing can be employed. For example, assume a distribution and smoothing parameter ε on the labels u(k) independent of the training example x. For a training example with the correct label y, the neural network training system 210 can replace the label distribution q(k|x) = δ k,y , where δ k,y It is the Dirac increment, which is 1 for k = y and 0 when k is not equal to y, where:

[0056] q′(k|x)=(1-∈)δ k,y +∈u(k)

[0057] Therefore, the new label distribution q′ is a mixture of the original correctly labeled ground-truth data distribution q(k|x) and the fixed distribution u(k), with weights 1-∈ and ∈, respectively. This can be viewed as the distribution of label k obtained by first setting k to the correctly labeled data label k = y, and then using a neural network training system 210 to replace k with samples drawn from the fixed distribution u(k) with probability ∈.

[0058] In some implementations, a uniform distribution can be used to achieve label smoothing regularization by assigning u(k) = 1 / K (where K is the number of labels), thereby...

[0059]

[0060] Figure 2 The examples provided illustrate the random modification of a single label for a single training data item. However, this disclosure does not require such limitation. In some implementations, multiple labels may be modified. Furthermore, in some implementations, some or all of the labels may be modified via smoothing using the label smoothing process described above to create a set of regularized training data.

[0061] Figure 3 This is a flowchart of process 300 for generating a set of regularized training data. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a system appropriately programmed according to this specification, such as... Figure 1 The neural network training system 100 can perform processing 300.

[0062] At stage 302, the system obtains a set of training data from a database of training data items. This set of training data may include multiple training data items. Each training data item may include a content item or a set of one or more features extracted from a content item. Each training data item in the set of training data items is associated with a corresponding label from a predetermined set of multiple labels. The labels may include a training label distribution, which comprises scores for training images for each label in the predetermined set of labels associated with the set of training images.

[0063] At stage 304, the system determines whether to modify the training data to generate regularized training data for training the regularized neural network. For each training data item in the set of training data items, the system determines whether to modify the label associated with that training item. Determining whether to modify the label associated with the training item may include, for example, randomly determining to modify the label associated with a particular training data item. Alternatively, the system may determine to modify the label associated with a particular training data item with a predetermined probability. For example, the system may determine to modify the label associated with a particular training data item, such as a training image, 10% of the time, 20% of the time, 30% of the time, etc.

[0064] At stage 306, when the system determines to modify the label associated with a training item, the system modifies the label associated with the training item by changing the label associated with the training item to a different label selected from a predetermined set of labels. In some implementations, modifying the label associated with a particular training data item may include changing the label associated with the particular training data item from a correct label to an incorrect label. For example, changing the label may include changing the distribution of scores in the training label distribution of the training data item from a distribution representing scores of correct labels to a distribution representing scores of incorrect labels. Alternatively, modifying the label may include, for example, changing the distribution of scores in the training label distribution of the training data item to reduce the recognition of labels of a particular class. For example, changing the distribution of scores in the training label distribution of the training data item may include, for example, reducing the highest score in the training label distribution by a predetermined amount. Similarly, for example, changing the distribution of scores in the training label distribution of the training data item may include, for example, increasing one or more of the lowest scores in the training label distribution by a predetermined amount. The resulting training data set may be referred to as a regularized training data set.

[0065] At stage 308, the system trains the neural network using a regularized training dataset. This training can be performed as described in the reference... Figure 1 Perform as described. However, instead of using the reference... Figure 1 The training data items described in database 120, stage 308, will use regularized training data sets generated by processing stages 302, 304 and 306 respectively to train the neural network.

[0066] Figure 4 This is a flowchart of another process 400 for generating a set of regularized training data. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a system appropriately programmed according to this specification, such as... Figure 1 The neural network training system 100 can perform 400 processes.

[0067] At stage 402, the system obtains a set of training data from a database of training data items. Each training data item in this set is associated with an initial target label distribution.

[0068] At stage 404, the system obtains data identifying a smooth label distribution. The smooth label distribution may include a smoothing score for each label in the distribution. In some implementations, the smooth label distribution may be a uniform distribution, which assigns the same smoothing score to each label in the distribution, as described above. Alternatively, in other implementations, the smooth label distribution may be a non-uniform distribution, which includes one or more smoothing scores that can differ from one or more other smoothing scores in the same distribution.

[0069] At stage 406, the system modifies the training data to generate regularized training data for training the regularizable neural network. In some implementations, modifying the training data includes, for each training data item in the set of training data items obtained in stage 402, combining the initial training distribution with a smoothed label distribution to generate a modified target label distribution. Combining the initial training distribution with the smoothed label distribution may include, for example, calculating a weighted sum of the initial target label distribution and the smoothed label distribution. The resulting set of training data may be referred to as the regularized training data set.

[0070] At stage 408, the system trains the neural network using a regularized training dataset, which includes training data with labels modified using the smoothed label distribution described at stage 406. (See reference...) Figure 1 The training is performed as described. However, instead of using the reference... Figure 1 The training data items described in database 120, stage 408, will use regularized training data sets generated by processing stages 402, 404 and 406 respectively to train the neural network.

[0071] Figure 5 This is a flowchart of a process 500 for minimizing a loss function with a regularization error term. For convenience, process 500 will be described as being executed by a system of one or more computers located in one or more locations. For example, a system appropriately programmed according to this specification, such as... Figure 1 The neural network training system 100 can perform 500 processing operations.

[0072] At stage 502, the system receives a request to train the neural network to optimize a loss function that includes a first error term. In one example, the first error term could be a cross-entropy loss. The cross-entropy loss can be defined as:

[0073]

[0074] Where p(k) is the probability that label k from the set of K labels is reflected by the label score generated by the neural network for a specific training item, and q(k) is the correctly labeled data value for each label k from the set of K labels.

[0075] Instead of modifying the label data associated with one or more training data labels, the system can alternatively seek to achieve label smoothing regularization by modifying the cross-entropy loss function as follows:

[0076]

[0077] Therefore, the label smoothing regularization described above is equivalent to replacing the single cross-entropy loss H(q,p) with a pair of losses, including a first error term H(q,p) and a second error term H(u,p), which are in the form of cross-entropy loss. The second error term H(u,p) is called the regularization error term, which penalizes the neural network based on the error calculated between the set of scores generated by the neural network and a smooth distribution that includes the smoothed score for each label in the set. For example, the smooth distribution could be a uniform distribution u, which, in association with the smooth distribution, assigns the same score to each label in the set of labels.

[0078] At stage 504, the system trains the neural network to minimize a regularization loss function that includes a first error term and a regularization error term. This training can be performed as described in the reference. Figure 1 It will be executed as described. However, the loss function 130 of system 100 will be replaced by a regularized loss function.

[0079] Embodiments of the subject matter, functional operation, and processing described in this specification (including the structures disclosed herein and their structural equivalents) can be implemented in digital electronic circuits, tangibly implemented computer software or firmware, computer hardware, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-volatile program carrier, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiving device for execution by the data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or combinations thereof.

[0080] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0081] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program may (but not necessarily) correspond to a file in a file system. A program may be stored as a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as multiple coordinating files (e.g., a file that stores one or more modules, subroutines, or code sections). A computer program can be deployed to execute on a single computer or on multiple computers located at a site or distributed across multiple sites and interconnected through a communications network.

[0082] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating outputs. The processing and logic flow can also be executed by special-purpose logic circuitry (e.g., FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit)), and the apparatus can also be implemented as such special-purpose logic circuitry.

[0083] As an example, a computer suitable for executing computer programs includes a central processing unit (CPU) that may be based on a general-purpose or special-purpose microprocessor, or both, or any other kind. Typically, the CPU receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are the CPU for executing or running instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or devices operatively coupled to the mass storage device to receive data from or transfer data to, or both. However, a computer does not need to have such a device. Furthermore, a computer can be embedded in another device, such as, to name just a few, mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, GPS receivers, or portable storage devices (e.g., Universal Serial Bus (USB) flash drives).

[0084] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices like EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM discs. Processors and memory may be supplemented or incorporated therein by dedicated logic circuitry.

[0085] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending and receiving documents to and from the device used by the user, for example by sending web pages to a web browser on the user's device in response to a request received from a web browser.

[0086] Embodiments of the subject matter described herein can be implemented in computing systems that include back-end components (e.g., as data servers), middleware components (e.g., application servers), front-end components (e.g., client computers having a graphical user interface or web browser through which users can interact with implementations of the subject matter described herein), or any combination of one or more such back-end, middleware, or front-end components. Components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0087] A computing system can include clients and servers. Clients and servers are typically geographically separated and interact via communication networks. The relationship between clients and servers is generated by computer programs running on various computers that have a client-server relationship with each other.

[0088] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of the claims, but rather as descriptions of features that may be specific to particular embodiments. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, one or more features from a claimed combination may be removed from the combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0089] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or to perform all shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0090] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions listed in the claims can be performed in a different order and still achieve the desired result. As an example, the processing depicted in the drawings does not necessarily require the specific order or sequence shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous. Additional steps or stages may be provided from the described processing, or steps or stages may be eliminated. Therefore, other implementations are within the scope of the appended claims.

Claims

1. A computer-implemented method for training a neural network, wherein the neural network is configured to receive input data items and process the input data items to generate a corresponding score for each label in a predetermined set of multiple labels, wherein: The input to the neural network is an image or features extracted from the image, and the output of the neural network is a score for each object category in the set of object categories of the image, where each score represents an estimated probability that the image contains an image belonging to an object of a certain category. The input to the neural network is an Internet resource, a document, or a portion of a document, or features extracted from the Internet resource, the document, or the portion of the document, and the output of the neural network is a score for each topic in a set of topics, each score representing the probability that the Internet resource, the document, or the portion of the document is an estimate of a topic. The input to the neural network is features for personalized recommendations to the user, and the output of the neural network is a score for each content item in the set of content items, each score representing the estimated probability that the user will respond favorably to being recommended a content item. The input to the neural network is text in one language, and the output of the neural network is a score for each text segment in a set of text segments in another language, each score representing an estimated probability that the text segment in the other language is an appropriate translation of the input text into the other language. The input to the neural network is features of spoken speech, and the output of the neural network is a score for each text segment in the set of text segments, each score representing an estimated probability that the text segment is a correct transcription of the spoken speech. The method includes: Multiple training items are obtained, each training item being associated with an initial target label distribution that assigns a corresponding target score to each label in the set of the predetermined multiple labels; For each training term, the initial target label distribution is combined with the smoothed label distribution to obtain a modified initial target label distribution, wherein the smoothed label distribution includes a corresponding smoothing score for each label in the predetermined set of multiple labels; and The neural network is trained using the plurality of training terms and an initial target label distribution with corresponding modifications to the plurality of training terms.

2. The method according to claim 1, wherein, The smoothed label distribution is a uniform distribution, which includes the same smoothing score for each label in the predetermined set of multiple labels.

3. The method according to claim 1, wherein, The smoothed label distribution is a non-uniform distribution, which includes a smoothed score for at least one label in the predetermined set of multiple labels that is different from the smoothed score for at least one other label in the predetermined set of multiple labels.

4. The method according to claim 1, wherein, Combining the initial target label distribution with the smoothed label distribution includes: Calculate the weighted sum of the initial target label distribution and the smoothed label distribution.

5. The method according to claim 1, wherein, For each training item: The target score for the known label of the training item is assigned a predetermined positive value in the initial target label distribution of the training item; and The target score for each label, excluding the known labels, is set to 0 in the initial target label distribution.

6. The method according to claim 4, wherein, Calculating the weighted sum of the initial target label distribution and the smoothed label distribution includes: Calculate the sum of the first term and the second term, where the first term is obtained by applying weight w to the smoothed label distribution, and where the second term is obtained by applying weight 1-w to the initial target label distribution.

7. A system for training a neural network, wherein, The neural network is configured to receive input data items and process the input data items to generate a corresponding score for each label in a predetermined set of multiple labels, wherein: The input to the neural network is an image or features extracted from the image, and the output of the neural network is a score for each object category in the set of object categories of the image, where each score represents an estimated probability that the image contains an image belonging to an object of a certain category. The input to the neural network is an Internet resource, a document, or a portion of a document, or features extracted from the Internet resource, the document, or the portion of the document, and the output of the neural network is a score for each topic in a set of topics, each score representing the probability that the Internet resource, the document, or the portion of the document is an estimate of a topic. The input to the neural network is features for personalized recommendations to the user, and the output of the neural network is a score for each content item in the set of content items, each score representing the estimated probability that the user will respond favorably to being recommended a content item. The input to the neural network is text in one language, and the output of the neural network is a score for each text segment in a set of text segments in another language, each score representing an estimated probability that the text segment in the other language is an appropriate translation of the input text into the other language. The input to the neural network is features of the spoken speech, and the output of the neural network is a score for each text segment in the set of text segments, each score representing an estimated probability that the text segment is a correct transcription of the spoken speech. The system includes: One or more data processing devices; and One or more storage devices storing instructions, which, when executed by the one or more data processing devices, cause the one or more data processing devices to perform operations for training the neural network, the operations including: Multiple training items are obtained, each training item being associated with an initial target label distribution that assigns a corresponding target score to each label in the set of the predetermined multiple labels; For each training term, the initial target label distribution is combined with the smoothed label distribution to obtain a modified initial target label distribution, wherein the smoothed label distribution includes a corresponding smoothing score for each label in the predetermined set of multiple labels; and The neural network is trained using the plurality of training terms and an initial target label distribution with corresponding modifications to the plurality of training terms.

8. The system according to claim 7, wherein, The smoothed label distribution is a uniform distribution, which includes the same smoothing score for each label in the predetermined set of multiple labels.

9. The system according to claim 7, wherein, The smoothed label distribution is a non-uniform distribution, which includes a smoothed score for at least one label in the predetermined set of multiple labels that is different from the smoothed score for at least one other label in the predetermined set of multiple labels.

10. The system according to claim 7, wherein, Combining the initial target label distribution with the smoothed label distribution includes: Calculate the weighted sum of the initial target label distribution and the smoothed label distribution.

11. The system according to claim 7, wherein, For each training item: The target score for the known label of the training item is assigned a predetermined positive value in the initial target label distribution of the training item; and The target score for each label, excluding the known labels, is set to 0 in the initial target label distribution.

12. The system according to claim 10, wherein, Calculating the weighted sum of the initial target label distribution and the smoothed label distribution includes: Calculate the sum of the first term and the second term, where the first term is obtained by applying weight w to the smoothed label distribution, and where the second term is obtained by applying weight 1-w to the initial target label distribution.

13. A non-transitory computer-readable medium for storing software, said software comprising instructions executable by one or more computers, said instructions, when executed, causing said one or more computers to perform operations for training a neural network, wherein, The neural network is configured to receive input data items and process the input data items to generate a corresponding score for each label in a predetermined set of multiple labels, wherein: The input to the neural network is an image or features extracted from the image, and the output of the neural network is a score for each object category in the set of object categories of the image, where each score represents an estimated probability that the image contains an image belonging to an object of a certain category. The input to the neural network is an Internet resource, a document, or a portion of a document, or features extracted from the Internet resource, the document, or the portion of the document, and the output of the neural network is a score for each topic in a set of topics, each score representing the probability that the Internet resource, the document, or the portion of the document is an estimate of a topic. The input to the neural network is features for personalized recommendations to the user, and the output of the neural network is a score for each content item in the set of content items, each score representing the estimated probability that the user will respond favorably to being recommended a content item. The input to the neural network is text in one language, and the output of the neural network is a score for each text segment in a set of text segments in another language, each score representing an estimated probability that the text segment in the other language is an appropriate translation of the input text into the other language. The input to the neural network is features of the spoken speech, and the output of the neural network is a score for each text segment in the set of text segments, each score representing an estimated probability that the text segment is a correct transcription of the spoken speech. The operation includes: Multiple training items are obtained, each training item being associated with an initial target label distribution that assigns a corresponding target score to each label in the set of the predetermined multiple labels; For each training term, the initial target label distribution is combined with the smoothed label distribution to obtain a modified initial target label distribution, wherein the smoothed label distribution includes a corresponding smoothing score for each label in the predetermined set of multiple labels; and The neural network is trained using the plurality of training terms and the corresponding modified initial target label distribution of the plurality of training terms.

14. The non-transitory computer-readable medium according to claim 13, wherein, The smoothed label distribution is a uniform distribution, which includes the same smoothing score for each label in the predetermined set of multiple labels.

15. The non-transitory computer-readable medium according to claim 13, wherein, The smoothed label distribution is a non-uniform distribution, which includes a smoothed score for at least one label in the predetermined set of multiple labels that is different from the smoothed score for at least one other label in the predetermined set of multiple labels.

16. The non-transitory computer-readable medium according to claim 13, wherein, Combining the initial target label distribution with the smoothed label distribution includes: Calculate the weighted sum of the initial target label distribution and the smoothed label distribution.

17. The non-transitory computer-readable medium according to claim 13, wherein, For each training item: The target score for the known label of the training item is assigned a predetermined positive value in the initial target label distribution of the training item; and The target score for each label, excluding the known labels, is set to 0 in the initial target label distribution.

18. The non-transitory computer-readable medium according to claim 16, wherein, Calculating the weighted sum of the initial target label distribution and the smoothed label distribution includes: Calculate the sum of the first term and the second term, where the first term is obtained by applying weight w to the smoothed label distribution, and where the second term is obtained by applying weight 1-w to the initial target label distribution.

19. A method for training a neural network, wherein the neural network is configured to receive input data items and process the input data items to generate a corresponding score for each label in a predetermined set of a plurality of labels, wherein: The input to the neural network is an image or features extracted from the image, and the output of the neural network is a score for each object category in the set of object categories of the image, where each score represents an estimated probability that the image contains an image belonging to an object of a certain category. The input to the neural network is an Internet resource, a document, or a portion of a document, or features extracted from the Internet resource, the document, or the portion of the document, and the output of the neural network is a score for each topic in a set of topics, each score representing the probability that the Internet resource, the document, or the portion of the document is an estimate of a topic. The input to the neural network is features for personalized recommendations to the user, and the output of the neural network is a score for each content item in the set of content items, each score representing the estimated probability that the user will respond favorably to being recommended a content item. The input to the neural network is text in one language, and the output of the neural network is a score for each text segment in a set of text segments in another language, each score representing an estimated probability that the text segment in the other language is an appropriate translation of the input text into the other language. The input to the neural network is features of the spoken speech, and the output of the neural network is a score for each text segment in the set of text segments, each score representing an estimated probability that the text segment is a correct transcription of the spoken speech. The method includes: Receive a request to train the neural network to optimize a loss function including a first error term; and The neural network is trained to optimize a regularized loss function, which includes a first error term and a regularized error term, the regularized error term penalizing the neural network based on an error between (1) the distribution of predictions generated by the neural network when processing a particular data item and (2) a smooth distribution including smooth scores for the set of predetermined multiple labels.

20. The method according to claim 19, wherein, The smooth distribution is a uniform distribution and is the distribution used prior to the predicted distribution.

Images