Neural Network Training Using Soft Nearest Neighbor Loss

Abstract

Methods, systems, and apparatus for training a classification neural network, including a computer program encoded on a computer storage medium. In one aspect, a method includes: for each of a plurality of network inputs: processing the network input using a classification neural network to generate a classification output defining a predicted class of the network input; determining a soft nearest neighbor loss, wherein the soft nearest neighbor loss encourages intermediate representations of network inputs of different classes to become more entangled, wherein the entanglement of intermediate representations of network inputs of different classes characterizes how similar pairs of intermediate representations of network inputs of different classes are relative to pairs of intermediate representations of network inputs of the same class; and adjusting current values ​​of the classification neural network parameters using the gradient of the soft nearest neighbor loss relative to the classification neural network parameters.

Description

Technical Field

[0001] This specification relates to processing data using machine learning models. Background Art

[0002] A machine learning model receives an input and generates an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate an output based on the received input and the values ​​of the model's parameters.

[0003] Some machine learning models are deep models that use a multi-layer model to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to a received input to generate an output. Summary of the invention

[0004] This specification describes a training system implemented as a computer program on one or more computers at one or more locations, the training system configured to train a classification neural network and a generative neural network using a soft nearest neighbor loss.

[0005] According to a first aspect, a method for training a classification neural network performed by one or more data processing devices is provided, the method comprising: for each of a plurality of network inputs: processing the network input using the classification neural network according to current values ​​of the classification neural network parameters to generate a classification output defining a predicted category of the network input. For each of a plurality of pairs of network inputs including a first network input and a second network input from the plurality of network inputs, a soft nearest neighbor loss is determined based on a corresponding similarity measure between: (i) an intermediate representation of the first network input, the intermediate representation of the first network input being generated by one or more hidden layers of the classification neural network by processing the first network input to generate a classification output for the first network input, and (ii) an intermediate representation of the second network input, the intermediate representation of the second network input being generated by one or more hidden layers of the classification neural network by processing the second network input to generate a classification output for the second network input. The soft nearest neighbor loss encourages the intermediate representations of network inputs of different categories to become more entangled, wherein the entanglement of the intermediate representations of network inputs of different categories characterizes the degree of similarity of the intermediate representation pairs of network inputs of different categories relative to the intermediate representation pairs of network inputs of the same category. The gradient of the soft nearest neighbor loss with respect to the classification neural network parameters is used to adjust the current values ​​of the classification neural network parameters.

[0006] In some implementations, determining a soft nearest neighbor loss includes: for each given network input in a plurality of network inputs: determining an intra-class variation of the given network input, the intra-class variation of the given network input characterizing how similar the intermediate representation of the given network input is to the intermediate representations of other network inputs in the plurality of network inputs belonging to the same category as the given network input. Determining a total variation of the given network input, the total variation of the given network input characterizing how similar the intermediate representation of the given network input is to the intermediate representations of other network inputs in the plurality of network inputs belonging to any category. Determining a ratio of the intra-class variation to the total variation of the given network input; and determining the soft nearest neighbor loss based on the respective ratios of the intra-class variation to the total variation for each given network input.

[0007] In some implementations, determining the intra-class variation for the given network input includes determining:

[0008]

[0009] Wherein, j indexes the network input of the plurality of network inputs, b is the total number of network inputs in the plurality of network inputs, i is the index of the given network input, and y i Represents the category of the given network input, y j represents the category of the network input corresponding to index j, x i represents the intermediate representation of the given network input, x j denotes the intermediate representation of the network input corresponding to index j, S(·, ·) is the similarity measure, and T is the temperature parameter.

[0010] In some implementations, determining the total change for the given network input includes determining:

[0011]

[0012] Wherein, j indexes the network input of the plurality of network inputs, b is the total number of network inputs in the plurality of network inputs, i is the index of the given network input, and x i represents the intermediate representation of the given network input, x j denotes the intermediate representation of the network input corresponding to index j, S(·, ·) is the similarity measure, and T is the temperature parameter.

[0013] In some implementations, determining the soft nearest neighbor loss based on respective ratios of the intra-class variation and the total variation for each given network input includes determining the soft nearest neighbor loss as:

[0014]

[0015] Wherein, b is the total number of network inputs in the plurality of network inputs, i indexes the given network input, and R i represents the ratio of the intra-class variation and the total variation for the given network input corresponding to index i.

[0016] In some implementations, using a gradient of a soft nearest neighbor loss with respect to the classification neural network parameter to adjust a current value of the classification neural network parameter includes using a gradient of a soft nearest neighbor loss with respect to the temperature parameter to adjust the current value of the temperature parameter.

[0017] In some implementations, the classification output defining the predicted class of the network input includes a respective likelihood score for each of a plurality of possible classes, wherein the likelihood score for a given class indicates the likelihood that the network input belongs to the given class.

[0018] In some implementations, the method further includes determining a classification loss based on a corresponding classification output defining a predicted class for each network input, and adjusting current values ​​of the classification neural network parameters using gradients of the classification loss relative to the classification neural network parameters.

[0019] In some implementations, the classification loss includes a cross-entropy loss.

[0020] In some implementations, the plurality of pairs of network inputs includes every possible pair of network inputs consisting of a first network input and a second, different network input from the plurality of network inputs.

[0021] In some implementations, the network input is an image.

[0022] In some implementations, the category of an image defines a classification of an object depicted in the image.

[0023] According to a second aspect, a method for training a generative neural network to generate synthetic data elements based on a training data set of real data elements, performed by one or more data processing devices, is provided, the method comprising: generating a set of synthetic data elements using the generative neural network according to current values ​​of parameters of the generative neural network. Obtaining a set of real data elements from a training data set of real data elements. For each of a plurality of data element pairs including a first data element and a second data element, determining a soft nearest neighbor loss based on respective similarity measures between the first data element and the second data element, the first data element and the second data element being from a combined data element set including the set of synthetic data elements and the set of real data elements. The soft nearest neighbor loss encourages data elements of different categories to become more entangled. The category of the data element defines whether the data element is a real data element or a synthetic data element. The entanglement of data elements of different categories characterizes the degree of similarity of data element pairs of different categories relative to data element pairs of the same category. Using the gradient of the soft nearest neighbor loss relative to the parameters of the generative neural network to adjust the current values ​​of the parameters of the generative neural network.

[0024] In some implementations, determining a soft nearest neighbor loss includes: for each given data element from a combined data element set including a set of synthetic data elements and a set of real data elements: determining an intra-class variation for the given data element, the intra-class variation of the given data element characterizing how similar the given data element is to other data elements from the combined data element set belonging to the same class as the given data element. Determining a total variation for the given data element, the total variation of the given data element characterizing how similar the given data element is to other data elements from the combined data element set belonging to any class. Determining a ratio of the intra-class variation to the total variation for the given data element. Determining the soft nearest neighbor loss based on the respective ratios of the intra-class variation to the total variation for each given data element.

[0025] In some implementations, determining the intra-class variation for the given data element includes determining:

[0026]

[0027] Where j indexes a data element from the combined data element set, b is the total number of data elements in the combined data element set, i is the index of the given data element, and y i Indicates the category of the given data element, y j represents the category of the data element corresponding to the index j, x i Represents the given data element, x jdenotes the data element corresponding to index j, S(·,·) is the similarity measure, and T is the temperature parameter.

[0028] In some implementations, determining the total variation for a given data element includes determining:

[0029]

[0030] Where j indexes a data element from the combined data element set, b is the total number of data elements in the combined data element set, i is the index of the given data element, and x i Represents the given data element, x j denotes the data element corresponding to index j, S(·,·) is the similarity measure, S(·,·) is the similarity measure, and T is the temperature parameter.

[0031] In some implementations, determining the soft nearest neighbor loss based on respective ratios of the intra-class variation and the total variation for each given data element includes determining the soft nearest neighbor loss as:

[0032]

[0033] Wherein, b is the total number of data elements in the combined data element set, i indexes the given data element, and R i represents the ratio of the within-class variation to the total variation for a given data element corresponding to index i.

[0034] In some implementations, using the gradient of the soft nearest neighbor loss with respect to the generative neural network parameter to adjust the current value of the generative neural network parameter includes: using the gradient of the soft nearest neighbor loss with respect to the temperature parameter to adjust the current value of the temperature parameter.

[0035] In some implementations, the data element is an image.

[0036] According to a third aspect, a method for training a generative neural network to generate synthetic data elements based on a training data set of real data elements, performed by one or more data processing devices, is provided. The method includes using the generative neural network to generate a set of synthetic data elements based on current values ​​of parameters of the generative neural network. A set of real data elements is obtained from a training data set of real data elements. For each data element in a combined data element set including the set of synthetic data elements and the set of real data elements, a discriminator neural network is used to generate an embedding of the data element. For each data element pair in a plurality of data element pairs including a first data element and a second data element, a soft nearest neighbor loss is determined based on a respective similarity measure between an embedding of the first data element and an embedding of the second data element, the first data element and the second data element being from a combined data element set including the set of synthetic data elements and the set of real data elements. The soft nearest neighbor loss encourages embeddings of data elements of different categories to become more entangled. The category of the data element defines whether the data element is a real data element or a synthetic data element. The entanglement of the embeddings of data elements of different categories characterizes how similar the embeddings of the data element pairs of different categories are to the embeddings of the data element pairs of the same category. The gradient of the soft nearest neighbor loss with respect to the generative neural network parameters is used to adjust current values ​​of the generative neural network parameters.

[0037] In some implementations, determining a soft nearest neighbor loss includes, for each given data element from a combined data element set comprising a set of synthetic data elements and a set of real data elements: determining an intra-class variation for the given data element, the intra-class variation for the given data element characterizing how similar an embedding of the given data element is to embeddings of other data elements from the combined data element set belonging to the same class as the given data element. Determining a total variation for the given data element, the total variation for the given data element characterizing how similar an embedding of the given data element is to embeddings of other data elements from the combined data element set belonging to any class. Determining a ratio of the intra-class variation for the given data element to the total variation. Determining the soft nearest neighbor loss based on the respective ratios of the intra-class variation to the total variation for each given data element.

[0038] In some implementations, determining the intra-class variation for the given data element includes determining:

[0039]

[0040] Where j indexes a data element from the combined data element set, b is the total number of data elements in the combined data element set, i is the index of the given data element, and y iIndicates the category of the given data element, y j represents the category of the data element corresponding to index j, E(x i ) represents the embedding of the given data element, E(x j ) denotes the embedding of the data element corresponding to index j, S(·, ·) is the similarity measure, and T is the temperature parameter.

[0041] In some implementations, determining the total change for a given data element includes determining:

[0042]

[0043] Where j indexes a data element from the combined data element set, b is the total number of data elements in the combined data element set, i is the index of the given data element, and E(x i ) represents the embedding of the given data element, E(x j ) denotes the embedding of the data element corresponding to index j, S(·, ·) is the similarity measure, and T is the temperature parameter.

[0044] In some implementations, determining the soft nearest neighbor loss based on respective ratios of within-class variation to total variation for each given data element includes determining the soft nearest neighbor loss as:

[0045]

[0046] Wherein, b is the total number of data elements in the combined data element set, i indexes the given data element, and R i represents the ratio of the within-class variation to the total variation for a given data element corresponding to index i.

[0047] In some implementations, using the gradient of the soft nearest neighbor loss with respect to the generative neural network parameter to adjust the current value of the generative neural network parameter includes: using the gradient of the soft nearest neighbor loss with respect to the temperature parameter to adjust the current value of the temperature parameter.

[0048] In some implementations, the data element is an image.

[0049] In some implementations, the method further includes: using the gradient of the soft nearest neighbor loss with respect to the discriminator neural network parameters to adjust current values ​​of the discriminator neural network parameters.

[0050] In some implementations, adjusting current values ​​of the discriminator neural network parameters encourages the discriminator neural network to generate embeddings of data elements of different classes that are less entangled.

[0051] According to a fourth aspect, there is provided a method for classifying data performed by one or more data processing devices, the method comprising: providing input data to a classification neural network, the classification neural network having been trained by performing the method of the first aspect; classifying the input data using the classification neural network; and receiving a classification output from the classification neural network, the output indicating a category of the input data.

[0052] According to a fifth aspect, there is provided a method for generating synthetic data executed by one or more data processing devices, the method comprising: providing input data to a generative neural network, the generative neural network having been trained by executing the method of the second or third aspect; using the generative neural network to generate synthetic data based on the input data; and receiving the synthetic data from the generative neural network.

[0053] According to a sixth aspect, a system is provided, comprising one or more computers and one or more storage devices, wherein the storage devices store instructions, which, when executed by the one or more computers, cause the one or more computers to perform corresponding operations of the method.

[0054] According to a sixth aspect, there is provided a computer program product storing instructions which, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the method.

[0055] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0056] This specification describes a discriminative training system that trains a classification neural network using a soft nearest neighbor loss that encourages intermediate representations of network inputs from different categories (i.e., generated by the hidden layers of the classification neural network) to become more entangled. The soft nearest neighbor loss can regularize the classification neural network and improve its generalization ability, i.e., achieve acceptable prediction accuracy on network inputs that are not used during training, by causing the intermediate representations of the network inputs to characterize category-independent features that are used to capture information that improves classification accuracy. A feature can be referred to as "category-independent" if its value is likely to be similar for different categories of network inputs. By acting as a regularizer, the soft nearest neighbor loss can also reduce the consumption of computational resources (e.g., memory and computational power) by using less training data, fewer training iterations, or both to train the classification network.

[0057] The present specification also describes a generative training system that can train a generative neural network using a soft nearest neighbor loss that encourages "synthetic" data elements generated by the generative neural network to become more entangled with "genuine" data elements from a training data set. Training the generative neural network using a soft nearest neighbor loss enables the generative neural network to generate "realistic" synthetic data elements, i.e., synthetic data elements that have similar characteristics to genuine data elements in the training data set. The soft nearest neighbor loss can train the generative neural network by using less training data, fewer training iterations, or both. Therefore, the generative training system may consume fewer computing resources (e.g., memory and computing power) when training the generative neural network compared to some traditional training systems that do not use a soft nearest neighbor loss.

[0058] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter of this specification will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 Included are four panels, each illustrating a set of data points and an associated soft nearest neighbor loss value, which measures the entanglement of data points of different classes.

[0060] Figure 2 An example discriminative training system is shown.

[0061] Figure 3 An example generative training system is shown.

[0062] Figure 4 A diagram indicating the entanglement of intermediate representations of different classes of network inputs generated by a hidden layer in a classification neural network is shown.

[0063] Figure 5 is a flow chart of an example process for computing a soft nearest neighbor loss for a set of data points each associated with a corresponding class.

[0064] Figure 6 is a flowchart of an example process for training a classification neural network using a soft nearest neighbor loss.

[0065] Figure 7 is a flowchart of an example process for training a generative neural network using a soft nearest neighbor loss.

[0066] Like reference numbers and designations throughout the various drawings represent like elements.

[0067] Specific implementation method

[0068] As used in this specification, for a set of data points respectively associated with corresponding categories, “entanglement” of data points of different categories characterizes the degree of similarity between a pair of data points of different categories and a pair of data points of the same category.

[0069] A data point refers to an ordered set of values, such as a vector or matrix of values, which can represent, for example, an image, an audio data segment, or a portion of text. In one example, a data point can be an intermediate representation of a network input generated by one or more hidden layers of a neural network by processing the network input; in another example, a data point can be an embedding of the network input generated by the output layer of the neural network by processing the network input.

[0070] The class of a data point refers to the label of the data point (e.g., in the case of data points representing images, the class of the data point may specify the classification of an object depicted in the image). The similarity between pairs of data points may be measured using a numerical similarity metric (e.g., the Euclidean similarity metric or the cosine similarity metric).

[0071] Figure 1 Four panels (i.e., panels 102, 104, 106, and 108) are included, each illustrating a set of data points, wherein the category of each data point is distinguished by its shape and color. Each panel is also associated with a soft nearest neighbor loss value that measures the entanglement of data points of different categories. Panel 102 illustrates the data points with the highest soft nearest neighbor loss (i.e., the highest entanglement), while panel 108 illustrates the data points with the lowest soft nearest neighbor loss (i.e., the lowest entanglement). It can be appreciated that the data points with low entanglement shown in panel 108 are grouped into class homogeneous clusters, while the data points with high entanglement shown in panel 102 are not grouped.

[0072] The present specification describes a discriminative training system that uses a soft nearest neighbor loss to train a classification neural network that is configured to process a network input to generate a corresponding classification output that predicts the category of the network input. Training the classification neural network using a soft nearest neighbor loss encourages intermediate representations of network inputs from different categories (i.e., generated by hidden layers of the classification neural network) to become more entangled. The soft nearest neighbor loss can regularize the classification neural network by encouraging the intermediate representations of the network input to characterize category-independent features that capture information that improves classification accuracy. The soft nearest neighbor loss can be added as an additional term to the classification loss, which encourages the classification neural network to generate a classification output that matches a target output for a category of a specified network input.

[0073] The specification further describes a generative training system that uses a soft nearest neighbor loss to train a generative neural network that is configured to generate "synthetic" data elements having similar properties to "real" data elements from a training data set. Training the generative neural network using a soft nearest neighbor loss may encourage the synthetic data elements (or their embeddings) and the real data elements (or their embeddings) to become more entangled. Increasing the entanglement between the synthetic data elements and the real data elements may increase the realism of the synthetic data elements, for example, by increasing their similarities to the real data elements.

[0074] These and other features are described in more detail below.

[0075] Figure 2 An example discriminative training system 200 is shown. The discriminative training system 200 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the following systems, components, and techniques are implemented.

[0076] The discriminative training system 200 trains a classification neural network 202. The classification neural network 202 is configured to process a network input 204 to generate a corresponding classification output 206 defining a predicted category of the network input 204. The network input can be any kind of digital data input, such as image data, video data, audio data, or text data. The classification output can include a corresponding score for each of a plurality of possible categories, wherein the score for a category indicates the likelihood that the network input is from that category; the predicted category of the network input can be identified as the category with the highest score.

[0077] In one example, the network input may be an image or a feature derived from an image, and the category of the network input may specify a classification of the image. For example, the category of an image may specify whether the image depicts a particular type of object, such as a vehicle, a pedestrian, a road sign, etc. As another example, the category of a medical image may specify the medical condition of a patient depicted in the medical image. As another example, the category of an image may specify whether the image depicts inappropriate (e.g., offensive) content. As another example, the category of an image may specify the type of action being performed by a person depicted in the image (e.g., sitting, standing, running, etc.).

[0078] In another example, the web input may be a text sequence and the category of the web input may specify the intent expressed by the web input, such as an intent to perform a certain action.

[0079] In another example, the network input may be a series of audio data samples, and the category of the network input may specify a phoneme or grapheme corresponding to the audio data samples.

[0080] In another example, the web input may be an Internet resource (eg, a web page) or a document (or a portion thereof, or extracted features thereof), and the category of the web input may specify the subject matter of the web input.

[0081] The classification neural network may have any suitable neural network architecture, such as a feedforward architecture or a recursive architecture, and may include any suitable type of neural network layer or block, such as a fully connected layer, a convolutional layer, or a residual block. Typically, the classification neural network includes one or more hidden layers 208, i.e., the layer after the input layer and before the output layer in the architecture of the classification neural network. The output layer refers to the neural network layer that generates the classification output of the classification neural network, i.e., the layer that generates the class score indicating the likelihood that the network input is from each class.

[0082] The discriminative training system 200 trains a classification neural network 202 over multiple training iterations on a training data set including a plurality of training examples. Each training example specifies: (i) a network input, and (ii) a target (i.e., actual) category of the network input. In each training iteration, the discriminative training system 200 may obtain (e.g., sample) a "batch" (i.e., set) of current training examples from the training data, and process the network input 204 specified by each training example to generate: (i) an intermediate representation 210 of the network input 204, and (ii) a classified output 206 of the network input 204. The intermediate representation 210 of the network input 204 refers to the output generated by one or more hidden layers 208 of the classification neural network by processing the network input. The intermediate representation 210 may be represented as an ordered set of numerical values, such as a vector or matrix of numerical values. In one example, the intermediate representation 210 of the network input 214 may be the output of an assigned hidden layer 208 of the classification neural network. In another example, the intermediate representation 210 of the network input 214 may be a combination (e.g., a cascade) of the outputs of multiple assigned hidden layers of the classification neural network.

[0083] After processing the current batch of network inputs 204 from the training data, the discriminative training system 200 updates the current parameter values ​​of the classification neural network using one or both of: (i) a soft nearest neighbor loss 212, and (ii) a classification loss 214. In particular, the discriminative training system 200 updates the current parameter values ​​of the classification neural network using the gradients of the soft nearest neighbor loss 212, the classification loss 214, or both with respect to the current parameter values ​​of the classification neural network. For example, the discriminative training system 200 may use a composite loss function given by The gradient is used to update the current parameter value of the classification neural network:

[0084]

[0085] in, is the classification loss, is a soft nearest neighbor loss, and α>0 is a hyperparameter that controls the relative importance of the classification loss and the soft nearest neighbor loss. In other examples, the composite loss function may include multiple soft nearest neighbor losses, each corresponding to an intermediate output generated by a corresponding hidden layer of the classification neural network. That is, the discriminative training system 200 may generate multiple intermediate representations of each network input, and the composite loss function may include a corresponding soft nearest neighbor loss corresponding to each of these intermediate representations.

[0086] The discriminative training system 200 may use back-propagation techniques to calculate the gradients of the soft nearest neighbor loss 212 and the classification loss 214 and update the current parameter values ​​of the classification neural network 202 using any suitable gradient descent optimization process, such as RMSprop or Adam.

[0087] Using the soft nearest neighbor loss 212 to update the current parameter values ​​of the classification neural network 202 encourages the intermediate representations 210 of the network inputs 204 of different categories to become more entangled. That is, the soft nearest neighbor loss 212 encourages increasing similarity between the intermediate representation pairs of the network inputs 204 of different categories relative to the intermediate representation pairs of the network inputs of the same category. Figure 5 An example technique for calculating the soft nearest neighbor loss 212 is described in more detail.

[0088] Updating the current parameter values ​​of the classification neural network 202 using the classification loss 214 encourages the classification output 206 of the network input 204 to match the target class specified by the training examples. The classification loss 214 can be, for example, a cross entropy loss.

[0089] The use of soft nearest neighbor loss 212 can improve the ability of the trained classification neural network to generalize from training data to previously unseen network inputs, that is, it can improve the accuracy of the classification neural network for network inputs that were not used to train the classification neural network. Specifically, the soft nearest neighbor loss can regularize the classification neural network by encouraging the intermediate representation of the network input to represent category-independent features that capture information that improves classification accuracy. In the absence of soft nearest neighbor loss, the intermediate representation of the network input can form category homogeneous clusters (i.e., groups of intermediate representations that are mostly from the same category); if the intermediate representation of the network input is not represented in one of these category homogeneous clusters, the classification output may be inaccurate. The soft nearest neighbor loss does not encourage the formation of category homogeneous clusters of intermediate representations, thereby improving the generalization and robustness of the classification neural network.

[0090] In addition to regularizing the training of the classification neural network, the use of a soft nearest neighbor loss can also facilitate determining the confidence of a class prediction generated by a classification neural network for a "test" network input (i.e., a training network input that is not included in the training data). In order to determine the confidence of a class prediction generated for a test network input, an intermediate representation of a predefined K training network input that is most similar (i.e., closest) to the intermediate representation of the test network input can be identified. The confidence of the class prediction for the test network input can then be determined based on the fraction of the K nearest intermediate representations that share the same category as the predicted category of the test network input. In general, a higher fraction of the K nearest intermediate representations that share the same category as the predicted category of the test network input indicates a higher confidence in the class prediction for the test network input, and vice versa.

[0091] As described above, using a soft nearest neighbor loss can increase the correlation between the confidence of a class prediction and the accuracy of a class prediction by discouraging the formation of class-homogeneous clusters of intermediate representations. This is particularly important when a classification neural network may be provided with outlier test data, for example, in a medical diagnostic setting (where a patient may be suffering from an unknown condition), or due to an adversarial attack. In an adversarial attack, network inputs are provided to a classification neural network in an attempt to cause the classification neural network to generate inaccurate class predictions. By facilitating the assessment of the confidence of class predictions generated by a classification neural network, a soft nearest neighbor loss can, for example, improve the security of a computer system using a classification neural network by making the classification neural network (and by extension, the computer system) less susceptible to adversarial attacks by malicious actors.

[0092] Using a soft nearest neighbor loss can better handle anomalous data that is different from the training data used to train the classification neural network. Data that is not from the training distribution can be identified by observing the data in the hidden layer because it has fewer than the normal number of neighbors from the predicted class. In one example, this can allow for the detection of adversarial attacks. In another example, this can be used to aid in medical diagnosis when the patient's condition is unknown.

[0093] Figure 3 An example generative training system 300 is shown. Generative training system 300 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0094] Generative training system 300 trains generative neural network 302 using soft nearest neighbor loss 308 to generate "synthetic" data elements 304 having similar characteristics to the training data set of "real" data elements 306. The data elements may be, for example, images, text snippets, or audio snippets.

[0095] In general, generative neural network 302 may have any suitable neural network architecture that enables it to generate data elements, and generative neural network 302 may generate data elements in any of a variety of ways. In one example, to generate synthetic data elements 304, generative neural network 302 may process latent variables sampled from a probability distribution over a space of latent variables. In this example, the space of latent variables may be, for example, a set of real numbers, and the probability distribution over the latent space may be, for example, a normal (0, 1) probability distribution. In another example, to generate synthetic data elements 304, generative neural network 302 may generate a probability distribution over a space of possible data elements, and sample synthetic data elements according to the probability distribution.

[0096] The generative training system 300 trains the generative neural network in multiple training iterations. At each training iteration, the generative training system 300 uses the generative neural network 302 to generate the synthetic data elements 304 of the current batch (set), and obtains (e.g., samples) the real data elements 306 of the current batch (set). The generative training system 300 provides the current synthetic data elements 304 and the real data elements 306 as respective inputs to the discriminative neural network 310 (which may also be referred to as a "discriminator" neural network), which is configured to process the input data elements to generate an embedding 312 of the input data elements. The embedding of a data element refers to representing the data element as an ordered aggregation of numerical values, such as a vector or matrix of numerical values. The discriminative neural network 310 can have any suitable neural network architecture that enables it to generate an embedding, such as a fully connected or convolutional neural network architecture.

[0097] After generating the embedding 312 of the current synthetic and real data elements, the generative training system 300 updates the current parameter values ​​of the generative neural network using the soft nearest neighbor loss 308. In particular, the generative training system 300 updates the current parameter values ​​of the generative neural network using the gradient of the soft nearest neighbor loss 308 relative to the current parameter values ​​of the generative neural network. The generative training system 300 can use a back-propagation technique to calculate the gradient of the soft nearest neighbor loss 308 and use any suitable gradient descent optimization process (e.g., RMSprop or Adam) to update the current parameter values ​​of the generative neural network 302.

[0098] In general, for each data element pair in a plurality of data element pairs (e.g., data element pairs including two real data elements, two synthetic data elements, or one real data element and one synthetic data element), a soft nearest neighbor loss 308 is based on a measure of similarity between embeddings 312 of the data elements in the pair. Updating the current parameter values ​​of the generative neural network 302 using the soft nearest neighbor loss 308 encourages the embeddings 312 of the synthetic data elements 304 and the real data elements 306 to become more entangled. That is, if the category of the data element is understood as defining whether the data element is synthetic or real, the soft nearest neighbor loss 308 encourages an increase in similarity between embedding pairs 312 of data elements of different categories relative to embedding pairs of data elements of the same category. Reference Figure 5 Example techniques for calculating the soft nearest neighbor loss 308 are described in more detail.

[0099] The generative training system 300 trains the discriminative neural network 310 in coordination with the generative neural network 302, for example, by alternating between training the discriminative neural network 310 and the generative neural network 302. Specifically, at each of a plurality of training iterations, the generative training system 300 uses the discriminative neural network to generate embeddings 312 of the synthetic data elements 304 and the real data elements 306. The generative training system 300 then uses the soft nearest neighbor loss 308 to update the current parameter values ​​of the discriminative neural network to encourage the embeddings 312 of the synthetic data elements 304 and the real data elements 306 to become less entangled.

[0100] During the training process, the generative neural network 302 continuously gets better at generating synthetic data samples with embeddings that are more entangled with the embeddings of the real data samples. At the same time, the discriminative neural network continuously adapts to make the embeddings of the synthetic data elements less entangled with the embeddings of the real data elements. The adversarial training of the generative neural network 302 and the discriminative neural network 310 results in the characteristics of the synthetic data elements 304 generated by the generative neural network 302 becoming more and more similar to the characteristics of the real data elements 306.

[0101] The discriminative neural network 310 enables the generative training system 300 to evaluate the entanglement of synthetic and real data elements in the learned embedding space, which can facilitate more efficient training of the generative neural network 302. However, in simplified implementations, the generative training system 300 can avoid using the discriminative neural network 310. In these implementations, the generative training system 300 can train the generative neural network 302 by using a soft nearest neighbor loss 308 to encourage synthetic data elements to become more entangled with real data elements, i.e., in the data element space rather than in the embedding space. That is, in these implementations, for each of a plurality of data element pairs (e.g., a data element pair including two real data elements, two synthetic data elements, or one real data element and one synthetic data element), the soft nearest neighbor loss is a measure based on the similarity between the data elements in the pair. If the category of the data element is understood as defining whether the data element is synthetic or real, the soft nearest neighbor loss encourages increased similarity between data element pairs of different categories relative to data element pairs of the same category. Reference Figure 5 Example techniques for computing the soft nearest neighbor loss are described in more detail.

[0102] Figure 4 A graph indicating the entanglement (measured by the soft nearest neighbor loss) of intermediate representations of different classes of network inputs generated by a series of layers in a classification neural network (specifically the layers in the last block of a ResNet trained on the CIFAR-10 training dataset) is shown. It can be appreciated that during training using the soft nearest neighbor loss, except for the output layer (in Figure 4 The entanglement of the intermediate representations at each layer generally increases, except for the last layer (labeled as “last layer” in Figure 2). The soft nearest neighbor loss is not applied to the output layer, allowing the output layer to remain discriminative.

[0103] Figure 5 is a flow chart of an example process 500 for calculating a soft nearest neighbor loss for a set of data points respectively associated with a corresponding class. For convenience, process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system appropriately programmed according to the present specification, such as Figure 2 The discriminant training system 200 or Figure 3 The generation training system 300 can perform process 500.

[0104] Process 500 is described with reference to a set of data points each associated with a corresponding class. In one example, each data point can be an intermediate output generated by a hidden layer of a classification neural network by processing network inputs from a current batch of network inputs during training of the classification neural network, and the class of each data point can be the class of the corresponding network input, such as reference Figure 2In another example, each data point may be a data element or an embedding of a data element from a current batch of data elements during training of a generative neural network, and the category of each data point may indicate whether the data element is real or synthetic (i.e., generated by a generative neural network), as described in reference Figure 3 described.

[0105] The system determines a corresponding intra-class variation for each data point (502). The intra-class variation of a given data point characterizes the degree of similarity between the given data point and other data points belonging to the same class as the given data point. In one example, the system can determine the intra-class variation of a given data point as:

[0106]

[0107] where j indexes the data point, b is the total number of data points (e.g., the data points corresponding to the current batch), i is the index of a given data point, and y i represents the category of a given data point (e.g., whether data point i corresponds to a real data element or a synthetic data element), y j represents the category of the data point corresponding to index j (e.g., indicating whether data point j corresponds to a real data element or a synthetic data element), p i For a given data point, p j denotes the data point corresponding to index j, S(·,·) is a similarity measure (e.g., S(p i , p j )=|p i -p j | 2 ), and T is a temperature parameter that controls the relative importance given to similarities between pairs of data points.

[0108] The system determines the corresponding total variation for each data point (504). The total variation for a given data point characterizes how similar the data point is to other data points belonging to any category. In one example, the system can determine the total variation for a given data point as:

[0109]

[0110] where j indexes the data point, b is the total number of data points, i is the index of a given data point, and p i For a given data point, p j represents the data point corresponding to index j, S(·,·) is a similarity measure (e.g., S(p i , p j )=|p i -p j | 2), and T is a temperature parameter that controls the relative importance given to similarities between pairs of data points.

[0111] The system determines a soft nearest neighbor loss based on the respective ratios of the intra-class variation to the total variation for each data point (506). For example, the system may Determined as:

[0112]

[0113] Where b is the total number of data points, i indexes the data points, and R i represents the ratio of the within-class variation to the total variation for the data point corresponding to index i, and is expressed as a function of the temperature parameter T (as described above).

[0114] From equations (2) to (4), we can see that the soft nearest neighbor loss uses all data points in a batch of b data points. This causes the intermediate representation to become more entangled than the intermediate representation that may be produced by other loss functions based on a single positive data point and a single negative data point.

[0115] In some implementations, the system may apply the soft nearest neighbor loss Determined as the minimum value at all temperatures, that is:

[0116]

[0117] Where, refer to equation (4) to describe During training of a discriminative neural network or a generative neural network (as described above), the soft nearest neighbor loss (described in reference to Equation (5)) is approximated by initializing T to a predetermined value and optimizing using gradient descent on T to minimize the soft nearest neighbor loss, i.e., by adjusting the current value of T using the gradient of the soft nearest neighbor loss with respect to T. That is, during training of the discriminative neural network or the generative neural network, the value of T is jointly adjusted with the parameter values ​​of the discriminative neural network or the generative neural network to optimize the soft nearest neighbor loss. Determining the soft nearest neighbor loss as the minimum value under all temperatures eliminates any need to manually set the temperature hyperparameter value, such as through trial and error testing.

[0118] Figure 6 6 is a flow chart of an example process 600 for training a classification neural network using a soft nearest neighbor loss. For convenience, process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system (e.g., Figure 2 The discriminant training system 200 can perform process 600.

[0119] The system processes each of a plurality of network inputs using a classification neural network (602). The classification neural network is configured to process the network inputs to generate a classification output defining a predicted class of the network input.

[0120] The system determines a soft nearest neighbor loss (604). For each pair of network inputs in a plurality of pairs of network inputs including a first network input and a second network input, the system determines a soft nearest neighbor loss based on a corresponding similarity measure between: (i) an intermediate representation of the first network input, and (ii) an intermediate representation of the second network input.

[0121] The system uses the gradient of the soft nearest neighbor loss with respect to the classification neural network parameters to adjust the current values ​​of the classification neural network parameters (606). The soft nearest neighbor loss encourages intermediate representations of different classes of network inputs to become more entangled.

[0122] After adjusting the current values ​​of the classification neural network parameters, the system can return to step 602 to perform another training iteration. The system can determine that training is complete when a training termination criterion is met, for example, when a predetermined number of training iterations have been performed, or when the accuracy of the classification neural network on the validation data set reaches a predefined threshold.

[0123] Figure 7 is a flow chart of an example process 700 for training a generative neural network using a soft nearest neighbor loss. For convenience, process 700 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system appropriately programmed according to the present specification, such as Figure 3 The generation training system 300 can perform process 700.

[0124] The system generates a current set of synthetic data elements using a generative neural network (702).

[0125] The system obtains a current set of real data elements from a training data set of real data elements ( 704 ), for example, by sampling a predefined number of real data elements from the training data set.

[0126] The system generates, using a discriminative neural network, a corresponding embedding for each data element in a combined set of data elements including the current set of synthetic data elements and the current set of real data elements (706).

[0127] The system determines a soft nearest neighbor loss 708. For each data element pair in a plurality of data element pairs including a first data element and a second data element from the combined data element set, the system determines a soft nearest neighbor loss based on a corresponding similarity measure between an embedding of the first data element and an embedding of the second data element.

[0128] The system uses the gradient of the soft nearest neighbor loss with respect to the generated neural network parameters to adjust the current values ​​of the generated neural network parameters (710). The soft nearest neighbor loss encourages the embedding of the synthetic data elements to become more entangled with the embedding of the real data elements.

[0129] After adjusting the current values ​​of the generated neural network parameters, the system can return to step 702 to perform another training iteration. The system can determine that training is complete when a training termination criterion is met, such as when a predefined number of training iterations have been performed.

[0130] This specification uses the term "configured" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination of software, firmware, hardware that, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0131] Embodiments of the subject matter and functional operations described in this specification may be implemented with digital electronic circuits, with tangibly implemented computer software or firmware, with computer hardware including the structures disclosed in this specification and their structural equivalents, or with a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0132] The term "data processing apparatus" refers to data processing hardware and includes all kinds of apparatus, devices and machines for processing data, including by way of example a programmable processor, a computer or a plurality of processors or computers. An apparatus may also be or further include special-purpose logic circuitry, for example, an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). An apparatus may optionally include, in addition to hardware, code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0133] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language including compiled or interpreted languages ​​or declarative or procedural languages; and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document; in a single file dedicated to the program or in multiple coordinated files, such as files storing one or more modules, subroutines or portions of code. A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a data communications network.

[0134] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed and run on the same computer or multiple computers.

[0135] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit such as an FPGA or ASIC, or by a combination of a special purpose logic circuit and one or more programmed computers.

[0136] A computer suitable for executing a computer program may be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or any other kind of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for executing or implementing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory may be supplemented by a dedicated logic circuit or incorporated in the dedicated logic circuit. Typically, a computer will also include one or more large-capacity storage devices for storing data, such as a disk, a magneto-optical disk, or an optical disk, or may be operationally coupled to receive data from the one or more large-capacity storage devices or to transfer data to the one or more large-capacity storage devices, or both for storing data. However, a computer need not have such a device. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game controller, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, etc.

[0137] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including by way of example semiconductor storage devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0138] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on the user's device in response to receiving a request from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device, such as a smart phone running a messaging application, and then receiving a response message from the user.

[0139] The data processing apparatus for implementing the machine learning model may also include, for example, dedicated hardware accelerator units for processing common and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0140] The machine learning model can be implemented and deployed using a machine learning framework, such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0141] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component, such as a data server; or includes a middleware component, such as an application server; or includes a front-end component, such as a client computer with a graphical user interface, a web browser, or an app that a user can use to interact with an implementation of the subject matter described in this specification; or includes any combination of one or more such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0142] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by means of computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data such as HTML pages to the user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from the user. Data generated at the user device, for example, the result of the user interaction, may be received from the device at the server.

[0143] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or that may be claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. In addition, although features may be described above as working in certain combinations and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may be directed to subcombinations or variations of subcombinations.

[0144] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0145] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims may be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for training a classification neural network performed by one or more data processing devices, the method comprising: For each of a plurality of network inputs, wherein the network input comprises image data, video data, audio data, or text data: processing the network input using the classification neural network based on current values ​​of the classification neural network parameters to generate a classification output defining a predicted class for the network input; For each pair of network inputs in a plurality of pairs of network inputs including a first network input and a second network input from the plurality of network inputs, a soft nearest neighbor loss is determined based on a corresponding similarity measure between: (i) an intermediate representation of the first network input, the intermediate representation of the first network input generated by one or more hidden layers of the classification neural network by processing the first network input to generate a classification output for the first network input, and (ii) an intermediate representation of the second network input, the intermediate representation of the second network input generated by the one or more hidden layers of the classification neural network by processing the second network input to generate a classification output for the second network input; wherein the soft nearest neighbor loss encourages intermediate representations of network inputs of different categories to become more entangled, wherein the entanglement of intermediate representations of network inputs of different categories characterizes the degree of similarity of intermediate representation pairs of network inputs of different categories relative to intermediate representation pairs of network inputs of the same category; and The current values ​​of the classification neural network parameters are adjusted using the gradients of the soft nearest neighbor loss with respect to the classification neural network parameters.

2. The method of claim 1, wherein: Determining the soft nearest neighbor loss includes: For each given network input of the plurality of network inputs: Determining an intra-class variation of the given network input, the intra-class variation of the given network input characterizing a degree of similarity between the intermediate representation of the given network input and intermediate representations of other network inputs of the plurality of network inputs belonging to the same class as the given network input; determining a total variation of the given network input, the total variation of the given network input characterizing how similar the intermediate representation of the given network input is to intermediate representations of other network inputs of the plurality of network inputs belonging to any category; and determining a ratio of the within-class variation to the total variation for the given network input; and The soft nearest neighbor loss is determined based on a respective ratio of the within-class variation to the total variation for each given network input.

3. The method of claim 2, wherein: Determining the intra-class variation for the given network input includes determining: Wherein, j indexes a network input in the plurality of network inputs, b is the total number of network inputs in the plurality of network inputs, i is the index of the given network input, and y i Represents the category of the given network input, y j represents the category of the network input corresponding to index j, x i represents the intermediate representation of the given network input, x j denotes the intermediate representation of the network input corresponding to index j, S(·,·) is the similarity measure, and T is the temperature parameter.

4. The method of claim 2, wherein: Determining the total change for the given network input includes determining: Wherein, j indexes a network input in the plurality of network inputs, b is the total number of network inputs in the plurality of network inputs, i is the index of the given network input, and x i represents the intermediate representation of the given network input, x j denotes the intermediate representation of the network input corresponding to index j, S(·,·) is the similarity measure, and T is the temperature parameter.

5. The method of claim 2, wherein: Determining the soft nearest neighbor loss based on the respective ratios of the within-class variation to the total variation for each given network input includes determining the soft nearest neighbor loss as: Wherein, b is the total number of network inputs in the plurality of network inputs, i indexes the given network input, and R i represents the ratio of the within-class variation to the total variation for the given network input corresponding to index i.

6. The method of claim 3, wherein: Using the gradient of the soft nearest neighbor loss with respect to the classification neural network parameter to adjust the current value of the classification neural network parameter includes: A current value of the temperature parameter is adjusted using a gradient of the soft nearest neighbor loss with respect to the temperature parameter.

7. The method of claim 1, wherein: The classification output defining the predicted class of the network input includes a respective likelihood score for each of a plurality of possible classes, wherein the likelihood score for a given class indicates a likelihood that the network input belongs to the given class.

8. The method of claim 1, further comprising: determining a classification loss based on a corresponding classification output defining the predicted class for each network input; as well as The current values ​​of the classification neural network parameters are adjusted using the gradients of the classification loss with respect to the classification neural network parameters.

9. The method of claim 8, wherein: The classification loss includes cross entropy loss.

10. The method of claim 1, wherein: The plurality of pairs of network inputs includes each possible pair of network inputs including a first network input and a second, different network input from the plurality of network inputs.

11. The method according to any one of claims 1 to 10, wherein: The network input is an image.

12. The method of claim 11, wherein: The class of an image defines the classification of objects depicted in the image.

13. A method for training a generative neural network to generate synthetic data elements based on a training data set of real data elements, wherein the training data set of real data elements includes image data, video data, audio data or text data, performed by one or more data processing devices, the method comprising: generating a set of synthetic data elements using the generative neural network based on current values ​​of parameters of the generative neural network; Obtaining a set of real data elements from a training data set of the real data elements; For each data element pair of a plurality of data element pairs including a first data element and a second data element, a soft nearest neighbor loss is determined based on a corresponding similarity measure between the first data element and the second data element, the first data element and the second data element being from a combined set of data elements including the set of synthetic data elements and the set of real data elements, wherein: The soft nearest neighbor loss encourages data elements of different categories to become more entangled, The category of a data element defines whether the data element is a real data element or a synthetic data element, and The entanglement of data elements of different categories characterizes the degree of similarity of pairs of data elements of different categories relative to pairs of data elements of the same category; as well as The current values ​​of the generative neural network parameters are adjusted using the gradients of the soft nearest neighbor loss with respect to the generative neural network parameters.

14. The method of claim 13, wherein: Determining the soft nearest neighbor loss includes: For each given data element from the combined set of data elements comprising the set of synthetic data elements and the set of real data elements: determining an intra-class variation for the given data element, the intra-class variation of the given data element characterizing a degree of similarity of the given data element to other data elements from the set of combined data elements belonging to the same class as the given data element; determining a total variation for the given data element, the total variation for the given data element characterizing how similar the given data element is to other data elements from the set of combined data elements belonging to any category; and determining a ratio of the within-class variation to the total variation for the given data element; and The soft nearest neighbor loss is determined based on a respective ratio of the within-class variation to the total variation for each given data element.

15. The method of claim 14, wherein: Determining the intra-class variation for the given data element comprises determining: Wherein, j indexes a data element from the set of combined data elements, b is the total number of data elements in the set of combined data elements, i is the index of the given data element, and y i represents the category of the given data element, y j represents the category of the data element corresponding to index j, x i Represents the given data element, x j represents the data element corresponding to index j, S(·,·) is the similarity measure, and T is the temperature parameter.

16. The method of claim 14, wherein: Determining the total variation for the given data element comprises determining: Wherein, j indexes a data element from the set of combined data elements, b is the total number of data elements in the set of combined data elements, i is the index of the given data element, and x i Represents the given data element, x j represents the data element corresponding to index j, S(·,·) is the similarity measure, S(·,·) is the similarity measure, and T is the temperature parameter.

17. The method of claim 14, wherein: Determining the soft nearest neighbor loss based on the respective ratios of the within-class variation to the total variation for each given data element includes determining the soft nearest neighbor loss as: Wherein, b is the total number of data elements in the combined data element set, i indexes the given data element, and R i Represents the ratio of the within-class variation to the total variation for the given data element corresponding to index i.

18. The method of claim 15, wherein: Using the gradient of the soft nearest neighbor loss with respect to the generative neural network parameter to adjust the current value of the generative neural network parameter comprises: A current value of the temperature parameter is adjusted using a gradient of the soft nearest neighbor loss with respect to the temperature parameter.

19. The method of claim 13, wherein: The data element is an image.

20. A method for training a generative neural network to generate synthetic data elements based on a training data set of real data elements, wherein the training data set of real data elements comprises image data, video data, audio data or text data, performed by one or more data processing devices, the method comprising: generating a set of synthetic data elements using the generative neural network based on current values ​​of parameters of the generative neural network; Obtaining a set of real data elements from a training data set of the real data elements; For each data element in a combined set of data elements including the set of synthetic data elements and the set of real data elements, generating an embedding of the data element using a discriminator neural network; For each of a plurality of data element pairs including a first data element and a second data element, determining a soft nearest neighbor loss based on a corresponding similarity measure between an embedding of the first data element and an embedding of the second data element, the first data element and the second data element being from the combined set of data elements including the set of synthetic data elements and the set of real data elements, wherein: The soft nearest neighbor loss encourages the embeddings of data elements of different categories to become more entangled, The category of a data element defines whether the data element is a real data element or a synthetic data element, and The entanglement of embeddings of data elements of different categories characterizes how similar embeddings of pairs of data elements of different categories are relative to embeddings of pairs of data elements of the same category; as well as The current values ​​of the generative neural network parameters are adjusted using the gradients of the soft nearest neighbor loss with respect to the generative neural network parameters.

21. The method of claim 20, wherein: Determining the soft nearest neighbor loss includes: For each given data element from the combined set of data elements comprising the set of synthetic data elements and the set of real data elements: determining an intra-class variation for the given data element, the intra-class variation of the given data element characterizing how similar the embedding of the given data element is to embeddings of other data elements from the set of combined data elements belonging to the same class as the given data element; determining a total variation for the given data element, the total variation for the given data element characterizing how similar the embedding of the given data element is to embeddings of other data elements from the set of combined data elements belonging to any category; and determining a ratio of the within-class variation to the total variation for the given data element; and The soft nearest neighbor loss is determined based on a respective ratio of the within-class variation to the total variation for each given data element.

22. The method of claim 21, wherein: Determining the intra-class variation for the given data element comprises determining: Wherein, j indexes a data element from the set of combined data elements, b is the total number of data elements in the set of combined data elements, i is the index of the given data element, and y i represents the category of the given data element, y j represents the category of the data element corresponding to index j, E(x i ) represents the embedding of the given data element, E(x j ) represents the embedding of the data element corresponding to index j, S(·, ·) is the similarity measure, and T is the temperature parameter.

23. The method of claim 21, wherein: Determining the total change for the given data element comprises determining: Wherein, j indexes a data element from the set of combined data elements, b is the total number of data elements in the set of combined data elements, i is the index of the given data element, and E(x i ) represents the embedding of the given data element, E(x j ) represents the embedding of the data element corresponding to index j, S(·, ·) is the similarity measure, and T is the temperature parameter.

24. The method of claim 21, wherein: Determining the soft nearest neighbor loss based on the respective ratios of the within-class variation to the total variation for each given data element includes determining the soft nearest neighbor loss as: Wherein, b is the total number of data elements in the combined data element set, i indexes the given data element, and R i Represents the ratio of the within-class variation to the total variation for the given data element corresponding to index i.

25. The method of claim 22, wherein: Using the gradient of the soft nearest neighbor loss with respect to the generative neural network parameter to adjust the current value of the generative neural network parameter comprises: A current value of the temperature parameter is adjusted using a gradient of the soft nearest neighbor loss with respect to the temperature parameter.

26. The method of claim 20, wherein: The data element is an image.

27. The method of claim 20, further comprising: The gradient of the soft nearest neighbor loss with respect to the discriminator neural network parameters is used to adjust the current values ​​of the discriminator neural network parameters.

28. The method of claim 27, wherein: Adjusting the current values ​​of the discriminator neural network parameters encourages the discriminator neural network to generate embeddings of data elements of different classes that are less entangled.

29. A method for classifying data performed by one or more data processing devices, the method comprising: Providing input data to a classification neural network, the input data comprising image data, video data, audio data or text data, the classification neural network having been trained by executing the method according to any one of claims 1 to 12; classifying the input data using the classification neural network; and A classification output is received from the classification neural network, the output indicating a category of the input data.

30. A method for generating synthetic data performed by one or more data processing devices, the method comprising: Providing input data to a generative neural network, the input data comprising image data, video data, audio data or text data, the generative neural network having been trained by executing the method according to any one of claims 13 to 28; generating synthetic data based on the input data using the generative neural network; and The synthesized data is received from the generative neural network.

31. A system comprising one or more computers and one or more storage devices, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform corresponding operations of the method of any one of claims 1-30.

32. A computer program product storing instructions, which, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the method of any one of claims 1-30.

Images