Classifying out-of-distribution data using contrastive loss

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network to (i) generate accurate network outputs for a machine learning task and (ii) generate intermediate outputs that can be used to reliably classify out-of-distribution inputs. In one aspect, a method includes training a neural network using a supervised and contrastive loss, including repeatedly performing the following operations: obtaining a first network input and a second network input; processing each network input using the neural network to generate its respective network input embedding, processing the first network input using the neural network to generate a network output; and adjusting network parameter values using a supervised and contrastive loss gradient, where: the supervised loss is based on (i) the network output and (ii) a corresponding target network output; and the contrastive loss is based on at least: (i) the first network input embedding, and (ii) the second network input embedding.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Application No. 63 / 035,578, filed June 5, 2020, the disclosure of which is incorporated herein by reference. Background Technology

[0003] This manual relates to processing data using machine learning models.

[0004] Machine learning models receive input and generate outputs, such as predicted outputs, based on the received input. Some machine learning models are parametric models and generate outputs based on the received input and the values ​​of the model's parameters.

[0005] Some machine learning models are deep models that use multiple layers to generate outputs from received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a non-linear transformation to the received input to generate an output. Summary of the Invention

[0006] This specification generally describes a system implemented as a computer program on one or more computers at one or more locations for training a neural network to: (i) generate accurate network outputs for machine learning tasks, and (ii) generate intermediate outputs that can be used to reliably classify out-of-distribution network inputs.

[0007] As used throughout this document, a neural network input can be referred to as "out-of-distribution" if it is drawn from a different distribution than the distribution of the set of network inputs used to train the neural network. Generally, out-of-distribution network inputs may differ significantly from the set of network inputs used to train the neural network. For example, out-of-distribution network inputs may differ significantly from the set of network inputs used to train the neural network in terms of "style" (e.g., visual appearance), "content" (e.g., semantic content), or both.

[0008] For example, if a neural network performs a classification task by categorizing network inputs into a predetermined set of possible classes, and a given network input is not included in any of the possible classes, then the given network input can be referred to as "out of distribution." In a specific example, if a neural network performs a classification task by categorizing medical images into classes of possible diseases, then a medical image corresponding to a disease not included in any of the possible disease classes may be out of distribution. For example, a given network input may also be referred to as out of distribution if (i) the similarity between a given network input and (ii) the set of training network inputs used to train the neural network does not meet a threshold. For example, if a neural network performs a classification task by categorizing medical images into classes of possible diseases, then a medical image corresponding to one of the possible disease classes but having visual characteristics significantly different from the training network input may be out of distribution.

[0009] If a given network input is not out-of-distribution, it can be called "in-distribution".

[0010] As used throughout this document, "embedded" can refer to an ordered collection of values, such as a vector or matrix of numbers.

[0011] As used throughout this article, the "intermediate output" of a neural network can refer to the data generated by one or more hidden layers of the neural network through processing the network input. The intermediate output of a neural network can be represented as an ordered set of numerical values, such as a vector or matrix of values.

[0012] According to a first aspect, a method is provided, executed by one or more data processing devices, for training a neural network having multiple network parameters to (i) generate accurate network outputs for a machine learning task, and (ii) generate intermediate outputs that can be used to reliably classify out-of-distribution network inputs. The method includes training the neural network using a supervised loss and a contrastive loss, the method comprising repeatedly performing the following operations: obtaining a first network input and a second network input; processing the first network input using the neural network to generate: (i) an embedding of the first network input, as an embedding of an intermediate output of the neural network, and (ii) a network output corresponding to the first network input; processing the second network input using the neural network to generate at least one embedding of the second network input as an intermediate output of the neural network; and adjusting the current values ​​of the network parameters using gradients of the supervised loss and the contrastive loss, wherein: the supervised loss is based on the error between (i) the network output corresponding to the first network output and (ii) a target network output corresponding to the first network input; and the contrastive loss is based at least on: (i) the embedding of the first network input and (ii) the embedding of the second network input.

[0013] In some implementations, using the gradient of the contrastive loss to adjust the current values ​​of the network parameters includes determining the contrastive loss based at least in part on the similarity between the embeddings of the first network input and the second network input.

[0014] In some implementations, the first network input and the second network input both correspond to the same training network input, and adjusting the current values ​​of the network parameters using the gradient of the contrastive loss includes: using the gradient to adjust the current values ​​of the network parameters to encourage the neural network to generate corresponding embeddings of the first network input and the second network input that are more similar.

[0015] In some implementations, obtaining the first network input and the second network input includes: sampling a first transformation from a set of possible transformations and applying the first transformation to the same training network input to generate the first network input; and sampling a second transformation from a set of possible transformations and applying the second transformation to the same training network input to generate the second network input.

[0016] In some implementations, the first network input and the second network input do not correspond to the same training network input, and adjusting the current values ​​of the network parameters using the gradient of the contrastive loss includes: using the gradient to adjust the current values ​​of the network parameters to encourage the neural network to generate corresponding embeddings of the less similar first network input and second network input.

[0017] In some implementations, determining the contrast loss based at least in part on the similarity between the embeddings of the first network input and the second network input includes: processing the embedding of the first network input using a projection function to generate a projected embedding of the first network input; processing the embedding of the second network input using a projection function to generate a projected embedding of the second network input; and determining the similarity between the embeddings of the first network input and the second network input based on a similarity measurement between (i) the projected embedding of the first network input and (ii) the projected embedding of the second network input.

[0018] In some implementations, the similarity measurement is a cosine similarity measurement.

[0019] In some implementations, the projection function includes a projection neural network with multiple projection neural network parameters.

[0020] In some implementations, the method also includes using the gradient of the contrastive loss to adjust the current values ​​of the projection neural network parameters.

[0021] In some implementations, the projected embedding of the first network input has a lower dimension than the first network input, and the projected embedding of the second network input has a lower dimension than the second network input.

[0022] In some implementations, the method further includes: after training the neural network, obtaining a test network input; processing the test network input using the neural network to generate: (i) an embedding of the test network input as an intermediate output of the trained neural network, and (ii) a network output corresponding to the test network input; and determining, at least in part based on the embedding of the test network input, that the test network input is an out-of-distribution input.

[0023] In some implementations, determining that a test network input is an out-of-distribution input based at least in part on the embedding of the test network input includes: determining the probability that the embedding of the test network input falls under a probability distribution in the space of possible embeddings; and determining that the test network input is an out-of-distribution input based on the probability that the embedding of the test network input falls under a probability distribution in the space of possible embeddings.

[0024] In some implementations, the probability distribution over the space of possible embeddings is determined based on a collection of embeddings in the space of possible embeddings, wherein each embedding is generated by processing the network input in the distribution using a trained neural network to generate an embedding of the network input in the distribution as an intermediate output of the trained neural network.

[0025] According to another aspect, a method is provided that includes using a neural network to perform a machine learning task, the neural network having been trained using any of the methods described above.

[0026] In some implementations, the machine learning task includes a classification task, and the neural network is configured to process network inputs to generate network outputs corresponding to the network inputs, the network outputs including a score for each of a plurality of classes.

[0027] In some implementations, the neural network is configured to process network inputs including image or audio signals.

[0028] In some implementations, the image is a medical image of the patient.

[0029] In some implementations, each class corresponds to a specific medical condition, and the corresponding score for each class represents the predictability that a patient has the corresponding medical condition.

[0030] According to another aspect, a system is provided, comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding methods described herein.

[0031] According to another aspect, one or more non-transitory computer storage media are provided that store instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding methods described herein.

[0032] Specific embodiments of the subject matter described in this specification may be implemented in order to achieve one or more of the following advantages.

[0033] The system described in this specification trains a neural network to: (i) generate accurate network outputs for machine learning tasks, and (ii) generate intermediate outputs that can be used to reliably classify out-of-distribution network inputs. This system does not require access to out-of-distribution labeled network inputs, which may be difficult to collect in practice. Instead, the system can train the neural network to generate intermediate outputs that can be used to reliably classify out-of-distribution network inputs based solely on the training network inputs in the distribution. Therefore, compared to some conventional systems that require access to out-of-distribution labeled network inputs, this system can consume fewer computational resources (e.g., memory and computing power), for example, by avoiding the need to store out-of-distribution network inputs and train the neural network using them.

[0034] For out-of-distribution network inputs, neural networks can have an increased likelihood of generating inaccurate network outputs. The system described in this specification enables reliable classification of out-of-distribution network inputs, which can then be processed separately, for example, by human experts, or prevented from downstream processing. Therefore, this system allows neural networks to be deployed in a wider range of applications, particularly safety-critical applications, compared to neural networks trained by conventional systems. Examples of safety-critical applications may include using neural networks to predict medical diagnoses or to control robots or (semi-)autonomous vehicles.

[0035] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Further features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0036] Figure 1 This is a diagram illustrating an example data stream of operations performed by the training system.

[0037] Figure 2 This is a block diagram of an example training system.

[0038] Figure 3 This is a flowchart illustrating an example process for training a neural network to generate network outputs for machine learning tasks and intermediate outputs for classifying out-of-distribution network inputs.

[0039] Figure 4 This is a flowchart of an example process for generating contrastive loss gradients.

[0040] Figure 5 This is a flowchart of an example process for generating network outputs and determining whether the corresponding network inputs are out-of-distribution inputs.

[0041] Similar reference numerals and naming conventions are used in the various figures. Detailed Implementation

[0042] Figure 1 A diagram illustrating an example data stream 100 performing operations by a training system is shown. An example of a training system capable of performing the operations of data stream 100 (e.g., training system 200) is referenced below. Figure 2 The description can be described in more detail.

[0043] The training system trains a neural network to process network inputs to generate: (i) a predicted network output that defines a characterization of the network input, and (ii) an intermediate output that can be used to determine whether the network input is an “out-of-distribution” network input (e.g., a “network input embedding”).

[0044] Neural networks can be configured to process any suitable type of network input, such as image data, audio data, text data, LiDAR data, hyperspectral data, or combinations thereof. (Throughout this specification, processing an image using a neural network, for example, means processing the intensity values ​​associated with the pixels of the image, such as color intensity values.)

[0045] Neural networks can be configured to generate any appropriate type of network output, such as classification output, regression output, or a combination of both.

[0046] The classification output may include a score for each class in a set of possible classes (e.g., a set of medical conditions), where the score for each class may define the likelihood that the network input is included in that class (e.g., a medical image of a patient represented by the network input corresponds to a medical condition in the set of medical conditions).

[0047] The regression output can include one or more numerical values ​​from a continuous range of possible values. For example, the network input could be a video showing the heart contracting, and the network output could define a prediction of the fraction of blood pumped out by the left ventricle of the heart during contraction.

[0048] The neural network trained by the system described in this specification can be deployed in any of a variety of applications, such as to control robotic agents or (semi-)autonomous (land, sea, or air) vehicles, or to predict medical diagnoses.

[0049] In some implementations, the system can train a neural network to perform medical data processing tasks, for example, by processing network inputs representing medical data (e.g., medical images, laboratory data (e.g., levels of various enzymes in a patient) or signal data (e.g., electrocardiogram (EKG) data)) to generate network outputs that define predictions characterizing the medical data (e.g., a patient's medical diagnosis).

[0050] Neural networks can be configured to process any appropriate modality of medical images, such as computed tomography (CT) images, magnetic resonance imaging (MRI) images, ultrasound (US) images, pathological images, or combinations thereof.

[0051] Medical images can depict any tissue area in a patient’s various tissue regions, such as the brain, liver, breast, prostate, intestines, or the whole body.

[0052] Neural networks can process medical images representing a patient's tissues to generate any appropriate network output, such as a classification output including one or more of the following: (i) a referral score for each of a plurality of referral decisions, representing the predictive probability that the referral decision is the most appropriate referral decision for the patient in the current state of a given tissue; (ii) a status score for each of a plurality of medical conditions, representing the predictive probability that the patient has that medical condition in the current state of a given tissue; (iii) a progression score for each of a plurality of condition states, representing the predictive probability that the state of the corresponding medical condition will progress to that condition state at a specific future time in the current state of a given tissue; or (iv) a treatment score for each of a plurality of treatments, representing the predictive probability that the treatment is the best treatment for the patient in the current state of a given tissue.

[0053] In some implementations, the neural network can process mammograms (i.e., X-ray images of the breast) to generate a network output that defines whether the breast is cancerous and the likelihood that the breast will become cancerous within a predetermined time period (e.g., one (1) year).

[0054] In some implementations, the neural network can process, for example, dermatological images showing areas of a patient's skin to generate a network output that defines whether the dermatological image shows cancerous areas of the skin or the likelihood that an area of ​​the skin will become cancerous within a predetermined duration (e.g., one (1) year).

[0055] In some implementations, the training system may train a neural network to process a set of network inputs representing pixels of an image to generate a classification output that includes a score for each object category from a set of possible object categories (e.g., vehicles, pedestrians, cyclists, etc.). The score for an object category can define the likelihood that the image depicts an object belonging to that object category.

[0056] In some implementations, the training system can train a neural network to process a set of network inputs representing audio samples in an audio waveform to perform speech recognition, i.e., to generate an output that defines a sequence of words or characters, words, or phonemes corresponding to the audio waveform.

[0057] In some implementations, the training system may train a neural network to process a set of network inputs representing words in a word sequence to perform natural language processing tasks, such as topic classification or summarization. To perform topic classification, the training system trains the neural network to generate a network output that includes a score for each topic category from a set of possible categories (e.g., sports, business, science, etc.). The topic category score defines the likelihood that a sequence of words belongs to that topic category. To perform summarization, the training system trains the neural network to generate a network output that includes an output word sequence with a shorter length than the input word sequence and that captures important or relevant information from the input word sequence.

[0058] In some implementations, the training system can train a neural network for a neural machine translation task, for example, to process a set of network inputs representing a sequence of texts in one language (e.g., word pieces or sequences of characters, phrases, or words) to generate a network output that can be a translation of the text sequence into another language (i.e., a sequence of texts in another language that is the translation of the input text sequence).

[0059] In some implementations, the training system can train a neural network to perform audio processing tasks. For example, if the network input represents spoken utterance, the output generated by the neural network could be a score for each text segment in a set of text fragments, each score representing an estimated probability that the text segment is a correct transcription of the utterance. As another example, if the network input represents spoken utterance, the output generated by the neural network could indicate whether a particular word or phrase (“hotword”) was spoken in the utterance. As yet another example, if the network input represents spoken utterance, the output generated by the neural network could identify the natural language used to speak the utterance.

[0060] In some implementations, the training system can train the neural network to perform natural language processing or understanding tasks that operate on a set of network inputs representing text in a natural language, such as entailment tasks, paraphrasing tasks, text similarity tasks, emotion tasks, sentence completion tasks, grammar tasks, etc.

[0061] In some implementations, the training system can train a neural network to perform a text-to-speech task, wherein the network input represents text in natural language or features of text in natural language, and the network output is a spectrogram, waveform, or other data that defines the audio of the text spoken in natural language.

[0062] In some implementations, the training system can train a neural network to perform a health prediction task, wherein the network input represents data derived from the patient's electronic health record data, and the output is a prediction related to the patient's future health, such as a predictive treatment that should be given to the patient, the likelihood of the patient experiencing an adverse health event, or a predictive diagnosis for the patient.

[0063] In some implementations, the training system can train a neural network to perform a text generation task, wherein the network input represents a sequence of text and the output is another sequence of text, such as a completion of the input text sequence, a response to a question posed in the input sequence, or a sequence of text about a topic specified by the first text sequence. As another example, the network input can represent data other than text, such as an image, and the output sequence can be text describing the data represented by the network input.

[0064] In some implementations, the training system can train a neural network to perform protein modeling tasks, where the network input represents a protein and the network output characterizes the protein. For example, the network output could characterize the predicted stability or predicted structure of a protein.

[0065] In some implementations, the training system can train a neural network to perform point cloud processing tasks, where the network input represents a point cloud (e.g., generated by a lidar or radar sensor), and the network output characterizes, for example, the type of object represented by the point cloud.

[0066] In some implementations, the training system can train a neural network to perform a combination of multiple individual machine learning tasks (e.g., the two or more machine learning tasks mentioned above).

[0067] Typically, a training system can train a neural network by adjusting the current values ​​of the neural network parameters in each of a series of training iterations. Figure 1The example data stream shown illustrates a single training iteration performed by the training system. The training system can initialize the values ​​of the neural network parameters before training begins, for example, by randomly sampling each parameter value from a uniform distribution within a range of values.

[0068] Neural networks may include: encoder neural network 108(f) θ Predictive Neural Network 110 and projection neural network 112(h ν In some cases, the projected neural network 112 is included in the neural network only during training (i.e., to facilitate training, as will be described in more detail below) and can be removed from the neural network after training is complete.

[0069] In each training iteration, the training system can obtain a first training network input and a second training network input. The training system can obtain the training network input, for example, by randomly sampling each training network input from the set of training network inputs, to obtain a representative sample of the set of training network inputs over multiple training iterations. The first and second network training inputs can correspond to the same training network input or two different training network inputs. (See reference...) Figure 1 From the example data stream, the system can obtain the first training network input x. i The second training network input x j , where i and j are the indices of the network input for training. Figure 1 In a specific example of the data flow, i is never equal to j, that is, x i and x j Corresponding to different inputs to the training network.

[0070] In each training iteration, the training system can obtain transformations from the set of transformations. The training system can obtain transformations, for example, by randomly sampling from the set of transformations to obtain representative samples from the set of transformations across multiple training iterations. Typically, the obtained transformations are distinctive; that is, they are different from each other. Each transformation can represent an intervention in the "style" of the input to the training network while leaving the "content" (i.e., the underlying semantic information) invariant. (See also...) Figure 1 For example data streams, the system can obtain three transformations from the set of transformations T, specifically transformation 102, T. (1) Transformation 104, T (2) , and transformation 106, T (3) .

[0071] In each training iteration, the training system can apply transformations to the acquired training network inputs to generate new network inputs (new "versions" of the training network inputs) from each input. The training system can apply differentiated transformations to the same training network inputs to generate differentiated versions of the same input, or it can apply transformations to different training network inputs to generate versions of network inputs generated from the different inputs. (See also...) Figure 1 Example data stream, the training system can perform the first transformation 102, T (1) It is applied to train the network input xi to generate the corresponding first network input x. i 1 Transform the second transformation 104, T (2) Applied to the same training network input x i To generate the corresponding second network input x i 2 And the third transformation 106, T (3) Applied to different training network inputs x j To generate the corresponding third network input x j 3 .

[0072] First network input x i 1 Second network input x i 2 Corresponding to the same training network input x i And the third network input x j 3 Corresponding to different training network inputs x j .

[0073] In each training iteration, the training system can generate embeddings for each network input as intermediate outputs of the neural network. For example, the training system can use an encoder neural network 108 to generate embeddings for each network input. The training system can generate corresponding network input embeddings by processing the respective network inputs using the encoder neural network 108, while generating the first network input x. i 1 The corresponding first network input embedding z i 1 Second network input x i 2 The corresponding second network input embedding z i 2 and the third network input x j 3 The corresponding third network input embedding z j 3.

[0074] In each training iteration, the training system can generate a network output corresponding to the first network input embedding. For example, the training system can use a predictive neural network to generate a network output corresponding to the first network input embedding. (See reference...) Figure 1 For example data streams, the training system can use a predictive neural network 110. To process the first network input embedding z i 1 This is used to generate network output. The network output can be, for example, a classification output or a regression output that characterizes the network input.

[0075] In each training iteration, the training system can generate a projected embedding for each network input embedding. For example, the training system can generate a projected embedding for each network input embedding by processing the corresponding network input embedding using a projected neural network. (See also...) Figure 1 Example data stream, the training system uses a projective neural network 112,h ν Generate a projected embedding for each network input embedding.

[0076] In each training iteration, the training system can generate contrastive loss gradients and supervised loss gradients to adjust the current values ​​of the neural network parameters. For example, the system can generate an "attractive" gradient for the contrastive loss function by processing projected embeddings generated from the same training network inputs (e.g., minimizing the differences among network input embeddings generated from the same training network inputs, thus "attracting" the embeddings), and a "repel" gradient for the contrastive loss function by processing projected embeddings generated from different training network inputs (e.g., maximizing the differences among network input embeddings generated from different training network inputs, thus "repelling" the embeddings). (See reference...) Figure 1 The example data stream shows how the system processes projected embeddings generated from different training network inputs (specifically, by processing the network input embedding z). i 2 and z j 3 The "repulsive" gradient is generated by processing the projected embeddings (specifically, by processing the network input embedding z) generated from the same training network input. i 1 and z i 2 The projection embedding is used to generate the "attractive" gradient 116, and the embedding from the first network input z is processed. i 1 The generated network output is used to generate the supervised loss gradient 114.

[0077] In each training iteration, the training system adjusts the current values ​​of the network parameters by backpropagating gradients. For example, the training system can adjust the current values ​​of the prediction neural network parameters based on the supervised loss gradient, the projection neural network parameters based on the contrastive loss gradient, and the encoder neural network parameters based on both the supervised loss and contrastive loss gradients. (See also...) Figure 1 The example data stream allows the training system to adjust the prediction neural network 110 based on the supervised loss gradient 114. The current parameter values ​​are used to adjust the projection neural network 112 based on the contrastive loss gradients 116 and 118. ν The current parameter values, and the encoder neural network 108 adjusted based on both the contrastive loss gradients 116 and 118 and the supervised loss gradient 114. θ The current parameter value.

[0078] Optimizing supervised loss and contrastive loss can encourage neural networks to generate accurate network outputs and more semantically meaningful intermediate network outputs (“network input embeddings”). Specifically, optimizing supervised loss can encourage neural networks to generate accurate predictions for machine learning tasks. Optimizing contrastive loss can encourage neural networks to generate embeddings that are more similar (more “clustered” or “attractive”) to network inputs generated from the same training network inputs, and more different (more “dissimilar” or “repulsive”) to network inputs generated from different training network inputs.

[0079] The embeddings of network inputs, which are generated as intermediate outputs of a neural network by processing network inputs, can be further processed to classify the network inputs as out-of-distribution or in-distribution inputs, as will be discussed below. Figure 5 The description will be discussed in further detail. Classifying network inputs as out-of-distribution inputs can enable network inputs to be labeled for further examination (e.g., manual examination by experts (e.g., doctors), to block downstream processing, or both.

[0080] Figure 2 An example training system 200 is shown. The training system 200 is an example of a system implemented as a computer program on one or more computers in one or more locations, wherein the system implements the following systems, components and technologies.

[0081] The training system 200 can train the neural network (e.g., neural network 210) to perform machine learning tasks, in particular by iteratively adjusting the values ​​of the neural network parameters 224, to optimize the objective function in each training iteration of a series of training iterations, as discussed in further detail below.

[0082] Neural network 210 may have any suitable neural network architecture that enables it to perform the functions described herein, namely, processing network inputs to generate: (i) a network output that defines a prediction representing the network inputs, and (ii) an embedding of the network inputs. In particular, neural network 210 may include any suitable number (e.g., 5, 10, or 25 layers) and any suitable type of neural network layers (e.g., fully connected layers, attention layers, convolutional layers, etc.) connected in any suitable configuration (e.g., as a sequence of linear layers).

[0083] In each training iteration, the training system 200 may obtain multiple training network inputs from the set of training network inputs 202, including a first training network input and a second training network input. The first training network input and the second network training input may be the same training network input, or they may be different training network inputs. The set of training network inputs 202 may include multiple training network inputs. The training system may obtain training network inputs, for example, by randomly sampling from the set of training network inputs 202 in each training iteration, to achieve representative sampling of the set of training network inputs across multiple iterations.

[0084] In each training iteration, the training system 200 can obtain multiple transformations from a set 204 of transformations, including a first transformation and a second transformation. Typically, the transformations are distinct. The training system can obtain transformations, for example, by randomly sampling from the set 204 of transformations in each training iteration, to generate representative samples of the set 204 of transformations over multiple training iterations. For example, the set of possible transformations representing a network input, such as an image, can include, for example, rotation (e.g., to various possible rotation angles), color shading (e.g., to various different colors), grayscale scaling, expansion and cropping (using various expansion and cropping parameters), shrinking and padding (using various shrinking and padding parameters), pixel-wise noise (using noise intensity levels), or any combination thereof.

[0085] In each training iteration, the training system 200 can obtain multiple network inputs (e.g., network input 208) including a first network input and a second network input to train the neural network. For example, the transformation engine 206 can generate each corresponding network input by applying a corresponding transformation to the corresponding training network input. In one example, the transformation engine 206 can generate a corresponding first network input by applying a first transformation to the first training network input to generate a corresponding first network input, and by applying a second transformation to the second training network input to generate a corresponding second network input. The first network input and the second network input can correspond to the same training network input, wherein the transformation engine 206 applies distinct transformations to the same training network input, or the first network input and the second network input can correspond to two different training network inputs, wherein the transformation engine 206 applies transformations to two different training network inputs.

[0086] In each iteration, the training system 200 can generate network input embeddings 212 by processing network inputs 208 using neural network 210. For example, neural network 210 can generate each network input embedding as an intermediate output of one or more hidden layers of the neural network by processing the corresponding network inputs. In one example, neural network 210 can process a first network input to generate a corresponding first network input embedding, and can process a second network input to generate a corresponding second network input embedding.

[0087] In each iteration, the training system 200 can generate a network output 216 by processing the first network input using a neural network 210. For example, the neural network 210 can process the first network input 208 to generate the network output 216, as shown below. Figure 3 The description will be discussed in further detail.

[0088] In each iteration, the training system 200 can generate a supervised loss gradient by processing the network output 216 using the supervised loss engine 218. For example, the supervised loss engine 218 can generate a supervised loss gradient of a supervised loss, whereby the supervised loss measures, for example, the error between: (1) the network output generated by the neural network through processing the network input and (2) the target output that should be generated by the neural network through processing the network input. The supervised loss gradient can be joined with the contrastive loss gradient generated by the contrastive loss system 214 in a linear combination to form a loss gradient 220, as shown below. Figure 3 The description will be discussed in further detail.

[0089] In each iteration, the training system 200 can process the network input embeddings 212 to generate a contrastive loss gradient using the contrastive loss system 214. The contrastive loss encourages network input embeddings generated from transformed versions of the same training network input to be more similar, and encourages network input embeddings generated from different training network inputs (i.e., not transformed versions of the same training network input) to be more different. The contrastive loss gradient can be combined with the supervised loss gradient in a linear combination to form the loss gradient 220. See below for reference. Figure 4 A more detailed example process for generating the contrastive loss gradient is described below.

[0090] In each training iteration, the training system 200 can adjust the network parameters 224 by processing the loss gradient 220 using the update engine 222. For example, the update engine 224 can backpropagate the loss gradient 220 through the neural network to adjust the current values ​​of the neural network parameters 224, as shown below. Figure 3 The description will be discussed in further detail.

[0091] After training, the neural network 210 can be deployed to generate predictions in a variety of applications, such as generating medical diagnoses. In addition to generating predictions, the neural network 210 can generate a network input embedding for each network input, which can be used to determine whether the network input is an out-of-distribution input. In response to determining that the network input is an out-of-distribution input, the predictions generated for the network input can be flagged for further inspection or to block downstream processing, as described in [reference]. Figure 5 The description will be discussed in further detail.

[0092] Figure 3 This is a flowchart illustrating an example process for training a neural network to generate predicted outputs for a machine learning task and intermediate outputs for classifying network inputs as out-of-distribution inputs. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system appropriately programmed according to this specification (e.g., Figure 2 The training system 200 can execute process 300.

[0093] Neural networks can include encoder neural networks, predictive neural networks, and projective neural networks. Encoder neural networks process network inputs to generate embeddings of those inputs as intermediate outputs of the neural network, while predictive neural networks process these embeddings to generate network outputs that represent the network inputs. Projective neural networks can be used during training to facilitate the training of encoder neural networks to generate more semantically meaningful network input embeddings, as shown in the following reference. Figure 4 The description will be discussed in further detail.

[0094] The steps of process 300 are performed in each training iteration in the sequence of training iterations. For convenience, the steps of process 300 are described with reference to the current training iteration.

[0095] The training system obtains multiple network inputs (302), including a first network input and a second network input. The network inputs may correspond to the same training network input or different training network inputs. For example, if two network inputs correspond to the same training network input, the training system can randomly sample one training network input from the set of training network inputs and randomly sample two different transformations from the set of transformations. The training system can then apply the first transformation to the same training network input to generate a corresponding first network input, and apply the second transformation to the same training network input to generate a corresponding second network input, denoted as...

[0096]

[0097] Where x i k This represents the result of applying transformation k to the training network input i, where i indexes the network input, k indexes the transformation, and T... (k) This represents the transformation k.

[0098] As will be described in more detail below, by applying differentiated transformations to the same training network input to generate differentiated network inputs from the same training network input, a training system can be enabled to train neural networks to generate more similar network input embeddings corresponding to the same training network input (i.e., more tightly clustered in the embedding space, or “attracting” network input embeddings).

[0099] In another example, if the two network inputs correspond to different training network inputs, the training system can randomly sample two different training network inputs from the set of training network inputs, and can randomly sample two transformations from the set of transformations. The training system can then apply a first transformation to the first training network input to generate a corresponding first network input, and can apply a second transformation to the second training network input to generate a corresponding second network input.

[0100] The set of possible transformations used to represent network inputs, such as images, may include, for example, rotation (e.g., to various possible rotation angles), color shading (e.g., to various different colors), grayscale scaling, expansion and cropping (using various expansion and cropping parameters), shrinking and padding (using various shrinking and padding parameters), pixel-by-pixel noise (using noise intensity levels), or any combination thereof.

[0101] The training system uses an encoder neural network to generate the embedding for each network input (304). For example, the system can generate the corresponding network input embedding by processing the network input using an encoder neural network. Processing the network input using an encoder neural network to generate the network input embedding can be represented as:

[0102]

[0103] Among them, z i k Let f be the network input embedding of the transformation k of network input i, i-indexed network input, k-indexed transformation, and f. θ This represents the encoder neural network.

[0104] The encoder neural network can have any suitable neural network architecture that enables it to perform the function it describes, namely, to process network inputs to generate embeddings of those inputs. In particular, the encoder neural network can include any suitable number (e.g., 1 layer, 5 layers, or 25 layers) and any suitable type of neural network layers (e.g., fully connected layers, attention layers, convolutional layers, etc.) connected in any suitable configuration (e.g., as a sequence of linear layers).

[0105] The training system can generate the network output from the embedding of the first network input (306). For example, the training system can generate the network output by processing the first network input embedding using a predictive neural network. The network output can define a prediction that characterizes the network input, such as a classification or regression prediction.

[0106] A predictive neural network can have any suitable neural network structure that enables it to perform the function it describes, i.e., to process network input embeddings to generate corresponding network outputs. In particular, each predictive neural network may include any suitable number (e.g., 1 layer, 5 layers, or 25 layers) and any suitable type of neural network layers (e.g., fully connected layers, attention layers, convolutional layers, etc.) connected in any suitable configuration (e.g., as a sequence of linear layers).

[0107] The training system can generate a supervised loss gradient based on the network output (308). For example, the training system can generate a supervised loss gradient of the supervised loss function based on the error between the network output corresponding to the first network input and the target network output corresponding to the first network input. The supervised loss function can be, for example, softmax cross-entropy loss, squared error loss, or any other suitable loss, and may also include, for example, label smoothing.

[0108] The training system can generate contrastive loss gradients by processing network input embeddings (310). Training a neural network using a contrastive loss function allows the network to generate more distinctive embeddings of network inputs corresponding to different training network inputs, as well as more similar embeddings of network inputs corresponding to the same training network inputs. Generating more semantically meaningful embeddings of network inputs can be useful for detecting out-of-distribution inputs, which could enable the neural network to be deployed more broadly in safety-critical tasks, such as predictive diagnostics based on medical data. (See also...) Figure 4 An example process for generating the gradient of the contrastive loss is described in more detail.

[0109] The training system can adjust the current values ​​of the network parameters by processing the gradients of the supervised loss and the contrastive loss (312). In some implementations, the training system can be trained in two phases. In the first phase, the training system can adjust the current values ​​of the network parameters (e.g., for projection neural networks and encoder neural networks) based solely on the gradient of the contrastive loss (while suppressing the gradients that generate the supervised loss). In the second phase, the training system may also include a supervised loss. In the second phase, the training system can combine the gradients of the supervised loss and the contrastive loss in a linear combination. In either phase, the training system can adjust the current values ​​of the network parameters by backpropagating the specified gradients.

[0110] The training system can determine whether the termination criterion (314) has been met. If the training system determines that the termination criterion has not been met, the training system loops back to step (302). For example, the training system can determine whether a predefined number of training iterations have been performed.

[0111] If the training system determines that the termination criteria have been met, the training system terminates the training (316).

[0112] Figure 4 This is a flowchart of an example process for generating contrastive loss gradients. For convenience, process 400 will be described as being executed by a system of one or more computers located in one or more locations. For example, a system appropriately programmed according to this specification... Figure 2 The contrast loss system 220 can execute process 400.

[0113] The system obtains multiple network input embeddings (402), including the first network embedding and the second network embedding. The network input embeddings may correspond to the same training network input or to different network inputs, as shown above. Figure 2 and Figure 3 Further detailed discussion is needed.

[0114] The system uses a projective neural network to generate projected embeddings (404) for each network input embedding. The projective neural network can be the entire trained neural network (e.g., in...). Figure 3 This is part of the neural network mentioned in the description. The corresponding projective embedding of the network input embedding can have a lower dimension than the network input embedding. For example, the projective embedding of a first network embedding can have a lower dimension than the first network input embedding, and the projective embedding of a second network input embedding can have a lower dimension than the second network embedding. A projective neural network can generate the corresponding projective embedding of the network input embedding by processing the corresponding network input embedding, expressed as:

[0115]

[0116] Where, π i k Indicates network input embedding z i k The projection embedding, i-indexed network input, k-indexed transform, and h ν This represents a projected neural network.

[0117] The encoder neural network can have any suitable neural network architecture that enables it to perform the function it describes, namely, to process the network input embeddings to generate the corresponding projected embeddings. In particular, the projected neural network can include any suitable number (e.g., 1 layer, 5 layers, or 25 layers) and any suitable type of neural network layers (e.g., fully connected layers, attention layers, convolutional layers, etc.) connected in any suitable configuration (e.g., as a sequence of linear layers).

[0118] The system can use similarity operations to generate similarity measurements between projected embeddings (406). For example, the similarity measurement could be a cosine similarity measurement, denoted as,

[0119]

[0120] in, Indicates projection embedding and projection embedding The cosine similarity between them is measured, with i and j indexed as network inputs, and k and l as indexed transforms, (.). T represents the transpose operation of projection embedding, and ‖.‖ represents the magnitude operation of projection embedding.

[0121] The system can generate the gradient of the contrastive loss function based on similarity measurements (408). For example, the contrastive loss function can attempt to maximize the cosine similarity of projected embedding pairs originating from the same network input, i.e., encourage their cosine similarity measurement to be one, while minimizing the cosine similarity of pairs originating from two different network inputs, i.e., encourage their cosine similarity measurement to be zero. The contrastive loss function for network input i can be expressed as follows:

[0122]

[0123] in, Let represent the similarity measurement between the projected embedding generated from the first version a of the training network input i and the projected embedding generated from the second version 1-a of the training network j, where i and j index the network inputs, a indexes the corresponding first transformation, 1-a indexes the corresponding second transformation, and τ represents the temperature parameter. The contrastive loss of the training network input i in equation (5) includes a numerator, which represents the similarity measurement between the corresponding first version and the corresponding second version generated from the same training network input i. The contrastive loss of the training network input i also includes a denominator, which represents (1) the sum of the similarity measurements between the corresponding first version of the training network input i and the corresponding second version of each network input (including the training network input i), and (2) the sum of the similarity measurements between the corresponding first version of the training network input i and the corresponding first version of each network input (excluding the training network input i).

[0124] Training a neural network using a contrastive loss function enables the network to generate embeddings of more distinct network inputs corresponding to different training network inputs (e.g., based on the denominator in Equation (5)) and more similar network inputs corresponding to the same training network inputs (e.g., using the numerator in Equation (5)). Generating embeddings of more semantically meaningful network inputs can be useful for detecting out-of-distribution inputs, which could allow neural networks to be deployed more broadly in safety-critical tasks, such as predictive diagnostics based on medical data.

[0125] Figure 5 This is a flowchart of an example process for generating network outputs for test network inputs and determining whether the test network inputs are out-of-distribution inputs. For convenience, process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, this includes a suitable training system (e.g., [system name missing]) that is appropriately programmed according to this specification. Figure 2 The neural network trained by the training system 200 (e.g., Figure 2 The out-of-distribution detection system of the neural network 210 can perform process 500.

[0126] The system generates a spatial probability distribution of possible network input embeddings (502). To generate the probability distribution, the system can process each of the multiple (distributed) network inputs using a neural network to generate a corresponding network input embedding for each network input, i.e., as an intermediate output of the neural network, as described in more detail above. The system can then generate the probability distribution by fitting the parameters of the probability distribution to the set of network input embeddings (e.g., using the method of moments or any other suitable technique). In one example, each network input is associated with a corresponding class c in a set of predetermined classes, and the system can generate the probability distribution by applying an n-dimensional multivariate Gaussian distribution for each class c. A probability distribution is generated by fitting the set of network input embeddings corresponding to class c (where n equals the dimension of each network input embedding). In this example, the set of probability distributions corresponding to the respective classes collectively defines the probability distribution in the space of possible network input embeddings.

[0127] The system receives test network input (504). Test network input can be provided to the system, for example, by a user of the system.

[0128] The system generates a network input embedding for the test network input (506). For example, the system can use a neural network to process the test network input to generate a network input embedding for the test network input as an intermediate output of the neural network, as described in more detail above.

[0129] The system generates a network output (508) corresponding to the test network input. The network output can define a prediction (e.g., a classification or regression prediction) that characterizes the network input and can be generated by the output layer of the neural network, as described in more detail above.

[0130] The system generates the probability distribution of the network input embedding of the test network input under the space of possible network input embeddings (510). For example, if each network input belongs to a corresponding class c, the system can generate the corresponding probability λ of the test network input embedding relative to each class c. c (x), for example:

[0131]

[0132] Among them, f θ μ represents the input embedding of the test network. c and Σ c For a multivariate Gaussian probability distribution corresponding to class c (as described in step 502), det represents the determinant of the matrix, (.). -1 This represents the inverse of a matrix, and (.). T This represents the transpose of a vector.

[0133] The system determines whether a test network input is out-of-distribution based on the probability distribution of the test network input in the space of possible network input embeddings (512). For example, the system can determine whether a test network input x is out-of-distribution based on the probability λ of the test network input x relative to each class c. c (x) Generate scores to determine whether the test network input is out of distribution, for example:

[0134] s(x) = max c (λ c (x)), (7)

[0135] Where s(x) represents the out-of-distribution score of the test network input x, and max is the maximization function of the probability of the test network input with respect to each class c. A high score s(x) indicates that the embedding of the test network input is close to the set of typical network input embeddings of at least one class c. A low score s(x) indicates that the embedding of the test network input is far from all training set examples, i.e., the test network input may be an out-of-distribution input. If the probability of the test network input embedding under the probability distribution in the space of possible network input embeddings (e.g., as characterized by s(x) in equation (7)) exceeds a predefined threshold (e.g., 0.5, or any other appropriate threshold), the system can determine that the test network input is a network input in the distribution. Otherwise, the system can determine that the test network input is an out-of-distribution network input.

[0136] Determining that the test network input is an off-distribution input can enable the test network input to be flagged for further inspection, blocked from subsequent downstream processing, or both.

[0137] In some implementations, the system first determines whether the test network input is an out-of-distribution input before generating a network output corresponding to the test network input. In these implementations, in response to determining that the test network input is an out-of-distribution input, the system may suppress the generation of a network output for the test network input.

[0138] This specification uses the term "configured" in conjunction with system and computer program components. For one or more computers, being configured to perform a specific operation or action means that the system has software, firmware, hardware, or a combination thereof installed thereon, which, in operation, cause the system to perform the operation or action. For one or more computer programs, being configured to perform a specific operation or action means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform an operation or action.

[0139] Embodiments and functional operations of the subject matter described herein may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware (including the structures disclosed herein and their structural equivalents), or in a combination of one or more of these. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. Alternatively or additionally, the program instructions may be encoded on artificially generated propagated signals (e.g., machine-generated electrical, optical, or electromagnetic signals) that are generated as coded information to be transmitted to a suitable receiver device for operation by the data processing apparatus.

[0140] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The device may also be or include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the device may optionally include code that creates the runtime environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0141] A computer program (which may also be referred to or described as a program, software, software application, app, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file storing portions of one or more modules, subroutines, or code). A computer program can be deployed to execute on one or more computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0142] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same computer or multiple computers.

[0143] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGA or ASIC), or by a combination of special-purpose logic circuitry and one or more programmable computers.

[0144] A computer suitable for running computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into special-purpose logic circuitry. Generally, a computer will also include one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data or operatively coupled to one or more mass storage devices to receive data from or transfer data to one or more mass storage devices, or both. However, a computer does not necessarily have to have such devices. Additionally, a computer can be embedded in another device (e.g., to name just a few, mobile phones, personal digital assistants (PDAs), mobile audio or video players, game consoles, GPS receivers, or portable storage devices (e.g., Universal Serial Bus (USB) flash drives)).

[0145] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0146] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) for displaying information to the user, through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Additionally, the computer can interact with the user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in response.

[0147] The data processing apparatus for implementing machine learning models may also include, for example, dedicated hardware accelerator units for processing the ordinary and computationally intensive parts (i.e., inference, workload) of machine learning training or production.

[0148] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0149] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), middleware components (e.g., an application server), or front-end components (e.g., a client computer with a graphical user interface, web browser, or app through which a user can interact with embodiments of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0150] A computing system may include clients and servers. Clients and servers are generally geographically distant and typically interact via a communication network. The client-server relationship is formed by computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device acting as a client (e.g., for the purpose of displaying data to a user interacting with the device and receiving user input from the user interacting with the device). Data generated at the user device (e.g., the result of user interaction) can be received from that device at the server.

[0151] Although this specification contains numerous specific details of implementation, these details should not be construed as limiting the scope of any invention or the scope of possible claims, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases, one or more features from a claimed combination may be removed from the combination, and the claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0152] Similarly, although operations are depicted in a specific order in the accompanying drawings and recited in a specific order in the claims, this should not be construed as requiring the operations to be performed in the specific order shown or in sequential order, or requiring the performance of all shown operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments; rather, it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0153] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method performed by one or more data processing devices for training a neural network having multiple network parameters to (i) generate accurate network outputs for a machine learning task, and (ii) generate intermediate outputs that can be used to reliably classify out-of-distribution network inputs, the method comprising: Classifying network inputs generated by neural networks into out-of-distribution network inputs includes: Obtain the test network input; The test network input is processed using a neural network to generate: (i) an embedding of the test network input as an intermediate output of the trained neural network, and (ii) a predicted output for the test network input; and Based on the probability distribution of the embedding of the test network input in the space of possible embeddings, classify the test network input as out-of-distribution network input; and In response, the predicted output used to test the network input is prevented from undergoing downstream processing; The neural network has been trained by repeatedly performing the following operations: Obtain the first network input and the second network input; The first network input is processed using a neural network to generate: (i) an embedding of the first network input as an intermediate output of the neural network, and (ii) a network output corresponding to the first network input; The second network input is processed using a neural network to generate at least one embedding of the second network input as an intermediate output of the neural network; and The current values ​​of the network parameters are adjusted using the gradients of the supervised loss and the contrastive loss, where: The supervision loss is based on (i) the error between the network output corresponding to the first network output and (ii) the target network output corresponding to the first network input; The contrastive loss is based at least on: (i) the embedding of the first network input and (ii) the embedding of the second network input; and The neural network is configured to process network inputs, which include at least one of image data, audio data, text data, radar data, and hyperspectral data.

2. The method as described in claim 1, wherein, Adjusting the current values ​​of the network parameters using the gradient of the contrastive loss includes: The contrast loss is determined at least in part based on the similarity between the embeddings of the first network input and the embeddings of the second network input.

3. The method as described in claim 2, wherein, The first and second network inputs both correspond to the same training network inputs, and the current values ​​of the network parameters are adjusted using the gradient of the contrastive loss, including: The gradient is used to adjust the current values ​​of the network parameters to encourage the neural network to generate more similar embeddings of the first and second network inputs.

4. The method of claim 3, wherein, Obtaining the first network input and the second network input includes: A first transformation is sampled from the set of possible transformations and applied to the same training network input to generate a first network input; and A second transformation is sampled from the set of possible transformations and applied to the same training network input to generate a second network input.

5. The method of claim 2, wherein, The first and second network inputs do not correspond to the same training network inputs, and adjusting the current values ​​of the network parameters using the gradient of the contrastive loss includes: The gradient is used to adjust the current values ​​of the network parameters to encourage the neural network to generate corresponding embeddings of the first and second network inputs that are not very similar.

6. The method as described in any one of claims 2-5, wherein, Determining the contrastive loss based at least in part on the similarity between the embeddings of the first network input and the second network input includes: The embedding of the first network input is processed using a projection function to generate a projected embedding of the first network input. The embedding of the second network input is processed using a projection function to generate a projected embedding of the second network input; and The similarity between the embeddings of the first network input and the embeddings of the second network input is determined based on a similarity measurement between (i) the projected embedding of the first network input and (ii) the projected embedding of the second network input.

7. The method of claim 6, wherein, The similarity measurement is a cosine similarity measurement.

8. The method of claim 6, wherein, The projection function includes a projection neural network with multiple projection neural network parameters.

9. The method of claim 8, further comprising using the gradient of the contrastive loss to adjust the current values ​​of the projection neural network parameters.

10. The method of claim 6, wherein, The projection embedding of the first network input has a lower dimension than the first network input, and the projection embedding of the second network input has a lower dimension than the second network input.

11. The method of claim 1, wherein, The probability distribution of possible embeddings in the space of possible embeddings is determined based on the set of embeddings in the space of possible embeddings, wherein each embedding is generated by processing the network input in the distribution using a trained neural network to generate an embedding of the network input in the distribution as an intermediate output of the trained neural network.

12. The method of claim 1, wherein, The machine learning task includes a classification task, and the neural network is configured to process network inputs to generate network outputs corresponding to the network inputs, the network outputs including a score for each of a plurality of classes.

13. The method of claim 1, wherein, The image data refers to the patient's medical image data.

14. The method of claim 13, wherein, Each class corresponds to a specific medical condition, and the score for each class represents the predictability of a patient having that specific medical condition.

15. A data processing system, comprising: One or more computers; as well as One or more storage devices are communicatively coupled to one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of a corresponding method as described in any one of claims 1-14.

16. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of the corresponding method as described in any one of claims 1-14.

Images