Self-supervised contrastive learning with random feature corruption

Abstract

The present disclosure relates to self-supervised contrastive learning using random feature corruption, in particular to methods, systems for training a neural network having a plurality of network parameters and apparatuses comprising computer programs encoded on a computer storage medium. One of the methods comprises obtaining an unlabelled training input from a set of unlabelled training data; processing the unlabelled training input to generate a first embedding; generating a corrupted version of the unlabelled training input, comprising determining a proper subset of feature dimensions and, for each feature dimension in the proper subset of feature dimensions, applying corruption to a respective feature in that feature dimension using one or more feature values sampled from a marginal distribution of that feature dimension as specified in the set of unlabelled training data; processing the corrupted version of the unlabelled training input to generate a second embedding; and determining an update to current values of the plurality of network parameters.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Application No. 63 / 194,899, filed May 28, 2021. The disclosure of the earlier application is considered part of the disclosure of this application and is incorporated herein by reference. Technical Field

[0003] This manual relates to training neural networks. Background Technology

[0004] A neural network is a machine learning model that uses one or more layers of non-linear units to predict outputs from received inputs. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as input to the next layer in the network—the next hidden layer or output layer. Each layer of the network generates its output from the received inputs based on the current values ​​of its corresponding parameter set. Summary of the Invention

[0005] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which implements and trains a neural network capable of performing machine learning tasks on one or more received inputs. Specifically, the neural network is trained using a two-stage process: a pre-training stage and a fine-tuning stage. The pre-training stage of the neural network utilizes a self-supervised contrastive learning scheme.

[0006] The subject matter described herein can be implemented in specific embodiments to achieve one or more of the following advantages.

[0007] The system described in this specification pre-trains neural networks to generate task-agnostic representations that may later be useful in specific downstream tasks by processing network input pairs that do not require labeling. Specifically, network input pairs include unlabeled training inputs, such as sequences of images, videos, or text, and corrupted copies of the unlabeled training inputs automatically generated by the system by randomizing the feature values ​​of a random set of features from the unlabeled training inputs. Unlike existing self-supervised learning techniques that are typically highly specific to data from narrow technical fields such as computer vision or natural language processing, the edge-sampling corruption technique employed by the system is generally applicable to different formats or types of data across various technical fields, or both.

[0008] Furthermore, pre-trained neural networks can then be used to efficiently adapt to specific machine learning tasks using several orders of magnitude less data than those used for pre-training. For example, while a pre-trained network might utilize billions of unlabeled training inputs, adapting the network to a specific task might require only a few thousand labeled training inputs. Because training a network for a specific task requires significantly fewer, sometimes orders of magnitude, labeled training inputs than existing methods, the system can therefore utilize computational resources, such as memory, clock time, or both, more efficiently during fine-tuning. The system can also train neural networks with lower human labor costs associated with data labeling, while still ensuring competitive performance on a range of tasks that matches or even surpasses current-level technology, while additionally being generalizable and easily adaptable to new tasks.

[0009] Details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and description. Other features, aspects, and advantages of the subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0010] Figure 1A An example neural network system is shown during the pre-training phase.

[0011] Figure 1B An example neural network system is shown during the fine-tuning phase.

[0012] Figure 2 This is a flowchart illustrating an example process for pre-training a neural network using a self-supervised contrastive learning scheme.

[0013] Figure 3 This is a flowchart illustrating an example process for fine-tuning a neural network on a machine learning task.

[0014] Figure 4A -B are example illustrations of pre-trained and fine-tuned neural networks, respectively.

[0015] Similar reference numerals and names in the various figures indicate similar elements. Detailed Implementation

[0016] This specification describes a system implemented as a computer program on one or more computers at one or more locations, which implements and trains a neural network capable of performing a machine learning task on one or more received inputs. Depending on the task, the neural network can be configured to receive any kind of digital data input and process the received input according to the current parameter values ​​of the neural network to generate one or more outputs based on the input.

[0017] In some cases, neural network inputs include tabular data. Tabular data refers to numerical data or information arranged in rows and columns or as a matrix of cells. Tabular data refers to the arrangement of information, not a specific type of data found at a given location in a column, row, or cell. Tabular data also does not refer to the actual data that can be represented by tabular data. For example, each given location could have a numerical value representing a pixel (in the case of tabular data representing image data) or alternatively, numerical values ​​representing letters, words, phrases, or sentences (in the case of tabular data representing text data).

[0018] In some cases, the output of a neural network can include any kind of classification output. Classification can be, for example, type, class, group, category, or measure.

[0019] For example, a neural network can be configured to perform automatic pattern recognition tasks in a manufacturing plant setting. The neural network receives input data including multiple features describing a defect in a manufactured product—such as location, size, etc.—and processes the input data to generate a classification output that specifies the type of defect, such as scratches, stains, dirt, dents, etc. In this example, the input can be arranged in a table column data format with rows or columns corresponding to the multiple features describing the defect in the manufactured product; for example, it can have multiple columns, each containing a corresponding feature describing the defect.

[0020] As another example, neural networks can be configured to process input data describing the physical characteristics of plant leaf samples—such as shape, texture, edges, etc.—to generate a classification output for a specified plant species.

[0021] Examples of labeled datasets for this type of task and other similar classification tasks can be found in the University of California, Irvine Machine Learning Repository (UCI Repository) and OpenMedia Repository (OpenML).

[0022] In another example, the task could be a computer vision task, where the input is an image or point cloud and the output is a computer vision output of the image or point cloud. For example, a neural network can be configured to perform an image processing task, such as receiving input including image data comprising multiple pixels. The image data may, for example, include one or more images or features extracted from one or more images. The neural network can be configured to process the image data to generate an output for the image processing task.

[0023] For example, if the task is image classification, the output generated by the neural network for a given image can be a score for each object category in the set of object categories, where each score represents the estimated probability that the image contains an object belonging to that category.

[0024] As another example, if the task is object detection, the output generated by the neural network for a given image can be one or more bounded boxes associated with a corresponding score, where each bounded box represents an estimated location in the image and the corresponding score represents the estimated probability of an object at that location in the image—that is, within the bounded box.

[0025] As another example, if the task is semantic segmentation, the output generated by the neural network for a given image could be a label for each of a plurality of pixels in the image, where each pixel is labeled as belonging to one of a set of object categories. Alternatively, for each of the plurality of pixels, the output could be a set of scores including a corresponding score for each of the object categories in the set of object categories, which represents the probability that the pixel belongs to an object from that object category.

[0026] As another example, if the input to the neural network is an internet resource (e.g., a web page), a document, or a portion of a document, or features extracted from an internet resource, document, or portion of a document, then the output generated by the neural network for a given internet resource, document, or portion of a document can be a score for each topic in a set of topics, where each score represents an estimated probability that the internet resource, document, or portion of a document is relevant to that topic.

[0027] As another example, if the input to the neural network is features of the impression context of a particular ad, the output generated by the neural network could be a score representing the estimated probability that the particular ad will be clicked.

[0028] As another example, if the input to the neural network is features for personalized recommendations to a user, such as features characterizing the context of the recommendation, or features characterizing previous actions taken by the user, then the output generated by the neural network could be a score for each of the set of content items, where each score represents an estimated probability that the user will respond positively to the recommended content item.

[0029] As another example, a task can be a natural language processing or understanding task that operates on a sequence of texts in a natural language, such as entailment tasks, paraphrasing tasks, text similarity tasks, sentiment tasks, sentence completion tasks, grammatical tasks, etc.

[0030] For example, if the input to a neural network is a sequence of texts in one language, the output generated by the neural network could be a score for each text segment in a set of text segments in another language, where each score represents an estimated probability that the text segment in the other language is an appropriate translation of the input text into that other language.

[0031] As another example, if the input to the neural network is a sequence representing spoken utterances, the output generated by the neural network can be a score for each text segment in the set of text segments, with each score representing an estimated probability that the text segment is a correct transcription of the utterance.

[0032] As another example, the task could be a health prediction task, where the input is a sequence derived from a patient’s electronic health record data and the output is a prediction related to the patient’s future health, such as a predicted treatment that should be given to the patient, the likelihood of the patient experiencing an adverse health event, or a predicted diagnosis for the patient.

[0033] Figure 1A An example neural network system 100 is shown during the pre-training phase. The neural network system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, wherein the following systems, components and techniques are capable of being implemented.

[0034] The neural network system 100 includes a damage engine 120, a neural network 130, and a training engine 140. The neural network 130 is configured to receive input and generate output based on the received input and the values ​​of the network parameters 150 of the neural network 130.

[0035] Generally speaking, the neural network 130 can have any suitable neural network architecture that enables it to perform the machine learning tasks mentioned above. Figure 1A In the example, neural network 130 includes an encoder subnetwork 132 and an embedding generation subnetwork 134. A subnetwork of a neural network refers to a set of one or more neural network layers within the neural network. When the input includes text data, the encoder subnetwork 132 can be a fully connected subnetwork configured to process the input to generate an encoder network output; that is, the fully connected subnetwork includes one or more fully connected neural network layers, and in some implementations, the fully connected subnetwork includes one or more non-linear activation layers, such as ReLU activation layers. When the input includes image data, the encoder subnetwork 132 can additionally or alternatively include one or more convolutional neural network layers. The embedding generation subnetwork 134 can be similarly configured as a fully connected subnetwork and then able to process the encoder network output generated by the encoder subnetwork 132 to generate an embedding for the input, which is typically a numerical representation with a fixed dimension.

[0036] As another example, neural network 130 may be an attentional neural network comprising one or more attention layers. As used herein, an attention layer is a neural network layer that includes an attention mechanism—such as a multi-head self-attention mechanism. Examples of configurations of attentional neural networks and details of other components of attentional neural networks—such as embedding layers that embed input into the neural network or feedforward layers within layers of the attention network—are described in more detail in Vaswani, et al., AttentionIsAllYouNeed, arXiv:1706.03762 and Raffel, et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, arXiv:1910.10683, the entire contents of which are hereby incorporated herein by reference.

[0037] In some cases, the architecture of neural network 130 remains the same during both the pre-training and fine-tuning phases; however, in other cases, neural network 130 is able to have different architectures during the two phases. In the latter case, neural network 130 is able to have a common backbone subnetwork (e.g., ...) during both the pre-training and fine-tuning phases. Figure 1A The encoder subnetwork 132), and can have different auxiliary subnetworks used in each stage (e.g., embedding generation subnetwork 134 or...). Figure 1B The output subnetwork 136 used during the fine-tuning phase.

[0038] exist Figure 1A In the example, neural network 130 includes an embedding generation subnetwork 134 that is used only to assist in the training of encoder subnetwork 132 during the pre-training phase. In other words, once pre-training is complete, i.e. during the fine-tuning phase or deployment, the embedding generation subnetwork 134 will no longer be included as part of neural network 130.

[0039] In system 100, training engine 140 trains neural network 130 on unlabeled training data 110 to determine learned values ​​of network parameters 150 from initial values ​​of network parameters using an iterative training process. At each iteration of the training process, training engine 140 determines parameter value updates for the current values ​​of network parameters 150 (including parameters of encoder subnetwork 132 and embedding generator subnetwork 134), and then applies the updates to the current values ​​of network parameters 150.

[0040] In particular, in order to efficiently determine the training values ​​of the parameters 150 of the neural network 130 by utilizing unlabeled training data 110, which is relatively easier to obtain in large quantities across a wide range of machine learning tasks, i.e., compared with labeled (e.g., human-annotated) training data, the training engine 140, which works in conjunction with the corrupted engine 120, trains the neural network 130 by using a self-supervised contrastive learning technique.

[0041] Unlabeled training input 112 from unlabeled training data 110 refers to training input whose information regarding the known true value output that should be generated by neural network 130—e.g., the true value classification of the training input—is not used by system 100. Unlabeled training input 112 includes multiple features representing any kind of numerical data. In some examples, each feature can represent an attribute or feature from a set of attributes describing a subject of a classification task. In other examples, each feature can represent, for any appropriate task, different intensity values ​​of corresponding channels of corresponding pixels, different text terms in a text sequence, different amplitude values ​​in audio data, different points in a point cloud, etc.

[0042] During the pre-training phase, for each unlabeled training input 112, the corruption engine 120 processes the unlabeled training input to generate a corrupted version of the unlabeled training input (“corrupted training input”) 114 by corruption, i.e., modification, of a subset of features contained in the original unlabeled training input. In particular, the corruption engine 120 is configured to use an edge-sampling corruption technique to generate the corrupted training input 114.

[0043] Many contrastive learning and associated corruption techniques have been successful in the visual domain (e.g., image-based corruption techniques such as random cropping, color distortion, and blurring) and the natural language domain (e.g., text-based corruption techniques such as word masking, deletion, and padding). Despite being one of the most common data types in computing, tabular data is yet another seemingly missing type of data.

[0044] Specifically, in the tabular data format, the unlabeled training input 112 may have a corresponding feature in each of multiple feature dimensions, such as in each of multiple rows or columns, or both. Each corresponding feature may have a feature value representing that feature, which is typically numerical. Each corresponding feature may be a numerical feature or, alternatively, a discrete feature. In other words, the unlabeled training input 112 may include some features as numerical features and some features as discrete features. Numerical features are features that have values ​​that can be any value within a certain range; however, discrete features include binary features and other features that can only take one of a few possible values, such as categorical features.

[0045] By applying the disclosed edge sampling corruption technique, which is effective for tabular data, corruption engine 120 can generate corrupted training input 114 by first selecting which feature dimensions to corrupt and applying corruption to the feature values ​​in the feature dimensions based on the empirical marginal distribution of the feature values ​​in the training input for each selected feature dimension.

[0046] For each unlabeled training input 112, the neural network 130 processes the original, undamaged version of the unlabeled training input 112 to generate a first embedding 142. Furthermore, the neural network 130 processes the damaged training input 114, which has already been generated by the damage engine 120 from the unlabeled training input 112, to generate a second embedding 144. That is, the first embedding and the second embedding 144 are generated by the same neural network (with the same architecture and the same parameter values) for two different versions of the same training input—the original version and the damaged version.

[0047] The training engine 140 is then able to determine parameter value updates by backpropagating the gradient 146 of the contrastive loss function through the parameters of the embedding generator subnetwork 134 and the encoder subnetwork 132, the contrastive loss function measuring the difference between the first embedding 142 and the second embedding 144. For example, the contrastive loss function could be a noisy contrastive estimation (NCE) loss function, such as the InfoNCE loss function.

[0048] Figure 1B An example neural network system 100 is shown during the fine-tuning phase.

[0049] After pre-training, the training engine 120 of system 100 then uses labeled training data 116, which includes multiple labeled training inputs 118, to adapt the pre-trained neural network 130 to a downstream task, which can be any of the machine learning tasks mentioned above.

[0050] In some cases, all of the pre-trained neural networks 130 are then fine-tuned; in others, only a portion of the neural networks 130 is subsequently fine-tuned. Figure 1B In the example, in addition to having an encoder subnetwork 132, the neural network 130 also includes an output subnetwork 136 instead of an embedding generation subnetwork 134. The output subnetwork 136 can be configured to process the encoder network output generated by the encoder subnetwork 132 to generate output for downstream tasks. The embedding generation subnetwork 134 is no longer needed and therefore does not require further fine-tuning.

[0051] The labeled training input 118 from the labeled training data 116 refers to the training input that is defined or otherwise specified by the training input regarding the known true value output that should be generated by the neural network 130—for example, the true value classification of the training input—and is therefore available to the system 100.

[0052] Typically, the data used in the fine-tuning phase can be several orders of magnitude smaller than the data used in the pre-training phase. In some implementations, unlabeled training data 110 includes millions of unlabeled training inputs, while labeled training data 132 includes only a few thousand labeled training inputs. Furthermore, self-supervised contrastive learning techniques and corruption handling steps are no longer required. Instead, more conventional supervised learning techniques can be used during the fine-tuning phase.

[0053] Adapting the pre-trained neural network 130 to a downstream task involves adjusting the learning values ​​of some or all of the network parameters 150. Figure 1B In the example, during the fine-tuning phase, the parameters of the encoder subnetwork 132 and the output subnetwork 136, but not the parameters of the embedding generation subnetwork 134 which is no longer included as part of the neural network 130, are adjusted. The training engine 140 is able to determine parameter value updates by backpropagating the gradients of an objective function suitable for the downstream task through the parameters 148 of the output subnetwork 136 and the encoder subnetwork 132. For example, the objective function could be a cross-entropy loss function that measures the quality of the classification output generated by the neural network 130 by processing the training input, i.e., classifying it relative to the true values ​​associated with the training input.

[0054] Once the two-phase process is complete, system 100 can provide data for a specified trained neural network to another system—such as a server. This data may specify the architecture of the neural network (which can be the same architecture used during the fine-tuning phase rather than the pre-training phase) and the training values ​​of the network parameters 150, for use when processing new inputs. Alternatively or supplementing the provision of data for the specified trained neural network, system 100 can use the trained neural network to process new inputs and generate corresponding outputs.

[0055] Figure 2This is a flowchart of an example process 200 for pre-training a neural network using a self-supervised contrastive learning scheme. For convenience, process 200 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural network system appropriately programmed according to this specification, such as neural network system 100 of Figure 1, is capable of executing process 200.

[0056] The system obtains unlabeled training inputs from the unlabeled training data set (step 202). The unlabeled training data set can be obtained by randomly sampling from the unlabeled training data used to pre-train the neural network. The unlabeled training data set can include a fixed number of unlabeled training inputs, for example, 64, 128, or 256. The system typically performs one iteration of steps 202-208 for each unlabeled training input included in the unlabeled training data set.

[0057] Unlabeled training input can have a tabular data format. Unlabeled training input can have corresponding features in each of multiple feature dimensions. Each corresponding feature can have a feature value representing that feature, which is typically numerical. For example, unlabeled training input can include data describing a matrix, the matrix having features of the unlabeled training input arranged as matrix elements in rows or columns, where each row or column corresponds to a specific feature dimension. In other similar examples, unlabeled training input can include data describing vectors, tables, arrays, etc.

[0058] Figure 4A This is an example illustration of a pre-trained neural network. As shown, the unlabeled training input 402 is a 6-dimensional vector, meaning it has six feature dimensions. The unlabeled training input 402 has corresponding features in each of the six feature dimensions.

[0059] The system uses a neural network and processes the unlabeled training input according to the current values ​​of multiple network parameters to generate a first embedding of the unlabeled training input (step 204). The embedding can be a numerical representation with a fixed dimension.

[0060] exist Figure 4A In the example, the neural network includes an encoder subnetwork and an embedding generation subnetwork. In this example, the system is able to first process the unlabeled training input 402 according to the current values ​​of the encoder network parameters (denoted by f) to generate the encoder network output (embedding 406A), and then process the encoder network output according to the current values ​​of the embedding generation network parameters (denoted by g) to generate a first embedding 408A of the unlabeled training input.

[0061] The system generates a corrupted version of the unlabeled training input (step 206).

[0062] Generating a corrupted version of the unlabeled training input can include determining a proper subset of feature dimensions and, for each feature dimension in the proper subset of feature dimensions, applying a corrupted operation to the corresponding feature in the feature dimension using one or more feature values ​​sampled from the marginal distribution of the feature dimension specified in the unlabeled training dataset. Applying the corrupted operation can include replacing the feature in each feature dimension in the proper subset with one or more sampled feature values.

[0063] In some implementations, the system can determine the proper subset of a feature dimension by sampling a proper subset of the feature dimensions from multiple feature dimensions with uniform randomness. In other implementations, the system can determine the proper subset of a feature dimension according to a predetermined corruption rate of the total number of feature dimensions to be selected. For example, the predetermined corruption rate c can be a percentage value defined relative to the total number M of feature dimensions included in the unlabeled training input (e.g., 20%, 30%, 50%, etc.). In this example, the system can sample a total of c × M feature dimensions and then apply corruption to the corresponding features in each sampled feature dimension.

[0064] exist Figure 4A In the example, the system samples half of the six feature dimensions of the unlabeled training input 402, and then replaces the original feature values ​​in each sampled feature dimension with feature values ​​sampled from the empirical marginal distribution of the feature dimensions.

[0065] Specifically, the marginal distribution of a feature dimension can be defined as a uniform distribution across all values ​​of the feature that have been taken in the feature dimension across the unlabeled training data set. In other words, to determine one or more replacement feature values ​​for each feature dimension in a proper subset, the system can sample from a uniform distribution across all feature values ​​that appear at least a threshold number of times in the feature dimension across the unlabeled training data set. For example, the threshold is one, but in other examples the threshold can be increased.

[0066] In mathematics, let the unlabeled training dataset be... Where M is the number of feature dimensions. yes A uniform distribution above, where x j Let x represent the j-th feature dimension, for each unlabeled training input. The system can uniformly apply the feature dimensions {1, ..., M} of size q. Sampling is performed on a proper subset of the dataset, and a corrupted version of the unlabeled training input is generated. As follows: If but otherwise in

[0067] The system uses a neural network and processes a corrupted version of the unlabeled training input according to the current values ​​of multiple network parameters to generate a second embedding of the corrupted version of the unlabeled training input (step 208). In other words, the system uses the same neural network (with the same neural network architecture and the same parameter values) that was used to generate the first embedding of the unlabeled training input to generate a second embedding of the same unlabeled training input by processing a corrupted version of the unlabeled training input.

[0068] As shown in the figure, the system first processes the corrupted version 404 of the unlabeled training input according to the current value of the encoder network parameters (represented by f) to generate the encoder network output (embedding 406B), and then processes the encoder network output according to the current value of the embedding generation network parameters (represented by g) to generate the second embedding 408B of the corrupted version of the unlabeled training input.

[0069] The system, for example, computes the gradient of the contrastive learning loss function with respect to multiple network parameters via backpropagation (step 210). For each unlabeled training input in the unlabeled training data set, the contrastive learning loss function evaluates the difference between a first embedding of the unlabeled training input and a second embedding of a corrupted version of the unlabeled training input. Additionally, for each unlabeled training input in the unlabeled training data set, the contrastive learning loss function evaluates the difference between the first embedding of the unlabeled training input and a corresponding first embedding generated by the neural network for each other unlabeled training input in the set.

[0070] Contrastive learning loss functions train neural networks to generate robust representations of different versions of the same input by maximizing the similarity between corresponding embeddings of different versions of the same input (i.e., between embeddings of positive training pairs) and minimizing the similarity between corresponding embeddings of different versions of different inputs (i.e., between embeddings of negative training pairs). For example, the contrastive learning loss function can be a noisy contrastive estimation (NCE) loss function, such as the InfoNCE loss function.

[0071] The system continues to update the current parameter values ​​based on the gradient and by using appropriate gradient descent optimization techniques, such as stochastic gradient descent, RMSprop, or Adam techniques.

[0072] The system can repeatedly execute process 200 until the pre-training termination criteria are met, for example, after process 200 has been executed a predetermined number of times, after the gradient of the contrastive learning function has converged to a specified value, or after a certain early stopping criterion is met.

[0073] After determining that the pre-training termination criteria are met, the system can continue to adapt the neural network to a specific machine learning task. In some cases, fine-tuning is then performed on all pre-trained neural networks; in others, only a portion of the pre-trained neural networks is fine-tuned. In the latter case, the system can fine-tune the encoder subnetwork by co-training the encoder subnetwork with respect to labeled training data and the output subnetwork, including adjusting the learned values ​​of the encoder network parameters. Labeled training data includes training inputs specific to a particular machine learning task and each associated with a corresponding ground truth output.

[0074] Figure 3 This is a flowchart of an example process 300 for fine-tuning a neural network on a machine learning task. For convenience, process 300 will be described as being executed by a system of one or more computers located in one or more locations. For example, a neural network system properly programmed according to this specification, such as neural network system 100 of Figure 1, is capable of executing process 300.

[0075] The system uses an encoder subnetwork and processes one or more labeled training inputs from a set of labeled training inputs according to the learned values ​​of multiple encoder network parameters to generate an embedding of the labeled training inputs (step 302). For example, the set of labeled training inputs is sampled from a larger labeled training dataset.

[0076] The system uses the output subnetwork and processes the embedding according to the current values ​​of multiple output network parameters to generate training outputs for each labeled training input in the labeled training input set (step 304).

[0077] Figure 4B This is an example illustration of fine-tuning a neural network. As shown, the system can first process the labeled training input 412 according to the current values ​​of the encoder network parameters (represented by f) to generate the encoder network output (embedding 416), and then process the encoder network output according to the current values ​​of the output network parameters (represented by h) to generate the training output 416.

[0078] The system computes the supervised learning loss function (step 306). For each labeled training input in the labeled training input set, the supervised learning loss function evaluates the difference between the training output and the ground truth output associated with the labeled training input. The system also computes, for example, the gradients of the supervised learning loss function with respect to multiple encoder network parameters and with respect to multiple output network parameters via backpropagation.

[0079] exist Figure 4BIn the example, the specific machine learning task is a classification task, and the supervised learning loss function can be a classification loss function that evaluates the difference between the training output 416 and the true value output 414 associated with the labeled training input 412, such as the cross-entropy loss function.

[0080] The system then continues to update the current values ​​of the encoder network parameters and the output network parameters based on the gradient and by using appropriate gradient descent optimization techniques, such as stochastic gradient descent, RMSprop, or Adam (step 308).

[0081] In this way, the parameter values ​​learned during the pre-training process are adjusted to suit a specific machine learning task.

[0082] This specification uses the term "configuration" in conjunction with system and computer program components. For one or more computers, being configured to perform a particular operation or action means that software, firmware, hardware, or a combination thereof are installed on the system that, in operation, causes the system to perform the operation or action. For one or more computer programs, being configured to perform a particular operation or action means that one or more programs include instructions that, when executed by a data processing device, cause that device to perform the operation or action.

[0083] The embodiments of the subject matter and functional operation described in this specification can be implemented using digital electronic circuit systems, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their equivalents), or one or more combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or one or more combinations thereof. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by a data processing apparatus.

[0084] The term "data processing apparatus" refers to data processing hardware and includes all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0085] A computer program, which can also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages; and it can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but does not necessarily, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to said program, or in multiple coordinating files (e.g., a file storing portions of one or more modules, subroutines, or code). A computer program can be deployed to execute on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a data communication network.

[0086] In this specification, the term "database" is used broadly to refer to any collection of data: data that does not need to be structured in any particular way, or not structured at all, and that can be stored on storage devices in one or more locations. Thus, for example, an indexed database can comprise multiple collections of data, each of which can be organized and accessed differently.

[0087] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in others, multiple engines can be installed and run on the same one or more computers.

[0088] The processes and logic flows described in this specification can be executed by one or more programmable computers executing one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuitry such as FPGAs or ASICs, or by a combination of a dedicated logic circuitry system and one or more programmable computers.

[0089] A computer suitable for executing computer programs can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The essential components of a computer are the central processing unit for executing or carrying out instructions, and one or more memory devices for storing instructions and data. The central processing unit and memory can be supplemented by, or incorporated into, a special-purpose logic circuit system. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data, or transfer data to or to them, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device, such as a Universal Serial Bus (USB) flash drive, to name a few.

[0090] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD ROMs and DVD-ROMs.

[0091] To provide interaction with the user, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user—such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor—and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a webpage to a web browser on the user's device in response to a request received from a web browser. Furthermore, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user in return.

[0092] The data processing apparatus for implementing machine learning models can also include, for example, dedicated hardware accelerator units for handling common and computationally intensive parts of the workload, such as machine learning training or production, i.e., inference.

[0093] Machine learning models can be implemented and deployed using machine learning frameworks such as TensorFlow, Microsoft Cognitive Toolkit, Apache Singa, or Apache MXNet.

[0094] Embodiments of the subject matter described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or middleware components (e.g., an application server), or frontend components (e.g., a client computer having a graphical user interface, web browser, or app that a user can interact with through an implementation of the subject matter described herein), or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0095] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The client-server relationship occurs through computer programs running on respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data (e.g., HTML pages) to a user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from the user interacting with the device as a client. It is possible to receive data generated at the user device, such as the results of user interactions, from the device at the server.

[0096] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of a single embodiment can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented individually in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, one or more features from a claimed combination can be removed from the combination in some cases, and the claimed combination can be for sub-combinations or variations thereof.

[0097] Similarly, although operations are depicted in the accompanying drawings and described in the claims in a specific order, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or requiring the performance of all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0098] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require a specific order or sequence to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous.

[0099] What is being protected is...

Claims

1. A computer-implemented method for training a neural network with multiple network parameters, the method comprising: Obtain a set of unlabeled training inputs from an unlabeled training data set, wherein the unlabeled training inputs include images, videos, or text sequences, and each unlabeled training input in the set of unlabeled training inputs has a corresponding feature in each of multiple feature dimensions; The neural network is used to process each unlabeled training input in the set of unlabeled training inputs according to the current values ​​of the plurality of network parameters to generate a first embedding of the unlabeled training input, wherein processing each unlabeled training input includes: using the neural network to process the original undamaged version of each unlabeled training input without applying any damage to each unlabeled training input using the neural network to generate the first embedding of the unlabeled training input; For each unlabeled training input in the set of unlabeled training inputs, a corrupted version of the unlabeled training input is generated, including: Determine a proper subset of the feature dimensions from the plurality of feature dimensions, and For each feature dimension in the true subset of the feature dimensions, one or more feature values ​​sampled from the marginal distribution of that feature dimension specified in the unlabeled training dataset are used to apply corruption to the corresponding feature in that feature dimension; The neural network is used to process the corrupted version of each unlabeled training input in the set of unlabeled training inputs according to the current values ​​of the plurality of network parameters to generate a second embedding of the corrupted version of the unlabeled training input; and The neural network is trained based on an optimized contrastive learning loss function, wherein for each unlabeled training input in the set of unlabeled training inputs, the neural network is trained by optimizing the contrastive learning loss function to reduce the difference between: (i) the first embedding of the unlabeled training input generated by the neural network by processing the original undamaged version of the unlabeled training input and (ii) the second embedding of the damaged version of the unlabeled training input generated by the neural network by processing the damaged version of the unlabeled training input, and to improve the difference between: (i) the first embedding of the unlabeled training input generated by the neural network by processing the original undamaged version of the unlabeled training input and (ii) the different second embeddings of the damaged versions of the different unlabeled training inputs generated by the neural network by processing the damaged versions of the different unlabeled training inputs in the set of unlabeled training inputs.

2. The method according to claim 1, wherein, The unlabeled training input has a tabular data format.

3. The method according to claim 1, wherein, The contrastive learning loss function includes the Noise Contrast Estimation (NCE) loss function.

4. The method according to claim 3, wherein, The NCE loss function includes the InfoNCE loss function.

5. The method according to claim 1, wherein, Determining a proper subset of the feature dimensions includes sampling a proper subset of the feature dimensions from the plurality of feature dimensions with uniform randomness.

6. The method according to claim 5, wherein, The proper subset of the feature dimensions is sampled at uniform randomness according to a predetermined damage rate of the total number of feature dimensions to be selected.

7. The method according to claim 1, wherein, The one or more feature values ​​are sampled from a uniform distribution of the feature values ​​that appear at least a threshold number of times in the feature dimension across the unlabeled training input in the unlabeled training dataset.

8. The method according to claim 7, wherein, The threshold is 1.

9. The method according to claim 1, wherein, Applying the damage to the corresponding feature using one or more feature values ​​includes replacing the corresponding feature with one or more feature values.

10. The method according to claim 1, wherein, At least one feature in the feature dimension is a numerical feature.

11. The method according to claim 1, wherein, At least one feature in the feature dimension is a categorical feature.

12. The method according to claim 1, wherein, The features in the first feature dimension are numerical features, and the features in the second feature dimension are categorical features.

13. The method according to any one of claims 1-12, wherein, The neural network includes an encoder sub-neural network with multiple encoder network parameters and an embedding generator sub-neural network with multiple embedding generator network parameters.

14. The method of claim 13, further comprising: After training the neural network on the unlabeled training dataset, the encoder sub-neural network is adapted to a specific machine learning task, including adjusting the learned values ​​of the plurality of encoder network parameters using labeled data including labeled training inputs.

15. The method according to claim 14, wherein, Adapting the encoder sub-neural network to the specific machine learning task further includes: The encoder sub-neural network is used to process the labeled training input according to the learned values ​​of the plurality of encoder network parameters to generate an embedding of the labeled training input; The embedding is processed using an output sub-neural network based on the current values ​​of multiple output network parameters to generate training output; Calculate the supervised learning loss function that solves for the difference between the training output and the ground truth output associated with the labeled training input; and Based on calculating the gradient of the supervised learning loss function with respect to the plurality of encoder network parameters and with respect to the plurality of output network parameters, the adjustment of the learned values ​​of the plurality of encoder network parameters is determined.

16. The method according to claim 15, wherein, The specific machine learning task includes a classification task, and the supervised learning loss function includes a cross-entropy loss function.

17. The method of claim 14, further comprising providing the learned values ​​of the plurality of encoder network parameters for use when performing the particular machine learning task.

18. A system comprising one or more computers and one or more storage devices for storing instructions, wherein the instructions, when executed by the one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 1-17.

19. A non-volatile computer-readable storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 1-17.

Images