Comparison learning using positive example pseudo-tags

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing machine learning tasks on one or more received inputs by training a neural network using a hybrid training data set with semi-supervised learning techniques. The hybrid training data set includes a plurality of unmarked training inputs and a plurality of marked training inputs, and in some cases, the unmarked training inputs are more than the marked training inputs.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Application No. 63 / 344,026, filed on May 19, 2022. The disclosure of the prior application is considered a part of the present application and is incorporated by reference into the disclosure of the present application. Technical Field

[0002] This specification relates to training neural networks. Background Art

[0003] A neural network is a machine learning model that uses one or more layers of nonlinear units to predict outputs for received inputs. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as input to the next layer in the network, the next hidden layer or output layer. Each layer of the network generates an output from the received inputs based on the current values ​​of the corresponding set of parameters. Summary of the invention

[0004] This specification describes a system implemented as a computer program on one or more computers in one or more locations that implements and trains a neural network using both labeled and unlabeled data using semi-supervised learning techniques. Once trained, the neural network is capable of performing machine learning tasks on one or more received inputs.

[0005] According to one aspect, a computer-implemented method is provided, comprising: obtaining a batch of training inputs from a mixed training dataset, wherein the batch of training inputs includes one or more unlabeled training inputs and one or more labeled training inputs, each labeled training input having a corresponding true value label; generating a corresponding first enhanced view of each training input in the batch; processing the corresponding first enhanced view of each training input using an online neural network and according to an online network parameter value to generate a corresponding online embedding of the training input; generating a corresponding second enhanced view of each training input in the batch, wherein for each training input in the batch, the corresponding second enhanced view is different from the corresponding first enhanced view; processing the corresponding second enhanced view of each training input using a target neural network and according to a target network parameter value to generate a corresponding target embedding of the training input; updating an embedding queue to include The method includes generating corresponding target embeddings for one or more labeled training inputs in a batch by using a target neural network; for each unlabeled training input in one or more unlabeled training inputs in the batch, generating a pseudo-label based on a similarity measure between an online embedding of the unlabeled training input and each corresponding target embedding in an embedding queue; determining corresponding semantic positive samples for each training input in the batch, including sampling from the embedding queue an embedding that has been generated for labeled training inputs having the same pseudo-label or the same true value label as the training input as a semantic positive sample; determining a gradient of a loss function with respect to an online network parameter value, the loss function including a first term that encourages similarity between the corresponding online embedding of each training input and the corresponding semantic positive sample; and determining an update to the online network parameter value based on the gradient of the loss function with respect to the online network parameter value.

[0006] The loss function can include a second term that encourages similarity between the corresponding online and target embeddings of each training input.

[0007] The online neural network may include an online projection sub-neural network and an online prediction sub-neural network, and the online projection sub-neural network and the target neural network have the same network architecture but different parameter values.

[0008] The target network parameter value may be an exponential moving average of the online projection sub-network parameter values ​​of the online projection sub-neural network. The embedding queue may have a fixed capacity that depends on the size of a batch of training inputs.

[0009] The embedding queue may include corresponding target embeddings generated using the target neural network for one or more labeled training inputs in a previously obtained batch.

[0010] The training input may include image data.

[0011] The training input may include audio data.

[0012] Generating a respective first augmentation view for each training input in the batch may include: sampling one or more augmentation strategies from a set of augmentation strategies; and sequentially applying the one or more sampled augmentation strategies to each training input in the batch.

[0013] The set of augmentation strategies may include a random cropping strategy followed by a resizing strategy, a random color distortion strategy, or a random Gaussian blur strategy.

[0014] For each unlabeled training input of the one or more unlabeled training inputs in the batch, generating a pseudo label may include: determining k nearest embeddings of the unlabeled training input from the embedding queue using a k-nearest neighbor model, where k is a positive integer; and generating a pseudo label for the unlabeled training input from the true value labels associated with the k nearest embeddings.

[0015] In some cases, k=1, and the pseudo-label can be the same as the true value label associated with the determined most recent embedding.

[0016] In some cases, k>=2, and the pseudo-label can be the highest occurring true value label among the determined recent embeddings.

[0017] The k-nearest neighbor model can be configured to use cosine similarity to determine the k nearest embeddings for each unlabeled training input.

[0018] Generating a pseudo label for each of the one or more unlabeled training inputs in the batch may include: selecting one or more k-nearest neighbor models from a plurality of k-nearest neighbor models, each k-nearest neighbor model corresponding to a different enhancement strategy; and determining k nearest embeddings of the unlabeled training input from a queue of embeddings using each selected k-nearest neighbor model.

[0019] The mixed training dataset may include more unlabeled training inputs than labeled training inputs.

[0020] The method may further include training a task sub-neural network together with the online projection sub-neural network to optimize a supervised task-specific loss for a downstream task, wherein, for a training input from the hybrid training dataset, the task sub-neural network may be configured to process a projection embedding generated by the online projection sub-neural network according to the task sub-network parameter values ​​to generate a downstream task output for the training input.

[0021] The downstream task may include a classification task, and wherein the supervised task specific loss may include a cross entropy loss.

[0022] According to another aspect, one or more computer-readable storage media are provided, the one or more computer-readable storage media being encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the above-described method aspects.

[0023] According to yet another aspect, a system is provided, the system comprising one or more computers and one or more storage devices storing instructions, wherein when the instructions are executed by the one or more computers, the one or more computers perform corresponding operations of the above-mentioned method aspects.

[0024] According to another aspect, a method is provided, comprising: receiving a network input; and processing the network input using a neural network, the neural network comprising an online projection sub-neural network and a task sub-neural network trained by the method of any of the above aspects to generate one or more network outputs for the network input, comprising: processing the network input using the online projection sub-neural network to generate a projection embedding; and processing the projection embedding using the task sub-neural network to generate one or more network outputs.

[0025] It will be appreciated that features described in the context of one aspect may be combined with features described in the context of another aspect.

[0026] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages. The system described in this specification pre-trains a neural network using a semi-supervised learning technique that effectively combines labeled data and unlabeled data to generate information representations that may be useful later in specific downstream tasks. The system uses a relatively small amount of labeled data to impute pseudo-label information for a much larger amount of unlabeled data, and then incorporates the imputed pseudo-label information into a contrastive learning scheme to train the neural network to generate similar representations for each pair of training inputs with the same true value label or pseudo-label. Specifically, unlike many existing semi-supervised learning techniques that use available label information as supervision within a cross-entropy objective, the described system uses this label information to determine which training inputs should have similar representations.

[0027] In addition, the pre-trained neural network can then be used to effectively adapt to a specific machine learning task using orders of magnitude less data than the data used to pre-train the network. For example, while the pre-trained network may utilize billions of unlabeled training inputs, adapting the network to a specific task may only require a few thousand labeled training inputs. Compared to other conventional training methods, the system is therefore able to more efficiently utilize computing resources, such as processor cycles, memory, or both, during training. The system is also able to train the neural network using orders of magnitude less labeled data, and correspondingly, at orders of magnitude lower human costs associated with data labeling, while still ensuring that the trained neural network has competitive performance comparable to or even exceeding the state-of-the-art on a range of tasks, while additionally being generalizable and easily adaptable to new tasks.

[0028] Pre-training large neural networks that can be used for real-world tasks typically results in significant amounts of carbon dioxide (CO2) emissions and significant amounts of electricity usage. By reducing the number of FLOPs that need to be performed and performing fewer training iterations for the reasons described above, the described techniques significantly reduce the CO2 footprint of the pre-training process, while also significantly reducing the amount of electricity consumed by the pre-training process.

[0029] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 An example training system is shown.

[0031] Figure 2 is an example illustration of semantic positive examples via the pseudo-labeling (SEMPPL) training process.

[0032] Figure 3 is a flowchart of an example process for training an online neural network.

[0033] Figure 4 A quantitative example of the performance gain that can be achieved by using the SEMPPL process described in this specification is shown.

[0034] Like reference numbers and names in the various drawings indicate like elements. DETAILED DESCRIPTION

[0035] This specification describes a system implemented in the form of a computer program on one or more computers in one or more locations that implements and trains a neural network that can perform a machine learning task on one or more received inputs. Depending on the task, the neural network can be configured to receive any type of digital data input and generate any type of score, classification, or regression output based on the input.

[0036] For example, a neural network can be configured to perform an image processing task, for example, to receive an input including image data comprising a plurality of pixels. The image data can, for example, include one or more images or features that have been extracted from one or more images. The neural network can be configured to process the image data to generate an output of the image processing task.

[0037] For example, if the task is image classification, the output generated by the neural network for a given image may be a score for each object category in a set of object categories, where each score represents an estimated likelihood that the image contains an object belonging to that category.

[0038] As another example, if the task is object detection, the output generated by the neural network for a given image may be one or more bounding boxes, each bounding box being associated with a corresponding score, where each bounding box represents an estimated location in the image and the corresponding score represents the estimated likelihood that an object is depicted at that location in the image, i.e., within the bounding box.

[0039] As another example, if the task is semantic segmentation, the output generated by the neural network for a given image can be a label for each of a plurality of pixels in the image, wherein each pixel is labeled as belonging to one of a set of object categories. Alternatively, for each of a plurality of pixels, the output can be a set of scores including a corresponding score for each object category in the set of object categories, the score representing the likelihood that the pixel belongs to an object from that object category.

[0040] As another example, if the input to the neural network is an internet resource (e.g., a web page), a document or a portion of a document, or features extracted from an internet resource, a document or a portion of a document, then the output generated by the neural network for a given internet resource, document or portion of a document may be a score for each topic in a set of topics, where each score represents an estimated likelihood that the internet resource, document or portion of a document is related to that topic.

[0041] As another example, if the input to a neural network is characteristics of the impression context of a particular advertisement, the output generated by the neural network may be a score representing the estimated likelihood that the particular advertisement will be clicked.

[0042] As another example, if the input to the neural network is features of personalized recommendations for a user, e.g., features characterizing the context of the recommendation, e.g., features characterizing previous actions taken by the user, the output generated by the neural network may be a score for each content item in a set of content items, where each score represents an estimated likelihood that the user will respond positively to recommending the content item.

[0043] As another example, the task could be a health prediction task, where the input is a sequence derived from a patient's electronic health record data, and the output is a prediction related to the patient's future health, e.g., a predicted treatment that should be prescribed to the patient, the likelihood that the patient will experience an adverse health event, or a predicted diagnosis for the patient.

[0044] As another example, the task can be an agent control task, where the input is a sequence of observations or other data characterizing the state of an environment (such as a real world or simulated environment), and the output defines the action to be performed by the agent in response to the most recent data in the sequence. The agent can be, for example, a real world or simulated robot, a control system for an industrial facility (e.g., a temperature control system for the facility, or a system that divides tasks among units in the facility), or a control system that controls different types of agents. The observations can be the output of a sensor (e.g., a camera) that monitors the environment.

[0045] Figure 1 An example training system 100 is shown. Training system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0046] The training system 100 includes a neural network 110 and a semantically positive examples via pseudo labels (SEMPPL) training engine 120 , or simply “training engine.” The neural network 110 is configured to receive inputs 102 and generate one or more outputs 112 based on the received inputs 102 and network parameter values ​​of the neural network 110 .

[0047] At a high level, the neural network 110 includes an online projection sub-neural network 120 and a task sub-neural network 130. During the forward pass of each inference computation, the online projection sub-neural network 120 processes the input 102 to generate a projection embedding, which is then received and processed by the task sub-neural network 130 to generate one or more network outputs 112. An embedding refers to an ordered collection of values, such as a vector, matrix, or other tensor of values.

[0048] A sub-neural network of a neural network refers to a group of one or more neural network layers in a neural network. Each sub-neural network can be implemented using any suitable neural network architecture that enables the sub-neural network to perform its described function. The projected embedding means the embedding produced by successively applying the functions of consecutive layers of the sub-neural network 120 to the input 102.

[0049] In some examples, when the input 102 of the neural network 110 includes an image, the online projection sub-neural network 120 can be a convolutional sub-neural network, that is, including one or more convolutional layers configured to process the image to generate an embedding of the image. When the input 102 includes text data or other low-dimensional data, the online projection sub-neural network 120 can additionally or alternatively include one or more fully connected layers. The task sub-neural network 130 can include one or more output layers configured to process the embedding to generate an output. For example, when the task is a classification task, the task sub-neural network 130 can include one or more fully connected layers, followed by a softmax layer that generates a score distribution over a set of categories. Another example is that when the task is a regression task, the task sub-neural network 130 can include one or more linear layers that generate output values.

[0050] Before the neural network 110 can be used to perform any tasks, the training engine 140 of the training system 100 trains the neural network 110 on the mixed training data set 150, ie, enables the neural network 110 to effectively perform tasks on new data.

[0051] The hybrid training dataset 150 is a dataset that includes both labeled training inputs 152, for which known true value labels, such as true value classifications of the training inputs, which should be generated by the neural network 110, can be used to train the system 100, and unlabeled training inputs 154, for which no known true value labels are available. In various cases, the training inputs (labeled 152 or unlabeled 154) can be or include image data, audio data, text data, or some combination thereof. In various cases, because unlabeled training data is relatively easier to obtain in large quantities across a wide range of tasks, that is, compared to labeled (e.g., manually or machine annotated) training data, the hybrid training dataset 150 will include more - sometimes many times more - unlabeled training inputs 154 than labeled training inputs 152.

[0052] By utilizing a semi-supervised learning technique (“Semantic Positive Examples via Pseudo-Labels” or “SEMPPL” technique) and an abundant amount of unlabeled training data, the training engine 140 of the training system 100 pre-trains the online projection sub-neural network 120 on the mixed training data set 150 to determine the training parameter values ​​of the online projection sub-neural network 120. The purpose of the pre-training process can be viewed as learning to generate meaningful embeddings that may be useful in a wide range of downstream tasks and to make the adaptation of the larger neural network 110 with the online projection sub-neural network 120 to a specific downstream task faster and more computationally efficient.

[0053] Specifically, by maintaining a queue 145 of embeddings generated from labeled training inputs 152 during the pre-training phase, the training engine 140 is able to select one or more similar embeddings from the queue for unlabeled training inputs, and correspondingly use the true value labels of the selected similar embeddings to generate pseudo labels for the unlabeled training inputs. The selection of similar embeddings includes querying the embedding queue 145 using a k-nearest neighbor ("k-NN") algorithm or a similar technique. In this way, the training engine 140 ensures that any training input from the mixed training data set 150 is labeled, i.e., either has a true value label already available or has a pseudo label filled in.

[0054] After pre-training has been completed, the pre-trained online projection sub-neural network 120 can be adapted to the downstream task. The downstream task can be any of the tasks mentioned above. Specifically, the training system 100 can train the online projection sub-neural network 120 together with the task sub-neural network 130, which in some cases is an untrained neural network, for example, a neural network with randomly initialized parameter values ​​that has not been previously trained during the pre-training stage. During the adaptation process, the parameter values ​​of the task sub-neural network 130 are adjusted, and in some cases the parameter values ​​of the online projection sub-neural network 120 learned during the pre-training are adjusted, so that the neural network 110 with both the sub-neural network 120 and the sub-neural network 130 is adapted to the downstream task.

[0055] The training engine 140 can adjust parameter values ​​based on optimizing the supervised task-specific loss of the downstream task. In some examples, if the downstream task is a classification task, the supervised loss can be a cross entropy loss; if the downstream task is a regression task, the supervised loss can be a mean square error (MSE) loss function. The adaptation process can use labeled training inputs 152 within the same mixed training data set 150, or alternatively use a different labeled training data set specifically curated for the downstream task.

[0056] Once both the pre-training and adaptation processes have been completed, the training system 100 can provide data specifying the trained neural network 110, such as the training values ​​of the network parameters of the online projection sub-neural network 120 and the task sub-neural network 130 and the data specifying the architecture of the sub-neural networks 120 and 130, to another system, such as an inference system in a deployment environment, for use in processing new inputs. Instead of providing data specifying the trained neural network 110 or in addition, the training system 100 can also use the trained neural network to process new inputs 102 and generate corresponding outputs 112.

[0057] Figure 2 is an example illustration of a semantically positive examples via pseudo labels (SEMPPL) training process. Typically, one iteration of the SEMPPL training process can be performed for each training input 201 within a batch of training inputs sampled from a mixed training data set 250. As shown, the SEMPPL training process includes three phases: a contrastive learning phase, during which a contrastive learning loss term based on (optional) enhancements is calculated; followed by a pseudo-label generation phase (performed for each unlabeled training input); followed by a semantic positive example query phase, during which a contrastive learning loss term based on semantic positive examples is calculated. After performing one SEMPPL training iteration for each training input 201 within the batch, the parameters of the online neural network can then be updated based on optimizing a SEMPPL loss function that includes two contrastive learning loss terms that have been calculated for the entire batch of training inputs.

[0058] The contrastive learning phase begins by transforming each training input 201 in a batch of training inputs using one or more different augmentation strategies to generate a first augmented view 203 and a second augmented view 205 of the training input 201. The training inputs 201 can be labeled or unlabeled. The specific operations to be performed to transform each training input may vary from implementation to implementation, for example, depending on the data modality of the training inputs 201.

[0059] By way of illustration and not limitation, when the training inputs include image data, the following set of augmentation strategies can be used to transform each training input.

[0060] Random Cropping Crop the image at a randomly chosen point, such as in the center or one of the four corners Random resize Randomly select patches of the image between the minimum and maximum crop regions of the image, with aspect ratios log-uniformly sampled in [3 / 4, 4 / 3]. Upscale the patches to square images of size s×s via bicubic interpolation. Flip Horizontal --- Color Distortion Randomly adjust the brightness, contrast, saturation, and hue of the image uniformly in random order by values ​​in [−a, a], where a is the maximum adjustment Grayscale Scaling Combine the channels into one channel with the value 0.2989r + 0.5870g + 0.1140b Random Blur Apply a 23×23 Gaussian kernel with a standard deviation uniformly sampled in [0.1, 2.0] Random exposure Each channel value is thresholded so that all values ​​less than 0.5 are replaced by 0, and all values ​​greater than or equal to 0.5 are replaced by 1. Saliency Masking Remove at least some of the background with uniform grayscale noise with randomly sampled gray levels Table 1

[0061] Similarly, when the training input includes text data, the enhancement strategies that can be used to transform each training input can include, for example, synonym replacement, random insertion, random swap, random deletion, etc.; when the training input includes audio data, the enhancement strategies that can be used to transform each training input can include, for example, noise injection, time shift, spacing change, etc.

[0062] To transform each training input 201 in the sampled batch to generate a first enhanced view 203 and a second enhanced view 205 of the training input, the training engine can sample a first sequence (“a1”) of one or more enhancement strategies and a second sequence (“a2”) of one or more enhancement strategies from the set of enhancement strategies. The training engine can then sequentially apply a first sequence of the one or more sampled enhancement strategies within the first sequence to the training input to generate the first enhanced view, and apply a second sequence of the one or more sampled enhancement strategies within the second sequence to the training input to generate the second enhanced view.

[0063] When sampling from a set of augmentation strategies with at least some randomness, the first and second sequences of augmentation strategies, and thus augmented views of the same training input, will generally differ from each other. For example, the first augmented view 203 is generated from the training input 201 as a result of applying a random cropping strategy, followed by a resizing strategy, followed by a random Gaussian blur strategy, while the second augmented view 205 is generated from the training input 201 as a result of applying a random cropping strategy, followed by a random color distortion strategy, followed by a random exposure strategy.

[0064] For each training input 201 within the sample batch, the training engine provides the first enhanced view 203 of the training input as input to the online neural network 210. Figure 2 As shown, the online neural network 210 includes an online projection sub-neural network 220 and an online prediction sub-neural network 215. The online projection sub-neural network 220 further includes an encoder backbone 211 and a projection head 213.

[0065] The online neural network 210 processes the first augmented view 203 according to current values ​​of parameters of the online neural network (“online network parameter values”) to generate an online embedding 216 of the training input 201 .

[0066] Specifically, the online neural network 210 first uses the encoder backbone 211 to encode the first enhanced view 203 into the encoded embedding 212, then uses the projection head 213 to project the encoded embedding 212 into the projection embedding space to generate the projection embedding 214, and finally uses the online predictor neural network 215 to process the projection embedding 214 to generate the online embedding 216. In some embodiments, the output of the online predictor neural network 215 is directly used as the online embedding 216, while in other embodiments, the online embedding 216 is the output of the online predictor neural network 215 that is further transformed, such as L-2 normalized.

[0067] For each training input 201 in the sample batch, the training engine provides a second augmented view 205 of the training input as input to the target neural network 240, which is another instance of the online projection sub-neural network 220 used to assist the SEMPPL training process. Figure 2 As shown, the target neural network 240 includes a target encoder backbone 231 and a target projection head 233.

[0068] Specifically, the target neural network 240 has the same network architecture as the online projection sub-neural network 220, but at at least some points during training, it has different parameter values ​​than the online projection sub-neural network 220.

[0069] The training engine can cause the parameter values ​​of the target neural network 240 ("target network parameter values") to differ from the online projection sub-network parameter values ​​in any of a variety of ways.

[0070] For example, the training engine uses the target neural network 240 to emulate the online projection sub-neural network 220, wherein the online projection sub-network parameter values ​​from the online projection sub-neural network 220 are copied to the target neural network 240 at intervals.

[0071] As another example, the training engine can make the target network parameter value an exponential moving average of the online projection sub-network parameter values ​​of the online projection sub-neural network 220. That is, the target network parameter value is not to copy the online projection sub-network parameter value to the target neural network, but to calculate the target network parameter value based on the online projection sub-network parameter value. Slowly track the online projection subnetwork parameter values, where represents the target network parameter value, and represents the online projection subnetwork parameter value, and .

[0072] The target neural network 240 processes the second enhanced view 205 according to the target network parameter values ​​to generate a target embedding 234 of the training input 201. Specifically, the target neural network 240 first encodes the second enhanced view 205 into an encoded embedding 232 using the target encoder backbone 231, and then projects the encoded embedding 232 into a projected embedding space using the target projection head 233 to generate the target embedding 234. As with the online neural network 210, in some embodiments, the output of the target sub-neural network 240 is directly used as the target embedding 234, while in other embodiments, the target embedding 234 is the output of the target sub-neural network 240 that is further transformed, such as L-2 normalized.

[0073] Now it is possible to calculate an augmentation-based contrastive learning loss term that is optionally used by the system. For each training input 201, this loss term encourages similarity, i.e., reduces the embedding space distance between the online embedding 216 and the target embedding 234 that have been generated by the online neural network 210 and the target neural network 240, respectively. As described above, the two embeddings have been generated from different augmented views 203 and 205 of each training input 201, where each augmented view is generated by applying a different data augmentation strategy to the training input.

[0074] On the other hand, the loss term encourages dissimilarity, i.e., increasing the embedding space distance between the online embedding 216 and the target embedding 234 that have been generated by the online neural network 210 and the target neural network 240, respectively, from the training input 201 and any other different training input within the batch (which is different from the training input 201).

[0075] Therefore, the encoder backbone 211 of the online neural network 210 is trained based on the enhanced contrastive learning loss term to generate similar embeddings (which are close to each other in the embedding space) for a pair of different enhanced views of the same training input, and generate distinctive embeddings (which are separated from each other in the embedding space) for different training inputs.

[0076] The boost-based contrastive learning loss term when used can be calculated, for example, as follows: Where B represents the batch size of the training input ; is the scoring function, is the scalar temperature, and <∙,∙> denotes the Euclidean dot product; is the online embedding (the hat operator indicates that it is L-2 normalized), where h represents the parameters of the online predictor neural network, g represents the parameters of the projection head, and f represents the parameters of the encoder backbone of the online neural network; is the target embedding, where represents the parameters of the target projection head, and The parameters of the target encoder backbone representing the target neural network; is a set of negative examples (different training inputs randomly sampled from the same batch), and is the target embedding generated by the target neural network for each such negative example.

[0077] Additionally, if the training input 201 is a labeled training input, the training engine will update the first-in-first-out (FIFO) queue 250 to include (a copy of) the target embedding 234 for the training input 201. In this manner, the FIFO queue 250, which will be accessed during subsequent stages of the SEMPPL training iterations, stores the target embeddings that have been generated for one or more labeled training inputs within the batch using the target neural network 240. While logically described as a first-in-first-out (FIFO) queue, it will be appreciated that any other type of data structure (e.g., a buffer or list) can be used.

[0078] At the beginning of SEMPPL training, the FIFO queue 250 can be initialized with a random vector. Then, for each subsequent SEMPPL training iteration, the target embeddings generated by using the target neural network 240 for the labeled training inputs within the batch of training inputs sampled for the SEMPPL training iteration will be appended to the FIFO queue 250.

[0079] Figure 2 The embedding FIFO queue 250 in the example can have a fixed capacity C, which generally depends on the size B of the batch of training inputs. For example, the capacity C can be set to be no less than the size B of the batch of training inputs, so that the queue 250 can store not only the target embeddings generated for the labeled training inputs in the currently sampled batch, but also the target embeddings generated for the labeled training inputs in one or more previously sampled batches. For example, C = 10B or 20B, where B = 2048, 4096, etc. As the number and diversity of available target embeddings increase, the training effect may also be improved.

[0080] Entering now the pseudo label generation phase, the training engine accesses the FIFO queue 250 of target embeddings to generate a pseudo label 252 for each of the one or more unlabeled training inputs within the batch.

[0081] Specifically, for each unlabeled training input, the training engine can identify, from the FIFO queue 250 of target embeddings, the k nearest embeddings (where k is a positive integer) that are most similar to the online embeddings that have been generated by the online neural network from the unlabeled training inputs, using a k-nearest neighbor algorithm ("k-NN") or similar techniques (e.g., support vector machines (SVMs), random forest techniques, etc.), and then generate pseudo labels 252 for the unlabeled training inputs from the true value labels associated with the k nearest embeddings. Here, "similarity" is defined in terms of distance in the embedding space. Distance can be calculated in any suitable manner, such as using Euclidean distance, Hamming distance, cosine similarity, to name a few examples.

[0082] Depending on the exact number of recent embeddings selected, the training engine can generate pseudo labels 252 for unlabeled training inputs in any of a variety of ways. For example, when k = 1, the pseudo labels can be the same as the true value labels associated with a single determined recent embedding; when k >= 2, the pseudo labels 252 can be the highest occurring true value labels among multiple determined recent embeddings.

[0083] In addition, some embodiments of the system can maintain multiple k-nearest neighbor models, each corresponding to a different augmentation strategy. Thus, the training engine first selects a k-nearest neighbor model from the multiple k-nearest neighbor models for each augmentation strategy used to transform the unlabeled training input, and then uses each selected k-nearest neighbor model to determine the k most recent embeddings of the unlabeled training input from the FIFO queue 250 of target embeddings. In these embodiments, the pseudo-label 252 can similarly be the most frequently occurring true value label among all the most recent embeddings collectively selected by the multiple k-nearest neighbor models.

[0084] Now entering the semantic positive query phase (during which pseudo labels will be used), the training engine determines corresponding semantic positive samples for each training input in the batch 262. Specifically, unlike the pseudo label generation phase in which pseudo labels are generated only for unlabeled training inputs, in the semantic positive query phase, semantic positive samples are determined for the entire batch of training inputs.

[0085] For each training input 201 in the batch, the training engine uniformly and randomly samples the embeddings generated for the labeled training inputs with the same label as the training input 201 from the FIFO queue 250 of the target embeddings as the semantic positive samples 262. That is, if the training input 201 is a labeled training input, the embedding generated for the labeled training input with the same true value label as the training input 201 will be selected; alternatively, if the training input 201 is an unlabeled training input, the embedding generated for the labeled training input with the same pseudo label as the training input 201 will be selected.

[0086] Now it is possible to calculate a semantic positive-based contrastive learning loss term. For each training input 201, this loss term encourages similarity, i.e., reduces the distance in the embedding space between the online embedding 216 of the training input 101 and the corresponding semantic positive sample 262 obtained from the queue 205. The semantic positive sample 262 is a target embedding that has been generated for a labeled training input with the same pseudo label as the training input 201 or the same true value label.

[0087] On the other hand, the loss term encourages dissimilarity, i.e., increasing the distance in the embedding space between the online embedding 216 and the target embedding 234 that has been generated by the online neural network 210 and the target neural network 240 from the training input 201 and any other training inputs within the batch, respectively.

[0088] Therefore, the encoder backbone 211 of the online neural network 210 is trained based on a contrastive learning loss term based on semantic positive examples to generate similar embeddings (which are close to each other in the embedding space) for a pair of different training inputs that both have the same label, and to generate distinctive embeddings (which are separated from each other in the embedding space) for different training inputs.

[0089] The contrastive learning loss term based on semantic positive examples can be calculated, for example, as: in Represents each training input in batch B Semantic positive samples, where if is a labeled training input, then , and if is the unlabeled training input, then ,in is the true value label, and the bar symbol indicates that it is a pseudo label.

[0090] When the optional enhancement-based contrastive learning loss term is not calculated, the SEMPPL loss function can include only the semantic positive example-based contrastive learning loss term.

[0091] Alternatively, when putting the two loss terms together, the SEMPPL loss function can be calculated, for example, as: , in is a tunable hyperparameter that controls the ratio between the two loss terms. For example, Can take a value less than 1, such as 1 / 5. Setting it to a value less than 1 takes into account the following situation: when computing the contrastive learning loss term based on semantic positive examples, some other training inputs in the batch may have the same label as training input 201.

[0092] Figure 3 is a flow chart of an example process 300 for training an online neural network. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a suitably programmed training system, such as Figure 1 The training system 100 in can perform process 300.

[0093] The system obtains a batch of training inputs from the mixed training data set (step 302). The batch of training inputs that can be obtained by random sampling from the mixed training data set includes one or more unlabeled training inputs and one or more labeled training inputs. Each labeled training input is associated with a corresponding true value label.

[0094] The system generates a respective first augmentation view for each training input in the batch (step 304). This can include sampling one or more augmentation policies from a set of augmentation policies with at least some measure of randomness, and then sequentially applying the one or more sampled augmentation policies to each training input in the batch.

[0095] The system generates a corresponding online embedding for each training input in the batch using an online neural network (step 306). The online neural network includes an online projection sub-neural network and an online prediction sub-neural network. For each training input, the online neural network is configured to process a first enhanced view of the training input according to the online network parameter values ​​to generate an online embedding.

[0096] The system generates a corresponding second enhanced view for each training input in the batch (step 308). The system can do this by using a similar approach as described above in step 304. Due to the randomness of the augmentation strategy sampling, for each training input in the batch, the corresponding second enhanced view is typically different from the corresponding first enhanced view.

[0097] The system generates a corresponding target embedding for each training input in the batch using a target neural network (step 310). The target neural network has the same network architecture as the online projection sub-neural network, but different parameter values. For each training input, the target neural network is configured to process the second augmented view of the training input according to the target network parameter values ​​to generate a target embedding.

[0098] The system updates the embedding queue to include the corresponding target embeddings generated using the target neural network for one or more labeled training inputs in the batch (step 312). For example, when implemented as a FIFO queue, the system can append the target embeddings that have been generated for each labeled training input to the queue (potentially overwriting one or more already stored embeddings).

[0099] The system generates a pseudo-label for each of the one or more unlabeled training inputs in the batch, the pseudo-label being based on a similarity measure between the online embedding of the unlabeled training input and each corresponding target embedding in the embedding queue (step 314). This can include using a k-nearest neighbor algorithm or similar technique to determine the k nearest embeddings of the unlabeled training input from the embedding queue, where k is a positive integer, and then generating a pseudo-label for the unlabeled training input from the true value labels associated with the k nearest embeddings. In other words, the system determines which target embeddings are similar to the online embedding of each unlabeled training input, and since the similar target embeddings are each generated from labeled training inputs, the system can use the true value labels associated with those similar target embeddings to determine which pseudo-label among a possible set of labels can be assigned to the unlabeled training input.

[0100] The system determines a corresponding semantic positive sample for each training input in the batch (step 316). This can include sampling an embedding from a queue of embeddings as a semantic positive sample, the embedding having been generated for a labeled training input having the same pseudo label or the same true value label as the training input.

[0101] The system evaluates the SEMPPL loss function and determines the gradient of the SEMPPL loss function with respect to the online network parameter values ​​(step 318 ).

[0102] The SEMPPL loss function includes a semantically positive contrastive learning loss term that encourages similarity between the corresponding online embedding and the corresponding semantically positive sample for each training input within a batch, but discourages similarity between the corresponding online embedding and the target embedding that have been generated for different training inputs within the batch. Optionally, the SEMPPL loss function also includes an enhanced contrastive learning loss term that encourages similarity between the corresponding online embedding and the target embedding that have been generated for each training input within the batch, but discourages similarity between the corresponding online embedding and the target embedding that have been generated for different training inputs within the batch.

[0103] The system determines updates to the online network parameter values ​​based on the gradients of the SEMPPL loss function with respect to the online network parameter values ​​(step 320). The system can then determine the updates by applying an optimizer suitable for training online neural networks to the gradients, such as an Adam optimizer, an rmsProp optimizer, or a layer-by-layer adaptive rate scaling (LARS) optimizer.

[0104] The system can continue to perform iterations of process 300 to train the online neural network by repeatedly adjusting online network parameter values ​​until a termination criterion for online neural network training has been met, e.g., until the parameters have converged, until a threshold amount of wall clock time has elapsed, or until a threshold number of iterations of process 300 have been performed.

[0105] Figure 4 Quantitative examples of the performance gains that can be achieved by using the SEMPPL process described in this specification are shown. In particular, Figure 4The best (Top-1) accuracy (in %, higher is better) is shown for images from: ImageNetV2 (its three variants: Matching Frequency (MF), Threshold 0.7 (T-0.7), and Top Image (TI), which are described in more detail in BenjaminRecht, et al. Do imagenet classifiers generalize to imagenet? In Proc. of International Conference on Machine Learning (ICML), 2019), ImageNet-R (which is described in more detail in Dan Hendrycks, et al. The many faces of robustness: A critical analysis ofout-of distribution generalization. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 8340–8349, 2021), and ObjectNet (Described in more detail in Andrei Barbu, et al. A large-scale bias-controlled dataset for pushing the limits of object recognition models. Proc. of Advances in Neural Information Processing Systems (NeurIPS), 2019).

[0106] As can be appreciated, neural networks trained using the SEMPPL procedure typically have improved robustness and out-of-distribution generalization capabilities over neural networks trained using the PAWS procedure (described in more detail in Mahmoud Assran, et al. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2021), the SimMatch procedure (described in more detail in Mingkai Zheng, et al. Simmatch: Semi-supervised learning with similarity matching. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), 2022), or even supervised baselines.

[0107] This specification uses the term "configured" in conjunction with system and computer program components. To configure a system of one or more computers to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform the operation or action. To configure one or more computer programs to perform a particular operation or action means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0108] The embodiments and functional operations of the subject matter described in this specification can be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more thereof. The embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more computer program instruction modules, which are encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode information for transmission to a suitable receiving device for execution by a data processing device.

[0109] The term "data processing apparatus" refers to data processing hardware and covers all types of apparatus, devices and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus can also be or further include special-purpose logic circuits, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus can optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0110] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, such as one or more scripts stored in a markup language document, in a single file dedicated to related programs, or in multiple coordinated files, such as files storing one or more modules, subroutines, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers located at one site or across multiple sites and interconnected by a data communications network.

[0111] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and can be stored on storage devices in one or more locations. Thus, for example, an index database can include multiple data collections, each of which can be organized and accessed in a different way.

[0112] Similarly, in this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same one or more computers.

[0113] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating input data and generating output. The processes and logic flows can also be performed by a dedicated logic circuit such as an FPGA or ASIC, or a combination of a dedicated logic circuit and one or more programmed computers.

[0114] A computer suitable for executing a computer program can be based on a general or special purpose microprocessor or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a central processing unit for executing or implementing instructions and one or more storage devices for storing instructions and data. The central processing unit and the memory can be supplemented or incorporated into a dedicated logic circuit. Typically, the computer will also include or be operably coupled to receive data from or transmit data to or both of one or more large-capacity storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk. However, the computer does not need to have such a device. In addition, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name just a few examples.

[0115] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0116] To provide interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, as well as a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. In addition, the computer can interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, the computer can interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving a response message from the user in return.

[0117] The data processing apparatus for implementing the machine learning model can also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive part of machine learning training or production, namely the inference workload.

[0118] Ability to implement and deploy machine learning models using machine learning frameworks such as the TensorFlow framework or the JAX framework.

[0119] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, for example, as a data server, or includes a middleware component, for example, an application server, or includes a front-end component, for example, a client computer with a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0120] The computing system can include a client and a server. The client and the server are usually remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by means of computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server transmits data such as HTML pages to the user device, for example, for displaying data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device, such as the result of the user interaction, can be received from the device at the server.

[0121] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that may be claimed, but should be interpreted as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable sub-combination in multiple embodiments. In addition, although features may be described as working in certain combinations and even initially claimed as such, in some cases, one or more features in the claimed combination can be deleted from the combination, and the claimed combination can be directed to a sub-combination or a variant of the sub-combination.

[0122] Similarly, although operations are depicted in a particular order in the figures and described in the claims, it should not be understood as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0123] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or in that order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computer-implemented method, include: Obtaining a batch of training inputs from the mixed training data set, wherein the batch of training inputs includes one or more unlabeled training inputs and one or more labeled training inputs, each labeled training input having a corresponding true value label; generating a respective first augmented view for each training input in the batch; processing the respective first augmented view of each training input using an online neural network and according to online network parameter values ​​to generate a respective online embedding of the training input; generating a respective second enhanced view for each training input in the batch, wherein, for each training input in the batch, the respective second enhanced view is different from the respective first enhanced view; processing the respective second augmented view of each training input using a target neural network and according to target network parameter values ​​to generate a respective target embedding for the training input; updating an embedding queue to include corresponding target embeddings generated for the one or more labeled training inputs in the batch using the target neural network; generating, for each unlabeled training input of the one or more unlabeled training inputs in the batch, a pseudo label based on a similarity measure between an online embedding of the unlabeled training input and each corresponding target embedding in the embedding queue; For each training input in the batch, a corresponding semantic positive sample is determined, including Sampling, from the embedding queue, an embedding that has been generated for a labeled training input having the same pseudo label or the same true value label as the training input as the semantic positive sample; determining a gradient of a loss function with respect to the online network parameter value, the loss function comprising a first term that encourages similarity between the corresponding online embedding of each training input and the corresponding semantic positive sample; and Based on the gradient of the loss function with respect to the online network parameter value, an update to the online network parameter value is determined.

2. The method according to claim 1, in, The loss function includes a second term that encourages similarity between the corresponding online embedding of each training input and the target embedding.

3. The method according to any one of claims 1 to 2, in, The online neural network includes an online projection sub-neural network and an online prediction sub-neural network. The online projection sub-neural network and the target neural network have the same network structure but different parameter values.

4. The method according to claim 3, in, The target network parameter value is an exponential moving average of the online projection sub-network parameter values ​​of the online projection sub-neural network.

5. The method according to any one of claims 1 to 4, in, The embedding queue has a fixed capacity that depends on the batch size of the training input.

6. The method according to any one of claims 1 to 5, in, The embedding queue includes corresponding target embeddings generated using the target neural network for one or more labeled training inputs in a previously obtained batch.

7. The method according to any one of claims 1 to 6, in, The training input includes image data.

8. The method according to any one of claims 1 to 6, in, The training input includes audio data.

9. The method according to any one of claims 1 to 8, in, Generating the respective first augmented view of each training input in the batch comprises: sampling one or more enhancement strategies from the set of enhancement strategies; and One or more sampled augmentation strategies are applied sequentially to each training input in the batch.

10. The method according to claim 9, in, The set of enhancement strategies includes a random cropping strategy followed by a resizing strategy, a random color distortion strategy, or a random Gaussian blur strategy.

11. The method according to any one of claims 1 to 10, in, Generating the pseudo label for each unlabeled training input of the one or more unlabeled training inputs in the batch includes: Determining the k nearest embeddings of the unlabeled training input from the embedding queue using a k-nearest neighbor model, where k is a positive integer; and The pseudo labels are generated for the unlabeled training inputs from the true value labels associated with the k nearest embeddings.

12. The method according to claim 11, in, k = 1, and wherein the pseudo label is the same as the true value label associated with the determined most recent embedding.

13. The method according to claim 11, in, k >= 2, and wherein the pseudo-label is the highest occurring true value label among the determined recent embeddings.

14. The method according to any one of claims 11 to 13, in, The k-nearest neighbor model is configured to determine the k nearest embeddings for each unlabeled training input using cosine similarity.

15. The method according to any one of claims 11 to 14, in, Generating the pseudo label for each of the one or more unlabeled training inputs in the batch comprises: Selecting one or more k-nearest neighbor models from among a plurality of k-nearest neighbor models each corresponding to a different boosting strategy; and Using each selected k-nearest neighbor model, the k nearest embeddings for the unlabeled training input are determined from the embedding queue.

16. The method according to any one of claims 1 to 15, in, The mixed training data set includes more unlabeled training inputs than labeled training inputs.

17. The method of claim 3 or any one of claims 4 to 16 when dependent on claim 3, further comprising training a task sub-neural network together with the online projection sub-neural network to optimize a supervised task-specific loss for a downstream task, in, For a training input from the mixed training dataset, the task sub-neural network is configured to process the projection embedding generated by the online projection sub-neural network according to the task sub-network parameter values ​​to generate a downstream task output for the training input.

18. The method according to claim 17, in, The downstream task comprises a classification task, and wherein the supervised task-specific loss comprises a cross-entropy loss.

19. A system comprising one or more computers and one or more storage devices storing instructions, the instructions being operable when executed by the one or more computers to cause the one or more computers to perform the operations of the corresponding method of any preceding claim.

20. A computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any preceding claim.

21. A method, include: Receive network input; as well as Processing the network input using a neural network, the neural network comprising an online projection sub-neural network and a task sub-neural network trained by the method of any one of claims 17 to 18 to generate one or more network outputs for the network input, comprising: Processing the network input using the online projection sub-neural network to generate a projected embedding; as well as The projected embedding is processed using the task sub-neural network to generate the one or more network outputs.

Images