Supervised contrastive learning using plurality of positive examples

Abstract

To provide an improved training method that enables supervised contrastive learning to be performed simultaneously over a plurality of positive and negative training examples.SOLUTION: An exemplary embodiment of the present disclosure is directed to a supervised version with improved batch contract loss, which has been shown to be very effective for learning a strong expression in a self-supervised setting. The proposed method thus enables adaptation of contrastive learning to a fully supervised setting, and enables learning to be performed simultaneously over a plurality of positive examples.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] Related Applications This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 013,153, filed April 21, 2020. U.S. Provisional Patent Application No. 63 / 013,153 is incorporated herein by reference in its entirety.

[0002] TECHNICAL FIELD This disclosure relates generally to systems and methods for contrastive learning of visual representations, and more particularly to systems and methods for performing supervised contrastive learning across multiple positive examples. [Background technology]

[0003] Cross-entropy loss is likely the most widely used loss function for supervised learning. It is naturally defined as the KL divergence between two discrete distributions: the empirical label distribution (a discrete distribution of one-hot vectors) and the empirical distribution of logits.

[0004] Some work has explored the drawbacks of this loss, such as its lack of robustness to noisy labels and the possibility of insufficient margins leading to poor generalization performance. However, in practice, most proposed alternatives do not appear to perform better on large datasets such as ImageNet, as evidenced by the continued use of cross-entropy to achieve state-of-the-art results.

[0005] Many proposed improvements to the standard cross-entropy loss actually involve relaxing the definition of the loss, specifically, making the reference distribution axis-aligned. Models trained with these modifications show improved generalization, robustness, and calibration. However, the proposed improvements do not completely eliminate the shortcomings of the cross-entropy loss approach. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 113-123, 2019. [Non-patent document 2] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical data augmentation with no separate search. arXiv preprint arXiv:1909.13719, 2019 [Non-patent document 3] A variant of the strategy of Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020 to sequentially apply random color distortion and Gaussian blurring, where an additional sparse image warp is probabilistically added to the end of the sequence. Summary of the Invention [Means for solving the problem]

[0007] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0008] One exemplary aspect of the present disclosure is directed to a computing system for performing supervised control learning of visual representations. The computing system includes one or more processors; one or more non-transitory computer-readable media collectively storing: a base encoder neural network configured to process an input image to generate an embedded representation of the input image; a projection head neural network configured to process the embedded representation of the input image to generate a projection representation of the input image; and instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations include obtaining an anchor image associated with a first class of a plurality of classes, a plurality of positive images associated with the first class, and one or more negative images associated with one or more other classes of the plurality of classes, the one or more other classes being different from the first class. The operations include processing, with the base encoder neural network, the anchor image to obtain the anchor embedded representation of the anchor image, the plurality of positive images to obtain the plurality of positive embedded representations, and the one or more negative images to obtain the one or more negative embedded representations, respectively. The operations include processing, with a projection head neural network, the anchor embedded representation to obtain an anchor projection representation of the anchor image, the plurality of positive embedded representations to obtain a plurality of positive projection representations each, and the one or more negative embedded representations to obtain one or more negative projection representations each. The operations include evaluating a loss function that evaluates a similarity metric between the anchor projection representation, each of the plurality of positive projection representations, and each of the one or more negative projection representations. The operations include modifying one or more values ​​of one or more parameters of at least the base encoder neural network based at least in part on the loss function.

[0009] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0010] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain associated principles.

[0011] A detailed discussion of embodiments, directed to those skilled in the art, is set forth herein, and this specification refers to the accompanying drawings. [Brief explanation of the drawings]

[0012] [Figure 1A] FIG. 10 illustrates the difference between supervised versus self-supervised contrastive loss, according to an exemplary embodiment of the present disclosure. [Figure 1B] FIG. 10 illustrates the difference between supervised versus self-supervised contrastive loss, according to an exemplary embodiment of the present disclosure. [Figure 2] FIG. 10 illustrates the difference between cross-entropy, self-supervised control loss, and supervised control loss, in accordance with an exemplary embodiment of the present disclosure. [Figure 3] FIG. 1 illustrates an exemplary contrastive learning framework, according to an exemplary embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates an exemplary fine-tuning model featuring a base encoder trained according to an exemplary contrastive learning framework, in accordance with an exemplary embodiment of the present disclosure. [Figure 5A] FIG. 1 is a block diagram of an exemplary computing system according to an exemplary embodiment of the present disclosure. [Figure 5B] FIG. 1 is a block diagram of an exemplary computing device according to an exemplary embodiment of the present disclosure. [Figure 5C] FIG. 1 is a block diagram of an exemplary computing device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013] Reference numbers that are repeated among the drawings are intended to identify like features in various implementations.

[0014] overview In general, the present disclosure is directed to improved training methods that allow for supervised control learning to be performed simultaneously across multiple positive and negative training examples. In particular, exemplary aspects of the present disclosure are directed to an improved supervised version of batch control loss, which has been shown to be highly effective in learning powerful representations in self-supervised settings. Thus, the proposed technique adapts control learning to fully supervised settings and also allows for learning to be performed simultaneously across multiple positive examples.

[0015] By allowing contrastive learning to occur simultaneously across multiple positive training examples from the same class and multiple negative training examples from other classes, entire clusters of points belonging to the same class can be pulled together in the embedding space, while clusters of samples from different classes are simultaneously pushed apart. Thus, the proposed technique operates to increase the similarity between positive pairs of samples and decrease the similarity between negative pairs. Similarity can be defined, for example, as the dot product between the low-dimensional representations. The resulting representation is a very good representation that can be used for various downstream transfer tasks.

[0016] Thus, the proposed system and method can exploit label or other class assignment information more effectively than systems that use cross-entropy loss. In particular, rather than matching the representation of each class to a class value along a particular axis, as is done in cross-entropy loss, the use of a contrastive learning approach allows the representation of each class to be learned at any point in the embedding space, allowing more robust cross-class representations to be learned.

[0017] Exemplary models trained according to the proposed techniques consistently outperform cross-entropy in supervised learning tasks across different architectures and data augmentation. Specifically, as demonstrated by exemplary experimental results contained in U.S. Provisional Patent Application No. 63 / 013,153, for both ResNet-50 and ResNet-200, exemplary models trained according to the proposed techniques outperform cross-entropy by more than 1%, setting a new technical figure of 78.8% among methods using AutoAugment data augmentation.

[0018] The proposed loss also shows clear advantages in terms of robustness to natural corruptions on standard benchmarks for both calibration and accuracy. Furthermore, compared to cross-entropy, the proposed supervised control loss is more robust to hyperparameter settings such as optimizer or data augmentation. Additional aspects of the present disclosure leverage important factors such as large batch sizes and normalized representations, which have been shown to be beneficial for self-supervised learning.

[0019] More specifically, exemplary aspects of the present disclosure are directed to a new loss for supervised training that eliminates the reference distribution entirely; instead, exemplary implementations of the proposed system simply impose that normalized representations from the same class are closer to each other than representations from different classes. The proposed loss is built on a family of contrastive objective functions that have recently achieved excellent performance in self-supervised learning in the image and video domains and have relevance to the extensive literature on metric learning.

[0020] As the name suggests, contrastive loss consists of two "opposing forces": for a given anchor point, the first force pulls the anchor closer to other points in the representation space, and the second force pushes the anchor away from other points. The former set is known as positive, the latter as negative.

[0021] One aspect of the present disclosure is to consider many positives per anchor in addition to many negatives (e.g., as opposed to the practice in self-supervised control learning, which uses only a single positive). In some implementations, provided labels or other class-based designations can be used to select positives and negatives. Figures 1A-1B and 2A-2C provide a visual illustration of the proposed loss.

[0022] Specifically, FIGS. 1A and 1B illustrate the difference between supervised versus self-supervised contrastive loss. In an exemplary implementation of the supervised contrastive loss proposed herein (shown generally in FIG. 1A), positives from one class are contrasted with negatives from other classes (since labels are provided). This results in images from the same class (but not necessarily showing the exact same scene or object) being mapped to nearby points within a low-dimensional hypersphere. In contrast, in self-supervised contrastive loss (shown generally in FIG. 1B), no labels are provided. Thus, positives are generated as data augmentations of given samples (e.g., crops, flips, color changes), and negatives are randomly sampled from the mini-batch. This results in false negatives (shown in the bottom right), which may not be correctly mapped and may result in a worse representation.

[0023] 2A-2C illustrate the differences between cross-entropy, self-supervised control loss, and supervised control loss. In particular, cross-entropy loss (shown generally in FIG. 2A) uses labels and softmax loss to train a model, while self-supervised control loss (shown generally in FIG. 2B) uses control loss and data augmentation to learn representations for classes. However, in some implementations, the proposed supervised control loss (shown generally in FIG. 2C) has two stages: in the first stage, labels are used to select images for control loss, including multiple positive examples and one or more negative examples. In the second stage, the learned representations are frozen, and then a classifier or other task-specific head can be trained (e.g., on a linear layer) using softmax or cross-entropy loss. This two-stage approach combines all the advantages of using labels, control loss, and softmax or cross-entropy loss.

[0024] The resulting loss is stable over training, as shown by the exemplary empirical results included in U.S. Provisional Patent Application No. 63 / 013,153. As an example, exemplary models trained according to the proposed supervised control technique achieve very good top-1 accuracy on the ImageNet dataset in ResNet-50 and ResNet-200 architectures. For ResNet-50 using auto-augmentation, the exemplary implementation achieves 78.8% top-1 accuracy, a 1.6% improvement over cross-entropy loss using the same data augmentation and architecture. Furthermore, the increase in top-1 accuracy is accompanied by improved robustness, as measured on the ImageNet-C dataset.

[0025] The disclosed system and method provide several technical effects and benefits. As one exemplary technical effect, the proposed loss enables models to learn state-of-the-art representations compared to cross-entropy, resulting in significant improvements in top-1 accuracy and robustness. Thus, the proposed technique improves the performance and functionality of computing systems themselves in various tasks, such as image classification tasks.

[0026] As another exemplary technical effect, the proposed loss is less sensitive to hyperparameter ranges than cross-entropy, which is an important practical consideration. This reduced sensitivity results from the loss's more natural formulation, which pulls together representations of samples from the same class rather than forcibly pulling them toward a specific target, as does cross-entropy. In a typical training system, multiple training runs may be required to "tune" hyperparameters to find acceptable or optimal performance. By reducing the sensitivity of the loss and model to hyperparameters, fewer training rounds may need to be performed to tune hyperparameters. Reducing the number of training rounds performed results in savings in computing resources, such as processor usage, memory usage, and network bandwidth.

[0027] As yet another exemplary technical effect, as analytically demonstrated herein, the gradient of the proposed loss function promotes learning from hard positives and hard negatives. In some existing systems, explicit, computationally intensive approaches to identifying specific hard negatives are typically implemented. One exemplary approach of this nature is "hard negative mining." These approaches improve performance but require the expenditure of significant computational resources. By naturally promoting learning from hard positives and hard negatives, such explicit attempts to identify hard examples (e.g., hard negatives) can be avoided, enabling savings in computing resources such as processor usage, memory usage, and network bandwidth.

[0028] As yet another exemplary technical effect, the proposed approach can reduce the number of false negatives generated by a trained model. For example, by enabling learning about positive examples that represent different objects or scenes within the same class, such positive examples can be avoided from being treated as negative examples. Thus, the trained model learns to provide consistent representations for all class members, rather than only for specific objects or scenes. A trained model with reduced false negatives represents an improvement in the functionality of the computing system itself.

[0029] Thus, aspects of the present disclosure provide an improved control learning loss that outperforms cross-entropy in classification accuracy and robustness benchmarks. Furthermore, exemplary experiments show that this loss is less sensitive to changes in hyperparameters, which may be a useful practical consideration. The loss function provides a natural connection between fully unsupervised training at one end and fully supervised training at the other. This opens up the possibility of applications in semi-supervised learning that can leverage the benefits of a single loss that can smoothly shift behavior based on the availability of labeled data.

[0030] Exemplary Systems and Methods for Supervised Contrastive Learning This section begins by reviewing contrastive learning losses for self-supervised representation learning. We then show how to modify these losses to make them suitable for fully supervised learning, while preserving properties important to self-supervised approaches.

[0031] An illustrative representation learning framework One exemplary representation learning framework is shown in Figure 3 and is structurally similar to that typically used for self-supervised contrastive learning. As shown in Figure 3, one exemplary representation learning framework consists of the following components (see also Figures 1A-1B and 2A-2C for an explanation of the differences between supervised and self-supervised scenarios):

[0032] The optional data augmentation module 203, A(·), divides the input image 202, x, into a randomly augmented image 212,

number

[0033] Referring to the data augmentation module 203, as one example, the first stage of augmentation may include applying a random crop to the image and then resizing it to the image's native resolution. As another example, in light of the observation that self-supervised contrast losses require significantly different data augmentation than cross-entropy losses, the second stage may include some or all of the different augmentations described in any of the following:

[0034] AutoAugment: Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 113-123, 2019.

[0035] RandAugment: Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical data augmentation with no separate search. arXiv preprint arXiv:1909.13719, 2019

[0036] SimAugment: A variant of the strategy of Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020 to sequentially apply random color distortion and Gaussian blurring, where an additional sparse image warp is probabilistically added to the end of the sequence.

[0037] More generally, the data augmentation module 203 may perform any different combination of one or more augmentations. Furthermore, some example implementations of the present disclosure do not perform augmentation of the input image 202, but instead simply use the input image 202 without augmentation.

[0038] Referring again to FIG. 3, the framework generates an augmented image 212

number

number

[0039] In some example implementations of the framework that generate two augmented images for each input image, both augmented images for each input image may be input separately to the same encoder, resulting in a pair of representation vectors. More generally, the same encoder network 204 is typically used to generate representations for all images in a training batch.

[0040] Two commonly used encoder architectures for the encoder network 204 include ResNet-50 and ResNet-200, which, for example, have a final pooling layer (D E =2048) can be used as the representation vector 214. In some example implementations, this representation layer is

number

[0041] The framework also converts the normalized representation vector r214 into a projected representation 216 suitable for computing the contrastive loss.

number

[0042] The loss function for the input image 202 can be evaluated based on the projection representation 216 (e.g., via comparison with other projection representations generated for other input images, such as other positive and negative training examples).

[0043] In some implementations, the projection network 206 is used only to train the supervised control loss. After training is complete, this network can be discarded and replaced with a task-specific head (e.g., a single linear layer). The embedded representations 214 from the encoder 204 provide improved performance in downstream tasks than the projected representations 216 from the projection network 206. Thus, in some implementations, the proposed inference-time model can include exactly the same number of parameters as their cross-entropy equivalents.

[0044] As an example, Figure 4 shows a graphical representation of an exemplary use of the base encoder neural network 204 after it has been trained in the exemplary framework shown in Figure 3. In particular, the base encoder neural network 204 has been extracted and an additional task-specific model 250 has been added to the base encoder neural network 204. For example, the task-specific model 250 can be any type of model, including a linear model or a non-linear model such as a neural network.

[0045] The task-specific model 250 and / or the base encoder neural network 204 may be additionally trained (e.g., “fine-tuned”) on additional training data (which may be, for example, task-specific data). The additional training may be, for example, supervised learning training.

[0046] After fine-tuning, additional inputs 252 can be provided to the base encoder neural network 204, which can generate embedded representations 254. The task-specific model 250 can receive and process the embedded representations 254 to generate task-specific predictions 256. By way of example, the task-specific predictions 256 can be classification predictions, detection predictions, recognition predictions, regression predictions, segmentation predictions, and / or other prediction tasks.

[0047] Furthermore, in some implementations, the embedded representations 254 may be used directly for tasks such as similarity search or retrieval without the need for further training / fine-tuning (e.g., without the need for additional task-specific models 250).

[0048] While this disclosure focuses on example data from the image domain for ease of explanation, the framework is extendable to example data from different domains, including text and / or audio domains. Exemplary types of imagery that may be used include video frames, LiDAR point clouds, computed tomography scans, X-ray images, hyperspectral images, and / or various other forms of imagery.

[0049] Exemplary contrastive losses: self-supervised and supervised Example implementations of the present disclosure provide a contrastive loss function that allows for the impactful incorporation of labeled data while preserving the beneficial properties of contrastive loss that have been most important to the success of self-supervised representation learning. Similar to self-supervised contrastive learning, example implementations of the present disclosure can generate mini-batches by randomly sampling data. As an example, consider a set of N randomly sampled image / label pairs, {x k , y k} k=1...N For , the corresponding mini-batch used for training is 2N pairs

number

number

number

number

[0050] An exemplary self-supervised contrastive loss Within a mini-batch, let i∈{1...2N} be the index of any augmented image, and j(i) be the index of other augmented images derived from the same source image. In self-supervised control learning, the loss takes the form:

number

number

number

number

[0051] It is insightful to consider the effect on the encoder of minimizing Equation 1. During training, for any i, the encoder is adjusted to maximize the numerator of the logarithmic argument in Equation 2 while simultaneously minimizing its denominator. Both the numerator and denominator contain the term exp(z iz j(i) ) ensures that the logarithmic argument is no larger than 1, and because Equation 1 sums over all pairs of indices ((i,j) and (j,i)), the encoder is constrained to minimize the denominator or maximize the numerator without also doing the other. As a result, the encoder learns to map similar views to adjacent representations while mapping dissimilar views to non-adjacent representations, but proceeds based on only a single positive example per evaluation.

[0052] Exemplary supervised contrastive loss For supervised learning, the contrastive loss in Equation 2 cannot handle the case where two or more samples are known to belong to the same class with certainty. To generalize the loss to handle any number of positives belonging to the same class, the following new loss function is proposed:

number

number

number

number

[0053] Generalization to an arbitrary number of positives. One of the major structural changes in Equation 4 relative to Equation 2 is that now, for any anchor, all positives in the mini-batch (i.e., those in the augmentation base as well as any of the remaining 2(N-1) entries from the same class) contribute to the numerator. For large mini-batch sizes with respect to the number of classes, there are multiple additional terms (on average,

number

[0054] The contrast power increases with more positives and / or negatives. The exemplary general form of the self-supervised contrast loss given in Equation 4 exploits the principle that adding more positive and / or negative examples improves the ability to distinguish between signal and noise (negatives). This property has been shown to be important for representation learning via self-supervised contrast learning, and many studies have shown that performance improves with an increasing number of negatives.

[0055] The exemplary supervised control loss in Equation 4 maintains this structure: adding a large number of negatives to the denominator increases the positive controls, and similarly, adding a large number of positives to the numerator increases the negative controls. More positives allow the model to better capture intra-class variation. More negatives allow the model to capture inter-class variation.

[0056] Example supervised contrastive loss gradient property description Here, in this subsection, we provide further motivation for the exemplary supervised control loss form of Equation 4 by showing that its gradient has a structure that naturally focuses learning on hard positives and negatives (i.e., anchors for which continuing to contrast anchors greatly benefits the encoder) rather than weak ones (i.e., anchors for which continuing to contrast anchors only slightly benefits the encoder). Thus, the loss proves efficient in its training. Other control losses, such as triplet loss, often use the computationally expensive technique of mining hard negatives to improve training effectiveness. As a by-product of this analysis, we show that the addition of a normalization layer at the end of the projection network is beneficial, as its presence allows the gradient to have this structure.

[0057] If W is the projection network output just before normalization (ie, z=w / ||w||), then the gradient of Equation 4 with respect to w has the form:

number

number

number

number

number

[0058] However, in the case of hard positives, z i z j ≒0, and P ij is moderate, so:

number

[0059] Thus, in the case of a weak positive where further control effort reduces returns,

number

[0060] Exemplary Devices and Systems 5A illustrates a block diagram of an exemplary computing system 100 according to an exemplary embodiment of the present disclosure. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150, which are communicatively coupled via a network 180.

[0061] The user computing device 102 may be any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0062] The user computing device 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 114 may include one or more non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing device 102 to perform operations.

[0063] In some implementations, the user computing system 102 can store or include one or more machine-learned models 120. For example, the machine-learned models 120 can be or otherwise include various machine-learned models, such as neural networks (e.g., deep neural networks) or other types of machine-learned models including nonlinear and / or linear models. The neural networks can include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Exemplary machine-learned models 120 are discussed with reference to FIGS. 3 and 4.

[0064] In some implementations, one or more machine-learned models 120 may be received from a server computing system 130 over a network 180, stored in a user computing device memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, a user computing device 102 may implement multiple parallel instances of a single machine-learned model 120 (e.g., to perform parallel predictions across multiple instances of an input).

[0065] Additionally or alternatively, one or more machine-learned models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship. For example, the machine-learned models 140 may be implemented by the server computing system 130 as part of a web service (e.g., a prediction service). Thus, one or more models 120 may be stored and implemented at the user computing device 102 and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0066] The user computing device 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be useful for implementing a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0067] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operatively connected processors. The memory 134 may include one or more non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0068] In some implementations, server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0069] As described above, the server computing system 130 can store or otherwise include one or more machine-learned models 140. For example, the models 140 can be or otherwise include various machine-learned models. Exemplary machine-learned models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Exemplary models 140 are discussed with reference to FIGS. 3 and 4.

[0070] The user computing device 102 and / or the server computing system 130 can train the models 120 and / or 140 by interacting with a training computing system 150 that is communicatively coupled via a network 180. The training computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130.

[0071] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple operably connected processors. The memory 154 may include one or more non-transitory computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0072] The training computing system 150 may include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored at the user computing device 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backpropagation (e.g., according to the framework shown in FIG. 3 ). For example, a loss function may be backpropagated through the model to update one or more parameters of the model (e.g., based on the gradient of the loss function). Various loss functions may be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions such as those described herein in Equations 1-4. Gradient descent techniques may be used to iteratively update the parameters for several training iterations.

[0073] In some implementations, performing backpropagation may include performing abbreviated temporal backpropagation. The model trainer 160 can implement several generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0074] In particular, model trainer 160 can train machine-learned models 120 and / or 140 based on a set of training data 162. Training data 162 can include data of different modalities, such as, for example, images, audio samples, text, etc. Exemplary types of images that may be used include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images, and / or various other forms of imagery.

[0075] In some implementations, if the user provides consent, the training examples may be provided by the user computing device 102. Thus, in such implementations, the model 120 provided to the user computing device 102 may be trained by the training computing system 150 against user-specific data received from the user computing device 102. In some cases, this process may be referred to as personalizing the model.

[0076] Model trainer 160 includes computer logic used to provide the desired functionality. Model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general-purpose processor. For example, in some implementations, model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored on a tangible computer-readable storage medium, such as RAM, a hard disk, or optical or magnetic media.

[0077] Network 180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. Generally, communications over network 180 may be carried over any type of wired and / or wireless connection, using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0078] 5A illustrates one exemplary computing system that may be used to implement the present disclosure. Other computing systems may be used as well. For example, in some implementations, the user computing device 102 may include a model trainer 160 and a training dataset 162. In such implementations, the model 120 may be both trained and used locally on the user computing device 102. In some such implementations, the user computing device 102 may implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0079] 5B illustrates a block diagram of an exemplary computing device 10 for performing according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0080] The computing device 10 includes several applications (e.g., applications 1-N). Each application includes its own machine learning library and machine-learned model. For example, each application may include a machine-learned model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc.

[0081] 5B, each application may communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application may communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0082] 5C illustrates a block diagram of an exemplary computing device 50 for performing according to an exemplary embodiment of the present disclosure. Computing device 50 may be a user computing device or a server computing device.

[0083] Computing device 50 includes several applications (e.g., applications 1-N). Each application communicates with a central intelligence layer. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a common API across all applications).

[0084] The central intelligence layer includes several machine-learned models. For example, as shown in FIG. 5C , a respective machine-learned model (e.g., model) may be provided to each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine-learned model. For example, in some implementations, the central intelligence layer may provide a single model (e.g., single model) to all applications. In some implementations, the central intelligence layer is included in or otherwise implemented by the operating system of computing device 50.

[0085] The central intelligence layer can communicate with a central device data layer, which can be a centralized repository of data for computing device 50. As shown in FIG. 5C , the central device data layer can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0086] Additional Disclosures The technology discussed herein refers to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functionality among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications may be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0087] While the present subject matter has been described in detail with reference to various specific exemplary embodiments thereof, each example is provided by way of explanation, not limitation, of the disclosure. Those skilled in the art, upon understanding the foregoing, may readily conceive of modifications, variations, and equivalents to such embodiments. Accordingly, the present disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter as would be readily apparent to one skilled in the art. For example, features illustrated or described as part of one embodiment may be used with another embodiment to yield yet a further embodiment. Accordingly, the present disclosure is intended to cover such modifications, variations, and equivalents. [Explanation of symbols]

[0088] 10. Computing Devices 100 Computing Systems 102 User Computing Devices 112 processors 114 memory 116 Data 118 Command 120 machine learning models 122 User Input Components 130 Server Computing System 132 processors 134 memory 136 Data 138 Command 140 machine learning models 150 Training Computing System 152 processors 154 memory 156 Data 158 Command 160 Model Trainer 162 training data sets 180 Network 202 input images 203 Data Augmentation Module 204-based encoder neural network 206 Projection Network 212 Randomly Expanded Images 214 Embedding Representation Vector 216 Projection Network 250 task-specific models 252 additional inputs 254 Embedded Expressions 256 task-specific predictions

Claims

1. 1. A computing system for performing supervised contrastive learning of visual representations, comprising: one or more processors; one or more non-transitory computer-readable media; a base encoder neural network configured to process an input image to generate an embedded representation of the input image; a projection head neural network configured to process the embedded representation of the input image to generate a projection representation of the input image; instructions that, when executed by the one or more processors, cause the computing system to perform a plurality of operations, the plurality of operations comprising: operations of obtaining an anchor image associated with a first class of a plurality of classes, a plurality of positive images associated with the first class, and one or more negative images associated with one or more other classes of the plurality of classes, the one or more other classes being different from the first class; processing, with the base encoder neural network, an anchor image to obtain an anchor embedded representation of the anchor image, processing the plurality of positive images to obtain a plurality of positive embedded representations, respectively, and processing the one or more negative images to obtain one or more negative embedded representations, respectively; processing, by the projection head neural network, the anchor embedded representation to obtain an anchor projection representation of the anchor image, processing the plurality of positive embedded representations to obtain a plurality of positive projection representations respectively, and processing the one or more negative embedded representations to obtain one or more negative projection representations respectively; evaluating a loss function that evaluates a similarity metric between the anchor projection representation, each of the plurality of positive projection representations, and each of the one or more negative projection representations; modifying one or more values ​​of one or more parameters of at least the base encoder neural network based at least in part on the loss function; Including instructions and one or more non-transitory computer-readable media collectively storing: a computing system including:

2. The computing system of claim 1 , wherein the anchor image and at least one of the one or more positive images depict different objects belonging to a same first class of the plurality of classes.

3. 3. The computing system of claim 1, wherein the plurality of positive images includes all images contained in a training batch associated with the first class, and the one or more negative anchor images includes all images contained in the training batch that are not associated with any of the plurality of classes other than the first class.

4. 4. The computing system of claim 1, wherein the operations further comprise: expanding the anchor image, the plurality of positive images, and each of the one or more negative images, respectively, before processing the anchor image, the plurality of positive images, and each of the one or more negative images with the base encoder neural network.

5. 5. The computing system of claim 1, wherein the projection head neural network includes a normalization layer that normalizes the projection representation of the input image.

6. The computing system of claim 1 , wherein the similarity metric comprises a dot product.

7. 7. The computing system of claim 1, wherein the loss function comprises a product of a regularization term and a sum of a control loss term across all images in the training batch, the regularization term normalizing for the number of images in the first class of anchor images.

8. The regularization term is minus 1 divided by 2 times the number of images in the first class of anchor images minus 1 8. The computing system of claim 7, comprising:

9. the logarithm of the index of the similarity metric between the anchor image and the image under evaluation when the image under evaluation by the sum is in the first class, divided by the sum of the indexes of similarity between the anchor image and all images not in the first class; 9. A computing system according to claim 7 or 8, comprising:

10. after the plurality of operations modify one or more values ​​of one or more parameters of at least the base encoder neural network based at least in part on the loss function; adding a classification head to the base encoder neural network; fine-tuning the classification head based on a set of supervised training data; 10. The computing system of claim 1, further comprising:

11. after the plurality of operations modify one or more values ​​of one or more parameters of at least the base encoder neural network based at least in part on the loss function; providing additional inputs to the base encoder neural network; receiving an additional embedded representation for the additional input as an output of the base encoder neural network; generating a prediction of the additional input based at least in part on the additional embedded representation; and 11. The computing system of claim 1, further comprising:

12. The computing system of claim 11 , wherein the prediction comprises a classification prediction, a detection prediction, a recognition prediction, a regression prediction, a segmentation prediction, or a similarity search prediction.

13. The computing system of claim 1 , wherein the anchor image comprises an X-ray image.

14. The computing system of claim 1 , wherein the anchor image comprises a set of LiDAR data.

15. The computing system of claim 1 , wherein the anchor image comprises a video.

16. 16. A computer-implemented method comprising performing all of the operations of any one of claims 1 to 15 by a computing system including one or more computing devices.

17. 16. One or more non-transitory computer-readable media that collectively store at least a base encoder neural network trained by performing all of the operations of any one of claims 1 to 15.

Images