Supervised contrastive learning with multiple positives

Abstract

The present disclosure provides an improved training method that enables supervised contrastive learning to be performed simultaneously across multiple positive and negative training examples. In particular, example aspects of the present disclosure target an improved supervised version of the batch contrastive loss that has been shown to be very effective for learning strong representations in a self-supervised setting. Thus, the proposed techniques adapt contrastive learning to a fully supervised setting and also enable learning to occur simultaneously across multiple positive examples.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 013,153, filed April 21, 2020. U.S. Provisional Patent Application No. 63 / 013,153 is hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates generally to systems and methods for contrastive learning for visual representations. More specifically, the present disclosure relates to systems and methods that perform supervised contrastive learning across multiple positives. BACKGROUND

[0004] Cross-entropy loss can be the most widely used loss function in supervised learning. It is naturally defined as the KL-divergence between two discrete distributions: the empirical label distribution of logits (discrete distribution of 1-hot vectors) and the empirical distribution.

[0005] Many works have explored the shortcomings of this loss, such as lack of robustness to noisy labels and possible margin gap, which lead to reduced generalization performance. However, in practice, most of the proposed alternatives do not seem to work better for large-scale datasets, such as ImageNet, which can be evidenced by the fact that the state-of-the-art results are achieved by consistently using cross-entropy.

[0006] Many of the proposed improvements to regular cross-entropy actually involve relaxation of the loss definition, especially that the reference distribution is axis-aligned. Models trained with these modifications show improved generalization capability, robustness, and calibration. However, the proposed improvements do not completely eliminate the shortcomings of the cross-entropy loss method. SUMMARY

[0007] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be obvious from the description, or can be learned through practice of the embodiments.

[0008] One example aspect of this disclosure relates to a computational system for performing supervised contrastive learning of visual representations. The computational system includes one or more processors and one or more non-transitory computer-readable media, which jointly store: a basic encoder neural network configured to process an input image to generate an embedded representation of the input image; a projection head neural network configured to process the embedded representation of the input image to generate a projected representation of the input image; and instructions that, when executed by the one or more processors, cause the computational system to perform operations. These operations include obtaining an anchored image associated with a first class of a plurality of classes, a plurality of positive images associated with the first class, and one or more negative images associated with one or more other classes of a plurality of classes, the other classes being different from the first class. The operations include processing the anchored image with the basic encoder neural network to obtain an anchored embedded representation of the anchored image, processing the plurality of positive images to obtain a plurality of positive embedded representations respectively, and processing the one or more negative images to obtain one or more negative embedded representations respectively. The operations include processing the anchored embedded representation with the projection head neural network to obtain an anchored projected representation of the anchored image, processing the plurality of positive embedded representations to obtain a plurality of positive projected representations respectively, and processing the one or more negative embedded representations to obtain one or more negative projected representations respectively. These operations include evaluating a loss function that assesses a similarity measure between the anchored projection representation and each of a plurality of orthogonal projection representations and each of one or more negative projection representations. These operations also include modifying one or more values ​​of one or more parameters of at least a basic encoder neural network, at least in part, based on the loss function.

[0009] Other aspects of this disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0010] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0011] A detailed discussion of embodiments applicable to those skilled in the art is set forth in the description with reference to the accompanying drawings, wherein:

[0012] FIG. 1A and 1B The difference between supervised and self-supervised contrastive losses according to an example embodiment of this disclosure is shown.

[0013] FIG. 2A -C illustrates the differences between cross-entropy, self-supervised contrastive loss, and supervised contrastive loss according to an example embodiment of this disclosure.

[0014] FIG. 3An example contrastive learning framework according to example embodiments of the present disclosure is shown.

[0015] FIG. 4 An example fine-tuned model is shown featuring a base encoder learned according to an example contrastive learning framework according to example embodiments of the present disclosure.

[0016] FIG. 5A A block diagram of an example computing system according to example embodiments of the present disclosure is depicted.

[0017] FIG. 5B A block diagram of an example computing device according to example embodiments of the present disclosure is depicted.

[0018] FIG. 5C A block diagram of an example computing device according to example embodiments of the present disclosure is depicted.

[0019] Reference numbers repeated in multiple figures are intended to identify the same feature in various embodiments. DETAILED DESCRIPTION

[0020] Overview

[0021] In general, the present disclosure relates to an improved training method that enables simultaneous supervised contrastive learning across multiple positive and negative training examples. In particular, example aspects of the present disclosure are directed to an improved supervised version of the batch contrastive loss that has been shown to be very effective for learning strong representations in a self-supervised setting. Thus, the proposed techniques adapt contrastive learning to a fully supervised setting and also enable learning to happen simultaneously across multiple positives.

[0022] By enabling contrastive learning to happen simultaneously across multiple positive training examples from the same class and multiple negative training examples from other classes, the entire cluster of points belonging to the same class can be pulled together in the embedding space while the sample clusters from different classes are pushed apart simultaneously. Thus, the proposed techniques serve to increase the similarity between positive pairs of samples and decrease the similarity between negative pairs. For example, the similarity can be defined as the inner product between low-dimensional representations. The resulting representation is a very good representation that can be used for various downstream transfer tasks.

[0023] Thus, the proposed systems and methods are able to make more effective use of label or other class assignment information than systems using cross-entropy loss. In particular, using a contrastive learning approach allows learning of a representation for each class at any point within the embedding space, rather than forcing the representation of each class to align with the class value of a particular axis as is done in cross-entropy loss, thereby allowing learning of more robust inter-class representations.

[0024] Example models trained according to the proposed techniques consistently outperform cross-entropy on supervised learning tasks across different architectures and data augmentations. Specifically, as shown in example experimental results contained in U.S. Provisional Patent Application No. 63 / 013,153 with respect to ResNet-50 and ResNet-200, example models trained according to the proposed techniques outperform cross-entropy by over 1%, setting a new state-of-the-art number by 78:8% using the method of automatic augmentation data augmentation.

[0025] The proposed loss also shows a clear benefit in robustness to natural corruption on standard benchmarks in both calibration and accuracy. Furthermore, the proposed supervised contrastive loss is more stable to hyperparameter settings such as optimizer or data augmentation compared to cross-entropy. Additional aspects of the present disclosure leverage key ingredients such as large batch and normalized representations that have been shown to benefit self-supervised learning.

[0026] More specifically, example aspects of the present disclosure are directed to a new loss for supervised training that completely removes the reference distribution; instead, example implementations of the proposed system simply exploit that normalized representations from the same class are closer than representations from different classes. The proposed loss builds on the family of contrastive objective functions that have achieved superior performance in self-supervised learning in images and videos in recent years and have connections to the large body of literature on metric learning.

[0027] As the name suggests, the contrastive loss consists of two “opposing forces”: for a given anchor, the first force pulls the anchor closer to other points in the representation space, and the second force pushes the anchor further away from other points. The former group is called positive, and the latter is called negative.

[0028] One aspect of the present disclosure is that, in addition to many negative examples, many positive examples are considered for each anchor (e.g., as opposed to the convention in self-supervised contrastive learning that uses only a single positive example). In some implementations, the provided labels or other class-based designations can be used to select the positive and negative examples. FIG. 1A - B and 2A-C provide a visual explanation of the proposed loss.

[0029] Specifically, FIG. 1A and 1B differences between supervised and self-supervised contrastive losses are shown. In example implementations of the proposed supervised contrastive loss (typically as FIG. 1A shown), positive examples from one class are contrasted with negative examples from other classes (as labels are provided); this results in images from the same class (but not necessarily depicting the same exact scene or subject matter) being mapped to proximal points in the low-dimensional hypersphere. In contrast, in self-supervised contrastive losses (as FIG. 1BThe labels are not provided (as shown in the right column). Thus, positive examples are generated as data augmentations (crop, flip, color change, etc.) of the given sample, and negative examples are randomly sampled from the mini-batch. This can result in false negatives (shown in the bottom right) that can not be correctly mapped, resulting in a worse representation.

[0030] FIG. 2A -C shows the difference between cross-entropy, self-supervised contrastive loss, and supervised contrastive loss. In particular, cross-entropy loss (as shown in the left column) trains the model using labels and softmax loss, while self-supervised contrastive loss (as shown in the middle column) learns a representation about classes using contrastive loss and data augmentations. However, in some implementations, the proposed supervised contrastive loss (shown generally in the right column) has two stages; in the first stage, labels are used to select images for the contrastive loss, including multiple positive examples and one or more negative examples. In the second stage, the learned representation is frozen, and then a softmax or cross-entropy loss can be used to learn a classifier or other task-specific head (e.g., on a linear layer). This two-stage approach combines all the benefits of using labels and contrastive loss, as well as softmax or cross-entropy loss. FIG. 2A FIG. 2B FIG. 2C

[0031] As shown by example empirical results contained in U.S. Provisional Patent Application No. 63 / 013,153, the resulting loss is stable for training. As one example, an example model trained according to the proposed supervised contrastive technique achieved very good top-1 accuracy on the ImageNet dataset on ResNet-50 and ResNet-200 architectures. On ResNet-50 with automatic augmentation, an example implementation achieved a top-1 accuracy of 78.8%, which is a 1.6% improvement over cross-entropy loss with the same data augmentations and architecture. The improvement in top-1 accuracy was also accompanied by an improvement in robustness, as measured on the ImageNet-C dataset.

[0032] The systems and methods of the present disclosure provide a number of technical effects and benefits. As one example technical effect, the proposed loss allows the model to learn state-of-the-art representations, resulting in a significant improvement in top-1 accuracy and robustness compared to cross-entropy. Thus, the proposed techniques improve the performance and functionality of the computing system itself in a variety of tasks, such as image classification tasks.

[0033] ​​​As another example technical effect, the proposed loss is less sensitive to a range of hyperparameters compared to cross-entropy, which is an important practical consideration. This reduced sensitivity is due to the more natural loss formulation that pulls representations of samples from the same class closer together, rather than forcing them to be pulled towards a particular target as in cross-entropy. In a typical training system, multiple rounds of training can be required to “tune” the hyperparameters to find acceptable or optimal performance. By reducing the sensitivity of the loss and model to hyperparameters, fewer training rounds can need to be performed to tune the hyperparameters. Reducing the number of training rounds performed results in savings of computational resources such as processor usage, memory usage, network bandwidth, and the like.

[0034] As another example technical effect, the gradient of the proposed loss function encourages learning from hard positives and hard negatives, as shown in the analysis herein. In certain existing systems, explicit and computationally expensive methods are typically performed to identify specific hard negatives. One example method of this nature is “hard negative mining.” These methods improve performance, but require the expenditure of significant computational resources. By naturally encouraging learning from hard positives and hard negatives, the explicit attempt to identify hard examples (e.g., hard negatives) can be avoided, thereby saving computational resources such as processor usage, memory usage, network bandwidth, and the like.

[0035] As yet another example technical effect, the proposed method can result in a reduction in the number of false negatives produced by a trained model. For example, by enabling learning of positives in the same class, but depicting different topics or scenes, such positives can avoid being treated as negatives. In this way, the trained model learns to provide consistent representations for all class members, rather than just for a particular topic or scene. A trained model with reduced false negatives represents an improvement in the functionality of the computing system itself.

[0036] Accordingly, aspects of the present disclosure provide an improved contrastive learning loss that outperforms cross-entropy on benchmarks of classification accuracy and robustness. Moreover, example experiments show that this loss is less sensitive to hyperparameter variations, which can be a useful practical consideration. The loss function provides a natural connection between fully unsupervised training on one end and fully supervised training on the other. This opens the possibility of semi-supervised learning applications that can leverage the benefits of a single loss that can change behavior smoothly based on the availability of labeled data.

[0037] Example systems and methods for supervised contrastive learning

[0038] This section first reviews the contrastive learning loss for self-supervised representation learning. Next, it shows how to modify this loss to fit fully supervised learning while preserving properties important to self-supervised methods.

[0039] Example representation learning framework

[0040] One example representation learning framework is shown in FIG. 3 and is structurally similar to the framework typically used for self-supervised contrastive learning. As shown in FIG. 3 One example representation learning framework includes the following components. (See also FIG. 1A - B and 2A-C, showing the distinction between supervised and self-supervised scenarios).

[0041] An optional data augmentation module 203, A(·), which transforms the input image 202, x, into a randomly augmented image 212, In some implementations, for each input image 202, the system can be implemented to generate two or more randomly augmented images 212, each of which represents a different view of the data, and thus contains some subset of the information in the original input image. However, other implementations can be configured to generate only a single augmented image. Expressions contained herein that refer to a two-fold quantity of images included in a class (e.g., example loss functions) are stylized in this way to explain example embodiments in which two augmented images are generated per input image. However, this stylization can be modified to explain different quantities of augmented images generated per input image.

[0042] Referring to the data augmentation module 203, a first stage of augmentation can include, as one example, applying random cropping to the image, followed by resizing it back to the original resolution of the image. As another example, depending on the finding that self-supervised contrastive loss requires significantly different data augmentation than cross-entropy loss, a second stage can include some or all of the different augmentations described in any of:

[0043] AutoAugment: Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. AutoAugment: Learning augmentation strategies from data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 113-123, 2019.

[0044] RandAugment: Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical data augmentation with no separate search. arXiv preprint arXiv: 1909.13719, 2019

[0045] SimAugment: Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv: 2002.05709, 2020 Sequential application of random color distortion and Gaussian blur, with additional sparse image cropping probabilistically added to the end of the sequence.

[0046] More generally, the data augmentation module 203 can perform any different combination of one or more augmentations. Moreover, some example embodiments of the present disclosure do not perform augmentations of the input images 202, but simply use the input images 202 without augmentations.

[0047] Referring again to FIG. 3 , the framework can include a base encoder neural network 204, E(·), that maps augmented images 212 to embedding representation vectors 214,

[0048] In some example embodiments that generate two augmented images per input image of the framework, the two augmented images per input image can be individually input to the same encoder, resulting in a pair of representation vectors. More generally, the same encoder network 204 is typically used to generate representations for all images in a training batch.

[0049] Two commonly used encoder architectures for the encoder network 204 include ResNet-50 and ResNet-200, where, for example, the activations of the final pooling layer (D E = 2048) can be used as the representation vectors 214. In some example embodiments, this representation layer can be normalized to the unit hypersphere in Such normalization can improve performance, consistent with other papers that have used metric losses. The new supervised contrastive loss is able to train both architectures with high accuracy without special hyperparameter tuning. In fact, example experimental results show that the supervised contrastive loss is less sensitive to small changes in hyperparameters, such as the choice of optimizer or data augmentation.

[0050] The framework can also include a projection network 206, P(·), that projects the normalized representation vector r 214 to a projected representation 216 suitable for computing the contrastive loss One example projection network 216 can be a multilayer perceptron with a single hidden layer of size 2048 and an output vector of size D P = 128. In some example implementations, this vector can again be normalized to lie on the unit hypersphere, which enables the use of inner products to measure distance in the projection space. Normalization adds mathematical structure to the form of the gradients, which makes training overall superior to the case where normalization is not performed.

[0051] A loss function for the input image 202 can be evaluated based on the projected representation 216 (e.g., via comparison to other projected representations generated for other input images, such as other positive and negative training examples).

[0052] In some implementations, the projection network 206 is only used for training the supervised contrastive loss. After training is complete, the network can be discarded and replaced with a task-specific head (e.g., a single linear layer). The embedding representation 214 from the encoder 204 gives improved performance on downstream tasks compared to the projected representation 216 from the projection network 206. Thus, in some implementations, the proposed inference-time models can contain exactly the same number of parameters as their cross-entropy equivalents.

[0053] As one example, FIG. 4 depicts an example use of a base encoder neural network 204 after the base encoder neural network 204 has been trained in the example framework shown. FIG. 3 Particularly, the base encoder neural network 204 has been extracted and an additional task-specific model 250 has been appended to the base encoder neural network 204. The task-specific model 250 can be any kind of model including a linear model or a non-linear model such as a neural network, for example.

[0054] The task-specific model 250 and / or the base encoder neural network 204 can be additionally trained (e.g., “fine-tuned”) on additional training data (e.g., which can be task-specific data). The additional training can be supervised learning training, for example.

[0055] After fine-tuning, the base encoder neural network 204 can be provided with additional input 252, which the base encoder neural network 204 can produce an embedding representation 254. The task-specific model 250 can receive and process the embedding representation 254 to generate a task-specific prediction 256. The task-specific prediction 256 can be a classification prediction; a detection prediction; an identification prediction; a regression prediction; a segmentation prediction; and / or other prediction tasks, as examples.

[0056] Further, in some implementations, the embedded representations 254 can be used directly for tasks such as similarity search or retrieval without further training / fine-tuning (e.g., without additional task-specific models 250).

[0057] While for ease of explanation the present disclosure focuses on data examples from the image domain, the framework is extendable to data examples from different domains, including text and / or audio domains. Example types of images that can be used include video frames, LiDAR point clouds, computed tomography, X-ray images, hyperspectral images, and / or various other forms of images.

[0058] Example contrastive loss: self-supervised and supervised

[0059] Example implementations of the present disclosure provide a contrastive loss function that allows for the efficient combination of labeled data while preserving the beneficial properties of contrastive loss that have been critical to the success of self-supervised representation learning. Similar to self-supervised contrastive learning, example implementations of the present disclosure can generate mini-batches by randomly sampling data. As one example, for a set of N randomly sampled image / label pairs, {x k ,y k} k＝1...N , the corresponding mini-batch for training can include 2N pairs, where and are two random augmentations of x k (k = 1...N) and . Again, some other number of augmented images can be generated for each input image; two are provided as examples.

[0060] Example self-supervised contrastive loss

[0061] Within the mini-batch, let i e {1...2N} be the index of an arbitrary augmented image, and let j(i) be the index of another augmented image that originated from the same source image. In self-supervised contrastive learning, the loss has the following form.

[0062]

[0063]

[0064] where 1 B e {0,1} is an indicator function that returns 1 iff B evaluates to true, and This is a temperature parameter. Within the context of Eq.2, index i is called the anchor, index j is called the positive example, and the other 2(N-1) indices (k = 1...2N) are... This is called a negative example. i ·z j(i) This is an example similarity measure that computes a normalized vector z in a 128-dimensional space. i and z j(i) The inner product (dot product) between them. Note that for each anchor i, there is 1 positive pair and 2N-2 negative pairs. The denominator has a total of 2N-1 terms (positive and negative examples).

[0065] It is insightful to consider minimizing the effect of Eq.1 on the encoder. During training, for any i, the encoder is adjusted to maximize the numerator of the log parameter in Eq.2 while minimizing its denominator. This term exp(z) i ·z j(i) The constraint that appears in both the numerator and denominator ensures that the log parameter is no higher than 1, and because Eq.1 sums over all index pairs ((i,j) and (j,i)), the encoder is restricted to minimizing the denominator or maximizing the numerator without performing the other operation. As a result, the encoder learns to map similar views to adjacent representations, while mapping dissimilar views to non-adjacent views, but only based on a single positive example per evaluation.

[0066] Example supervised contrastive loss

[0067] For supervised learning, the contrastive loss in Eq.2 cannot handle cases where more than one sample is identified as belonging to the same class. To generalize the loss to handle any number of positive examples belonging to the same class, the following novel loss function is proposed:

[0068]

[0069]

[0070] in It is a mini-batch with the same label as anchor i. The total number of images. Again, the concept of 2N is provided to explain the example setting where each training image generates two augmented images, but other numbers can also be used. The example loss given in equations (3) and (4) has important properties that are very suitable for supervised learning:

[0071] generalizes to any number of positives. One significant structural change of Eq. 4 compared to Eq. 2 is that now, for any anchor, all positives in the minibatch (i.e., one based on the augmentation plus any of the remaining 2(N-1) entries from the same class) contribute to the numerator. For minibatches that are large relative to the number of classes, multiple additional terms will arise (on average, where C is the number of classes). This loss encourages the encoder to give a tight alignment of representations for all entries from the same class in each instance of Eq. 4, leading to a more robust clustering of the representation space than the one generated from Eq. 2.

[0072] The power of contrast increases with the increase of positives and / or negatives. The example supervised contrastive loss given in Eq. 4 leverages the principle that the ability to distinguish signal from noise (negatives) is improved by adding more examples of positives and / or more examples of negatives. This property has been shown to be important for representation learning via self-supervised contrastive learning, with many studies showing that performance increases with the increase of the number of negatives.

[0073] The example supervised contrastive loss in Eq. 4 preserves this structure: adding more negatives in the denominator increases the contrast for positives; similarly, adding more positives in the numerator increases the contrast for negatives. More positives allow the model to better capture intra-class variation. More negatives enable the model to capture inter-class variation.

[0074] Discussion of example supervised contrastive loss gradient properties

[0075] This subsection now provides further motivation for the form of the example supervised contrastive loss in Eq. 4 by showing that its gradient has a structure that naturally leads to learning to focus more on hard positives and negatives (i.e., those for which continuing to contrast the anchor with them greatly benefits the encoder) rather than weak positives and negatives (i.e., those for which continuing to contrast the anchor with them only weakly benefits the encoder). From this, it can be seen that this loss is efficient in its training. Other contrastive losses, such as the triplet loss, typically use computationally expensive hard negative mining techniques to improve training efficiency. As a byproduct of this analysis, it is shown that adding a normalization layer at the end of the projection network is beneficial, as its presence allows the gradient to have this structure.

[0076] If we assume that w represents the output of the projection network immediately before normalization (i.e., z = w / ||w||), then the gradient of Eq. 4 with respect to w has the following form:

[0077]

[0078] where:

[0079]

[0080]

[0081] in:

[0082]

[0083] It is the temperature-scaled softmax distribution of the inner product of the representation of anchor i. Each component has a probability. Eq. 6 typically includes the contribution from positive examples in the mini-batch, while Eq. 7 includes the contribution from negative examples. It is now shown that simple positive and negative examples (i.e., those whose continued anchoring comparison only slightly benefits the encoder) have small gradient contributions, while difficult positive and negative examples (i.e., those whose continued anchoring comparison greatly benefits the encoder) have large gradient contributions. For simple positive examples, z... i ·z j ≈1, therefore P ij It is large. Therefore (see Eq. 6):

[0084]

[0085] However, for difficult correct examples, z i ·z j ≈0 and P ij It is of medium quality, therefore:

[0086]

[0087] Therefore, for weak positive examples, where the reward for further comparative effort diminishes, the outcome is different. The contribution is small, but for difficult positive examples, further comparative efforts are still needed, resulting in a large contribution. For weak negative examples (z... i ·z k ≈-1) and difficult-to-bear examples (z) i ·z k ≈0), according to Eq.7 ||(z) k -(z i ·z k )·z i )||·P ik Similar calculations yield similar conclusions: the gradient contribution is large for hard negative examples and small for weak negative examples. This plays a crucial role in ensuring that the gradient is large for both hard positive and hard negative examples. The structure only appears when a normalization layer is added at the end of the projection network.

[0088] Example devices and systems

[0089] FIG. 5A A block diagram of an example computing system 100 in accordance with example embodiments of the present disclosure is depicted. The system 100 includes a user computing device 102, a server computing system 130, and a training computing system 150 communicatively coupled over a network 180.

[0090] The user computing device 102 can be any type of computing device such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0091] The user computing device 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as

[0092] In some implementations, the user computing device 102 can store or include one or more machine learning models 120. For example, the machine learning models 120 can be or can otherwise include various machine learning models such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear models and / or linear models. The neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. See, e.g., FIG. 3 and 4 Example machine learning models 120 are discussed.

[0093] In some implementations, the one or more machine learning models 120 can be received from the server computing system 130 over the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing device 102 can implement multiple parallel instances of a single machine learning model 120 (e.g., to perform parallel predictions across multiple input instances).

[0094] Additionally or alternatively, one or more machine-learned models 140 can be included in a server computing system 130 that communicates with the user computing device 102 according to a client-server relationship, or stored and implemented by the server computing system 130. For example, the machine-learned models 140 can be implemented by the server computing system 140 as part of a web service (e.g., a prediction service). Thus, one or more models 120 can be stored and implemented at the user computing device 102, and / or one or more models 140 can be stored and implemented at the server computing system 130.

[0095] The user computing device 102 can also include one or more user input components 122 that receive user input. For example, the user input components 122 can be a touch-sensitive component (e.g., a touch-sensitive display screen or trackpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can be used to implement a virtual keyboard. Other example user input components include a microphone, a conventional keyboard, or other mechanisms that a user can use to provide user input.

[0096] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as

[0097] In some implementations, the server computing system 130 includes or is implemented by one or more server computing devices. In instances in which the server computing system 130 includes multiple server computing devices, such server computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0098] As noted above, the server computing system 130 can store or otherwise include one or more machine-learned models 140. For example, the models 140 can be or can include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. See FIG. 3 and 4 Example models 140 are discussed.

[0099] User computing device 102 and / or server computing system 130 can train models 120 and / or 140 via interaction with training computing system 150, which is communicatively coupled to network 180. Training computing system 150 may be separate from server computing system 130, or it may be part of server computing system 130.

[0100] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 executed by the processor 152 to cause the training computing system 150 to perform operations. In some embodiments, the training computing system 150 includes or is implemented by one or more server computing devices.

[0101] Training computing system 150 may include model trainer 160, which uses various training or learning techniques (such as, for example, backpropagation of error) to train machine learning models 120 and / or 140 stored at user computing device 102 and / or server computing system 130 (e.g., according to...). FIG. 3 (See the framework shown). For example, a loss function can be backpropagated through (multiple) models to update one or more parameters of (multiple) models (e.g., based on the gradient of the loss function). Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions, such as the loss functions described in Equations 1-4 in this paper. Gradient descent techniques can be used to iteratively update parameters over multiple training iterations.

[0102] In some implementations, backpropagation of the execution error may include performing truncated backpropagation over time. The model trainer 160 may perform various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the trained model.

[0103] Specifically, the model trainer 160 can train machine learning models 120 and / or 140 based on a set of training data 162. The training data 162 may include, for example, data in different forms, such as images, audio samples, text, etc. Example types of images that can be used include video frames, LiDAR point clouds, X-ray images, computed tomography scans, hyperspectral images, and / or various other forms of images.

[0104] In some implementations, the training examples can be provided by the user computing device 102 if the user has provided consent. Thus, in such implementations, the model 120 provided to the user computing device 102 can be trained by the training computing system 150 on user-specific data received from the user computing device 102. In some instances, this process can be referred to as personalizing the model.

[0105] The model trainer 160 includes computer logic in the form of software, hardware, or a combination of software and hardware that provides desired functionality to the computing system 150. The model trainer 160 can be implemented in hardware, firmware, and / or software to control a general purpose processor, which can be a dedicated processor or well as a shared processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions that are stored in tangible computer-readable storage media such as RAM hard disks or optical or magnetic media.

[0106] The network 180 can be any type of communications network, such as a local area network (e.g., intranet), wide area network (e.g., Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communications over the network 180 can be carried out using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL), via any type of wired and / or wireless connection.

[0107] FIG. 5A An example computing system that can be used to implement the present disclosure is shown. Other computing systems can also be used. For example, in some implementations, the user computing device 102 can include the model trainer 160 and the training data set 162. In such implementations, the model 120 can be trained and used locally at the user computing device 102. In some such implementations, the user computing device 102 can implement the model trainer 160 to personalize the model 120 based on user-specific data.

[0108] FIG. 5B A block diagram of an example computing device 10 that is performed in accordance with example embodiments of the present disclosure is depicted. The computing device 10 can be a user computing device or a server computing device.

[0109] The computing device 10 includes a plurality of applications (e.g., applications 1 through N). Each application contains its own machine learning library and machine learning model(s). For example, each application can include a machine learning model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like.

[0110] As FIG. 5B illustrated, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0111] FIG. 5C A block diagram of an example computing device 50 that is executed in accordance with example embodiments of the present disclosure is depicted. Computing device 50 can be a user computing device or a server computing device.

[0112] Computing device 50 includes a number of applications (e.g., applications 1 through N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a public API that is common across all applications).

[0113] The central intelligence layer includes a number of machine learning models. For example, as FIG. 5C illustrated, a respective machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included in, or implemented by, an operating system of computing device 50.

[0114] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized data repository of computing device 50. As FIG. 5C illustrated, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0115] Additional disclosure

[0116] The technology discussed herein makes reference to servers, databases, software applications and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0117] While the subject matter has been described in detail with respect to various specific embodiments of it, it is not intended that the application be limited to such specific embodiments. Those skilled in the art will recognize that many modifications, changes and variations are possible to the specific embodiments described. It is therefore contemplated that the subject matter described herein be considered in all its novel embodiments and obvious modifications. For example, the features of one embodiment can be used with another embodiment to produce a yet another embodiment. Thus, the foregoing description is intended to be illustrative rather than limiting.

Claims

1. A computational system for performing supervised contrastive learning of visual representations, the computational system comprising: One or more processors; and One or more non-transitory computer-readable media, which collectively store: A basic encoder neural network is configured to process an input image to generate an embedded representation of the input image; A projection head neural network is configured to process the embedding representation of the input image to generate a projection representation of the input image; and Instructions, when executed by the one or more processors, cause the computing system to perform operations, the operations including: Obtain an anchored image associated with a first class among a plurality of classes, a plurality of positive images associated with the first class, and a plurality of negative images associated with one or more other classes among the plurality of classes, wherein the one or more other classes are different from the first class; The anchored image is processed by the basic encoder neural network to obtain the anchored embedding representation of the anchored image, the plurality of positive images are processed to obtain a plurality of positive embedding representations respectively, and the one or more negative images are processed to obtain one or more negative embedding representations respectively; The projector neural network processes the anchoring embedding representation to obtain the anchoring projection representation of the anchored image, processes the plurality of positive embedding representations to obtain a plurality of positive projection representations respectively, and processes the one or more negative embedding representations to obtain one or more negative projection representations respectively; An evaluation loss function is used to assess a similarity measure between the anchored projection representation and each of the plurality of orthogonal projection representations and each of the one or more negative projection representations. The loss function includes a first summation of the contrastive loss term across all images belonging to the same class as the anchor image in the training batches, and The contrast loss term comprises an exponent of the similarity measure between the anchored projection representation and the orthographic projection representation of the image evaluated under the first summation, divided by the logarithm of the second summation of the exponent of the similarity measure between the anchored projection representation and the negative projection representation of the image evaluated under the second summation for all other images in the training batch; and The loss function is used to modify one or more values ​​of one or more parameters of at least the basic encoder neural network.

2. The computing system of claim 1, wherein the anchoring image and at least one of the one or more positive images depict different objects belonging to the same first class among the plurality of classes.

3. The computing system of claim 1, wherein the plurality of positive images comprises all images associated with the first class contained in the training batch, and wherein one or more negative images comprises all images not associated with any of the plurality of classes other than the first class contained in the training batch.

4. The computing system of claim 1, wherein the operation further comprises enhancing each of the anchor image, the plurality of positive images, and the one or more negative images respectively before processing each of the anchor image, the plurality of positive images, and the one or more negative images with the basic encoder neural network.

5. The computing system according to claim 1, wherein the projection head neural network includes a normalization layer, the normalization layer normalizing the projection representation of the input image.

6. The computing system of claim 1, wherein the similarity measure includes an inner product.

7. The computing system of claim 1, wherein the loss function further comprises a normalization term comprising negative one divided by twice the number of images included in the first class of the anchored images minus one.

8. The computing system of claim 1, wherein after modifying one or more values ​​of one or more parameters of at least the basic encoder neural network based at least in part on the loss function, the operation further comprises: Add a classification head to the basic encoder neural network; as well as The classification head is fine-tuned based on a set of supervised training data.

9. The computing system of claim 1, wherein after modifying one or more values ​​of one or more parameters of at least the basic encoder neural network based at least in part on the loss function, the operation further comprises: Provide additional input to the basic encoder neural network; The additional embedding representation that receives the additional input is used as the output of the basic encoder neural network; as well as The prediction of the additional input is generated at least in part based on the additional embedding representation.

10. The computing system according to claim 9, wherein the prediction includes classification prediction, detection prediction, recognition prediction, regression prediction, segmentation prediction, or similarity search prediction.

11. The computing system of claim 1, wherein the anchored image comprises an X-ray image.

12. The computing system of claim 1, wherein the anchored image comprises a set of LiDAR data.

13. The computing system of claim 1, wherein the anchored image includes video.

14. The computing system according to claim 1, wherein, The loss function includes in, This indicates that the training batch contains images with the same label as the anchored image i. The total number of images, N represents the number of randomly sampled image / label pairs in the training batch, j represents the index of the positive image in the training batch, and k represents the index of the negative image in the training batch. The label for image j, z i z represents the anchored projection representation of the anchored image i. j Let z be the projection representation of image j. k Let τ represent the projection of image k, and let τ be the temperature parameter.

15. A computer-implemented method comprising performing some or all of the operations of any one of claims 1 to 13 by a computing system including one or more computing devices.

16. One or more non-transitory computer-readable media, collectively storing at least one basic encoder neural network, said basic encoder neural network having been trained by performing some or all of the operations of any one of claims 1-14.

Images