Self-supervised representation learning using bootstrap latent representations

Abstract

A computer-implemented method for training a neural network. The method includes: processing a first transformed view of training data items (e.g., images) using a target neural network to generate a target output; processing a second transformed view of the training data items (e.g., images) using an online neural network to generate a prediction of the target output; updating the parameters of the online neural network to minimize the error between the prediction and the target output; and updating the parameters of the target neural network based on the parameters of the online neural network. This method can efficiently train an encoder neural network without using labeled training data items and without using contrastive loss, i.e., it does not require "negative samples" that include transformed views of different data items.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 035,583, filed June 5, 2020, and U.S. Provisional Application No. 63 / 039,343, filed June 15, 2020. The disclosure of the earlier application is considered part of the disclosure of this application and is incorporated herein by reference. Technical Field

[0003] This manual pertains to image processing using machine learning models. Background Technology

[0004] A neural network is a machine learning model that uses one or more non-linear units to predict the output from a received input. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as the input to the next layer in the network, i.e., the next hidden layer or output layer. Each layer of the network generates its output from the received input based on the current values ​​of its corresponding parameter set. Summary of the Invention

[0005] This specification describes a system implemented as a computer program located on one or more computers at one or more locations, configured to learn representations of data items such as images through a self-supervised learning process.

[0006] A first aspect of this disclosure provides a computer-implemented method for training a neural network. The method includes: processing a first transformed view of training data items (e.g., images) with a target neural network to generate a target output; processing a second transformed view of the training data items (e.g., images) with an online neural network to generate a prediction of the target output; updating the parameters of the online neural network to minimize the error between the prediction of the target output and the target output; and updating the parameters of the target neural network based on the parameters of the online neural network.

[0007] The term "transformed view" refers to a transformed version of training data items, such as images, and is used to distinguish training data items that have undergone transformation (e.g., image transformation) from the original (untransformed) training data items.

[0008] The parameters of a neural network can include the weights of the neural network, and updating the parameters of a neural network can include adjusting the values ​​of the weights.

[0009] Updating one or more parameters of the target neural network can include updating one or more parameters of the target neural network using a moving average of the parameters of the online neural network. The moving average can be an exponential moving average.

[0010] Updating one or more parameters of the target neural network may involve determining the updated values ​​of one or more parameters of the target neural network according to ξ←τξ+(1-τ)θ, where ξ represents the parameters of the target neural network, θ represents the parameters of the online neural network, and τ is the decay rate. The decay rate can be a value between zero and one.

[0011] The online neural network and the target neural network may each include a corresponding encoder neural network. Thus, the method may also include operations performed by each encoder neural network, including receiving a transformed view of training data items such as images, and processing the transformed view of the training data items such as images to generate a representation of the training data items.

[0012] Optionally, each encoder neural network may include a residual neural network, i.e., a neural network with one or more residual or skip connections around one or more layers of the neural network.

[0013] Once an online neural network has been trained, its encoder neural network can be used to generate representations of any suitable input data item, such as a representation of the pixels of an input image. The resulting image representation can then be used by other downstream tasks.

[0014] For example, image classification operations can be performed on an image representation. As another example, image segmentation operations can be performed on an image representation. Optionally or additionally, other image processing tasks can be performed.

[0015] The online neural network and the target neural network may each include a corresponding projective neural network. The method may also include operations performed by each projective neural network, said operations including receiving a representation of training data items such as images, and processing the representation of training data items such as images to reduce the dimensionality of the representation. Optionally, each projective neural network may include a multilayer perceptron.

[0016] The use of projective neural networks is optional because online neural networks can directly predict representations of training data items, such as images, generated by the target neural network (rather than predicting projections of those representations). However, using projective neural networks can provide improved performance.

[0017] The online neural network may include a predictive neural network. The method may also include: operations performed by the predictive neural network, said operations including receiving representations of training data items, such as images, and processing the representations of the training data items using a regression model embodied by the parameters of the predictive neural network to generate a prediction of a target output. Optionally, the predictive neural network may include a multilayer perceptron. The use of a predictive neural network is not required, but can help improve the stability of training. In some implementations, the target neural network does not include a predictive neural network.

[0018] Therefore, in the implementation, the online neural network and the target neural network can have the same neural network structure, but with different parameter values, except that in one of the neural networks, specifically one or more additional processing stages in the online neural network.

[0019] The target neural network can have a stopping gradient (“sg”). The stopping gradient prevents backpropagation into the target neural network, so that the parameters of the target neural network are not updated when the error is minimized.

[0020] The method may further include initializing the parameters of the online neural network and / or the target neural network to random values.

[0021] The method may further include applying a first data item transformation (e.g., an image transformation) to a training data item (e.g., an image) to generate a first transformation view of the training data item, and applying a second data item transformation (e.g., an image transformation) to the training data item to generate a second transformation view of the training data item (e.g., an image). The second data item transformation (e.g., an image transformation) is different from the first data item transformation (e.g., an image transformation).

[0022] When the training data includes training images, the first and second image transformations can include any combination of one or more of the following: random cropping; flipping along the horizontal and / or vertical axis; color dithering; conversion to grayscale; Gaussian blur; or solarization. Alternatively or additionally, other image transformations may be used. By using a transformed view of the training image, the online neural network learns a target network representation based on another transformed view of the same training image.

[0023] While this disclosure focuses on examples in which two different transformed views of a training data item (e.g., an image) are input into an online neural network and a target neural network, in other examples, only a transformed view of the training data item (e.g., an image) may be input into one of the online neural network and the target neural network. In these examples, the original (untransformed) training data item, such as an image, is input into the other of the online neural network and the target neural network.

[0024] Updating one or more parameters of an online neural network can involve using a machine learning optimizer, such as one based on stochastic gradient descent, to minimize the aforementioned error. Updating one or more parameters of an online neural network can include normalizing the prediction of the target output; and minimizing the squared error between the normalized prediction of the target output and the target output.

[0025] The following operations can be performed iteratively for each training data item in a batch that includes multiple training data items: processing a first transformed view of the training data item with a target neural network, processing a second transformed view of the training data item with an online neural network, updating one or more parameters of the online neural network, and updating one or more parameters of the target neural network. The parameters of both the online neural network and the target neural network can be updated after each training data item in the batch has been processed.

[0026] Another aspect of this disclosure provides a computer-implemented method for processing data items (e.g., processing images). The method includes providing an input data item (e.g., an image) to an online neural network (i.e., to a portion of a trained online neural network), the online neural network having been trained according to this disclosure; processing the input data item (e.g., the image) through the online neural network (i.e., using a portion of the trained online neural network); outputting a representation of the input data item from the online neural network (i.e., from a portion of the trained online neural network); and processing the representation of the input data item (e.g., the image).

[0027] Online neural networks may include residual neural networks, which are configured to generate representations of input data items (e.g., images).

[0028] When the input data includes images, processing the representation of the input image may include classifying the input image using the representation. Alternatively, processing the representation of the input image may include segmenting the input image using the representation. Alternatively, other image processing tasks may be performed.

[0029] Another aspect of this disclosure provides a system comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform any of the methods disclosed herein.

[0030] Another aspect of this disclosure provides one or more computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform any of the methods disclosed herein.

[0031] The subject matter described in this specification can be implemented in specific embodiments to achieve one or more of the following advantages.

[0032] Online neural networks are trained through a self-supervised learning process, whereby the online neural network learns to represent images from raw, unlabeled training data. Unlike self-supervised learning methods that use a contrastive loss function, the technique disclosed in this paper avoids the need for negative pairs (i.e., pairs of different training examples, such as images) of training examples. This, in turn, avoids the difficulties associated with using negative pairs, such as the need for a large memory bank consisting of sample representations from the training dataset, and the need for careful selection of negative pairs. However, when applied to tasks such as image classification, it has been found that image representations generated by online neural networks trained as disclosed in this paper achieve classification accuracy comparable to training using labeled examples and outperform some contrastive methods.

[0033] The technique disclosed in this paper also avoids the high computational requirements of some methods, and therefore is computationally efficient.

[0034] The input to both online neural networks and target neural networks can include, for example, any suitable type of image data, such as video image data. Image data can include color or monochrome pixel value data. This image data can be captured from image sensors such as cameras or LiDAR sensors.

[0035] The image representations generated by the online neural networks disclosed in this paper can be used for a wide variety of image processing tasks. For example, the image representations can be used for image classification, whereby a classification system outputs one or more class labels for a given input image representation. Continuing this example, the classification system can process the image representation and output a score for each of a set of object classes, each score representing an estimated probability that the image contains an object belonging to that class.

[0036] As another example of an image processing task, image representation can be used for image segmentation, whereby the segmentation system uses the image representation to label each pixel of the input image as belonging to one of several different categories. An example use case for image segmentation is in object detection, where the segmentation system labels pixels in the input image based on the type of object each pixel represents.

[0037] Object detection can be used as input for mechanical agents, such as robots or vehicles, that can operate in real-world environments. Detected objects can be, for example, obstacles (e.g., people, other mechanical media, walls) and / or paths (e.g., roads or other surfaces on which the mechanical agent can move). The detected objects can then be used by the mechanical agent's control system to determine how to perform mechanical tasks, such as controlling the mechanical agent's direction of movement and / or speed.

[0038] Another example use case for image segmentation is segmenting medical images, where the segmentation system labels pixels in the input medical image based on whether they indicate a region of a human or animal body with a specific medical condition. These are merely non-limiting examples of image segmentation; many other practical applications of image segmentation exist.

[0039] Throughout the specification, using neural networks to process images refers to using neural networks to process the intensity values ​​associated with the pixels of an image.

[0040] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Further features, aspects, and advantages of this subject matter will become apparent from the specification, drawings, and claims. Attached Figure Description

[0041] Figure 1 An example neural network system including an encoder neural network is shown.

[0042] Figure 2 The training method is shown. Figure 1 Example process of the system.

[0043] Figure 3 An example process using a trained encoder neural network is shown.

[0044] Figure 4a and 4b An example neural network system including a trained encoder neural network is shown, along with an example process for training and using the system.

[0045] Figure 5 The performance of the trained encoder neural network on the image classification task is shown.

[0046] In the accompanying drawings, the same reference numerals denote the same elements. Detailed Implementation

[0047] This specification describes a method for training an encoder neural network to generate representations of data items without using labeled training data items or contrastive loss. The method uses contrastive loss to learn to distinguish between positive examples (i.e., two transformed or "enhanced" views of the same data item) and negative examples that include enhanced transformed or "enhanced" views of different data items. The described methods are implemented only with respect to positive examples of their object function, without needing negative examples, which can be difficult to generate. They can also achieve performance close to that of neural networks with labeled training data.

[0048] In some of the described examples, the data items include images, but generally any type of data item can be processed. Examples of different data items will be described later. This method can be used to train an encoder neural network to perform any type of task involving processing the same type of data items used in training, such as images.

[0049] In the case of image data items, including video data items, the task can include any kind of image processing or vision task, such as image classification or scene recognition, image segmentation (e.g., semantic segmentation), object localization or detection, or depth estimation. When performing such tasks, the input can include pixels of an image or input derived from pixels of an image. For image classification or scene recognition tasks, the output can include a classification output that provides a score for each of a plurality of image or scene categories, such as an estimated probability that an input data item or an object or element of an input data item, or an action within a video data item, belongs to a certain category. For image segmentation tasks, for each pixel, the output can include an assigned segmentation category or the probability that the pixel belongs to a segmentation category, such as belonging to an object or action represented in an image or video. For object localization or detection tasks, the output can include data that defines the coordinates of bounding boxes or regions representing one or more objects in an image. For depth estimation tasks, the output can include an estimated depth value for each pixel such that the output pixels define a (3D) depth map of the image. Such tasks may also be helpful for higher-level tasks, such as object tracking across video frames; or gesture recognition, i.e., the recognition of gestures performed by entities depicted in a video.

[0050] Another example of an image processing task could be an image keypoint detection task, where the output includes the coordinates of one or more image keypoints, such as landmarks representing objects in the image, for example, a human pose estimation task where the keypoints define the positions of body joints. Another example is an image similarity determination task, where the output could include values ​​representing the similarity between two images, for example, as part of an image search task.

[0051] This method is used to train a neural network system, with an encoder neural network forming part of that system. The rest of the system is not needed after training. The trained encoder neural network can be used to perform tasks without further training, such as freezing its parameters, or it can be further trained to perform a specific task. Generally, a trained encoder neural network can be incorporated into a larger system to perform a specific task, such as a system configured to perform image classification, image segmentation, object localization, or depth estimation tasks. Therefore, an image processing system incorporating a trained neural network is also provided.

[0052] In the case of image data items, the transformed or "enhanced" view can be a transformed view of the same image. For example, an image can be randomly cropped or distorted in different ways to generate two views. More generally, the method learns to generate representations robust to such transformations.

[0053] Figure 1 A neural network system 100 for implementing an example of this method is shown. Figure 1 The system can be implemented as one or more computer programs on one or more computers at one or more locations.

[0054] System 100 includes an online neural network 110 and a target neural network 120.

[0055] The target neural network 120 is configured to receive a first transformed view (v′) of data item 102. The inline neural network 110 is configured to receive a second transformed view (v) of the same data item 102. As described further later, the transformed view is generated by applying a corresponding transformation (T′, T) to the data item 102. In some implementations, the data item 102 includes images, such as those used herein, which include image frames from a video.

[0056] The target neural network 120 is configured to process a first transformed view (v′) of the data items to generate a target output 126 (z′). The online neural network 110 is configured to process a second transformed view (v) of the data items to generate a prediction 118 (q(z)) of the target output. For example, the online neural network 110 is trained by the training engine 130 by updating the parameters of the online neural network using a machine learning optimizer to minimize the difference or error between the prediction 118 of the target output 126 and the target output 126.

[0057] The online neural network 110 includes an encoder neural network 112, an optional projection neural network 114, and a prediction neural network 116, and is defined by a set of parameters for these neural networks. The target neural network 120 includes an encoder neural network 122 and an optional projection neural network 124, and is defined by a set of parameters for these neural networks.

[0058] In some implementations, but not essentially, the target neural network 120 has the same architecture as the online neural network 110 (except for the prediction neural network), but with different parameters (weights). That is, the encoder neural networks 112 and 122 can be the same neural networks but with different parameters; and similarly, the projection neural networks 114 and 124 can be the same neural networks but with different parameters.

[0059] Encoder neural networks 112 and 122 each receive a transformed view of data item 102 and process it to generate a representation of their respective transformed view of data item 102, i.e., a high-dimensional feature vector. This can be received and processed by subsequent corresponding projection neural networks 114 and 124 to generate, respectively, a reduced-dimensional representation of their respective transformed views of data item 102, i.e., a reduced-dimensional feature vector z, z′. The reduced-dimensional representation from projection neural network 124 provides the target output 126.

[0060] Predictive neural network 116 receives a reduced-dimensional representation (or a representation from encoder neural network 112) from projection neural network 114 and processes it to generate prediction 118. In one implementation, prediction 118 includes a vector having the same dimensions as the target output 126. The input to predictive neural network 116 may include feature vector representations of data items, which are identical to the feature vector representations of the target output 126 except for their values. The target output 126 may be the output of projection neural network 124 as described above, or, in some implementations, a representation from encoder neural network 122.

[0061] The encoder neural networks 112 and 122 can have any architecture suitable for encoding data item 102. The training method described herein results in a trained encoder neural network 112. The encoder neural network 112 is trained to generate representations of the input data items and can be any neural network configured to receive one of the data items as input and generate a feature vector representation of the received data item.

[0062] For example, when the data items include images or videos, encoder neural networks 112, 122 can each include any type of image or video coding neural network configured to generate representations of the input image or video, such as feature vector representations. Such image or video coding neural networks can include one or more convolutional neural network layers, or can have any other architecture suitable for image or vision processing. In an implementation, such image or video encoder neural networks 112, 122 can each include a residual neural network, i.e., a neural network with one or more residual or skipped connections. As an illustrative example only, a convolutional neural network with one or more residual or skipped connections, such as one based on ResNet (He et al., arXiv:1512.03385), can be used; however, a convolutional neural network is not necessarily used.

[0063] Typically, encoder neural networks 112, 122 are configured to process data items of the type to which encoder neural network 112 is to be trained, such as images, videos, audio, text, or others.

[0064] The projected neural networks 114 and 124 can be any neural network configured to reduce the dimensionality of the feature representation. For example, each can include a multilayer perceptron with an output space smaller than the input space, such as including one or more fully connected layers, each optionally followed by a batch normalization layer (Ioffe and Szegedy, arXiv:1502.03167), followed by an optional fully connected linear output layer.

[0065] As an example only, for a 224×224 pixel image, the feature vector representation generated by each encoder neural network can have a dimension of 4096, and the dimension-reduced representation can have a dimension of 256.

[0066] In this implementation, the system can train the encoder neural network 112 to ignore transformations applied to the data items. Taking image data items as an example, if the transformations applied to the training image include color changes, the feature representation can learn to ignore color. Projecting the feature representation onto a dimensionality-reduced representation can help preserve useful information in the feature representation.

[0067] In one implementation, the predictive neural network 116 can be a relatively small neural network that learns to predict the target output, for example, predicting a dimensionality-reduced representation from the projective neural network 124 based on a dimensionality-reduced representation from the projective neural network 114. Therefore, the parameters of the predictive neural network 116 reflect a regression model. In one example implementation, the predictive neural network 116 includes another multilayer perceptron.

[0068] Alternatively, the output vectors of each of the target output 126 and the prediction 118 can be normalized, for example, using the L2 norm. For instance, the target output 126, z′, can be normalized to... Predicting 118, q(z), can be normalized to Where q(·) represents the prediction neural network 116, and z is the dimensionality reduction representation from the projection neural network 114.

[0069] The neural network system 100 also includes a training engine 130, which is configured to implement the training process of the system.

[0070] Figure 2 The training method is shown. Figure 1 The example process of the neural network system 100, specifically the encoder neural network 112. Figure 2 The process can be implemented as one or more computer programs on one or more computers at one or more locations.

[0071] The parameters of the online neural network 110 and the target neural network 120 can be initialized to random values. In step 200, training data items, such as training images, are obtained. These are then processed to obtain first and second different transformations or "enhanced" views of the data items (step 202). This may involve applying one or a series of transformations to the data items to generate each transformed view. For example, each transformed view can be obtained by applying each of a set of transformations with a predetermined probability.

[0072] For example, when the training data includes images, transformations can include one or more of the following: random cropping of images, image flipping, color dithering, color reduction, Gaussian blur, and overexposure. Random cropping can include selecting random patches of an image and then expanding them to the image's original size. Image flipping can involve applying a horizontal or vertical flip to an image. Color dithering can include changing one or more of the brightness, contrast, saturation, and hue of some or all pixels of an image through random offsets. Color reduction can include converting an image to grayscale. Gaussian blur can include applying a Gaussian blur kernel to an image; other types of kernels can be used for other types of filtering. Overexposure can include applying an exposure color transformation to an image; other color transformations can be used. Other transformations are also possible, such as rotating or cropping a portion of an image (setting the pixels of the random patch to uniform values).

[0073] For example, image transformations can include unfavorable perturbations, i.e., perturbations chosen to increase the likelihood that the encoder neural network 112 will generate incorrect representations. For example, an unfavorable attack can be performed on one of a pair of transformed views, e.g., to maximize the error between prediction 118 and target output 126, using the techniques described in Madry et al. arXiv:1706.06083.

[0074] Many different transformations can be used to obtain a transformed view. The specific type of transformation used can vary depending on, for example, the task to which the encoder neural network 112 is trained and the expected type of change between data items. Optionally, in the case of an image, the transformed view can be normalized, for example, on the color channels. Normalization can involve subtracting the mean and dividing by the standard deviation.

[0075] The first transformed view of the data item is processed by the target neural network 120 to generate the target output 126 (step 204), and the second transformed view of the data item is processed by the online neural network 110 to generate a prediction 118 of the target output (step 206). Then, the prediction error between prediction 118 and the target output 126 is determined, specifically the prediction loss based on prediction 118 and the target output 126 (step 208). The prediction loss can include any measure of the difference between prediction 118 and the target output 126, optionally normalized as described above. For example, the prediction loss or error can include (mean) squared error, negative cosine or dot product similarity, or cross-entropy loss (if the feature vector values ​​are interpreted as probabilities and normalized). For example, the prediction loss or error L can be determined as the L-norm, such as... Where ||·||2 represents the L2 norm.

[0076] In some implementations, the method determines a further prediction loss or error L′, where the first and second transformed views are swapped; that is, the first transformed view of the data item is processed by the online neural network 110, while the second transformed view of the data item is processed by the target neural network 120. The symmetric prediction loss or error L can then be determined by summing the losses. TOTAL L TOTAL =L+L′.

[0077] This process can accumulate prediction loss or error on a batch of training data items before proceeding.

[0078] In steps 210 and 212, the parameters of the online neural network 110 and the target neural network 120 are updated. However, only the parameters of the online neural network 110, not the parameters of the target neural network 120, are updated via gradient descent, i.e., via the backpropagation gradient of the predicted loss or error. This can be represented as providing a "stopping gradient" to the target neural network 120, i.e., by feeding the stopping gradient back into the target neural network 120. However, in practice, this can be achieved by training only the online neural network 110 using the predicted loss or error.

[0079] Therefore, in step 210, the parameters of the online neural network 110 are updated to minimize the prediction loss or error, while in step 212, the parameters of the target neural network 120 are updated based on the parameters of the online neural network. In some implementations, minimizing the error may involve maximizing the similarity between the prediction 118 and the target output 126, such as cosine similarity.

[0080] In this implementation, stochastic optimization steps implemented by a machine learning optimizer (such as a gradient descent-based optimizer) are used to update the parameters of the online neural network 110 to minimize the prediction loss or error. Other types of machine learning optimizers can also be used. In this implementation, the optimizer minimizes the prediction loss or error, for example, L or L0. TOTAL The updated parameters are relative only to the parameters of the online neural network 110, i.e., not relative to the parameters of the target neural network 120. For example, the updated parameters of the online neural network 110 can be updated by backpropagating the gradients of the predicted loss or error through the prediction neural network 116, the projection neural network 114 (if present), and the encoder neural network 112.

[0081] In this implementation, the parameters of the target neural network 120 are updated based on the parameters of the corresponding part of the online neural network 110, i.e., not based on the prediction loss or error. For example, the parameters of the target neural network 120 can be determined as a copy or moving average of the parameters of the online neural network 110, such as a weighted or exponential moving average. Typically, the parameters of the target neural network 120 include a delayed (and more stable) version of the parameters of the online neural network 110.

[0082] In some implementations, the parameters of the target neural network 120 can be determined using the update ξ←τξ+(1-τ)θ, where τ is the target decay rate in the range [0,1], ξ is a set of parameters for the target neural network 120, and θ is a set of parameters for the online neural network 110 other than the parameters of the prediction neural network, i.e., the parameters of the encoder neural network 112 and the projection neural network 114 (if present). As an example only, τ can be greater than 0.99 and can be increased during training.

[0083] Therefore, this method is implemented using bootstrapping, since the updated (i.e., partially trained) online neural network 110 is used to update the target neural network 120 to generate a new target output for further training of the online neural network 110. Training the online neural network 110 with the new target improves the representation from the encoder neural network 112.

[0084] For example, the representation of an enhanced view of an image (e.g., a random crop) can predict the representation of another enhanced view of the same image (e.g., a neighboring crop). Surprisingly, however, the above training does not cause the feature representation of the encoder neural network 112 to fold into a constant vector, which is the same for all enhancements. Instead, the target neural network helps stabilize the training. The target neural network 120 does not need to be updated simultaneously with or at the same frequency as the online neural network.

[0085] After the neural network system 100 has been trained, all systems except the encoder neural network 112 (and its trained parameters) can be discarded. That is, Figure 2 The result of the process is a trained version of the encoder neural network 112.

[0086] Figure 3 The process of using a trained encoder neural network 112 to process data items (e.g., images) is illustrated. This process can be implemented as one or more computer programs on one or more computers at one or more locations.

[0087] In step 300, input data items, such as images or videos, are provided to the trained encoder neural network portion of the trained online neural network 110. The input data items are processed using part or all of the trained encoder neural network 112 (step 302) to output a representation of the input data items (step 304). Further processing is then performed to perform a task (step 306), such as an image processing task as described above. The trained encoder neural network 112 can be used to perform any processing task, such as processing data items of the same type used to train the system.

[0088] Depending on the task, not all trained encoder neural networks 112 may be needed to process the data items. Therefore, the representation output from the trained encoder neural network 112 can be the feature vector representation described above, or the representation output can be the output from intermediate layers or the "backbone" of the encoder neural network 112, rather than, for example, the output from the final fully connected layer. For example, in the case of an encoder neural network with a ResNet architecture, the representation output could be the output from intermediate convolutional neural network layers.

[0089] Figure 4a A computer-implemented data item processing neural network system 400 is shown, which includes a trained encoder neural network 112 (or a portion thereof) and an optional system head 402 adapted to perform a data item processing task. The system 400 is configured to receive data items as input and process the data items using the trained encoder neural network 112 (or a portion thereof) to output a representation of the input data items.

[0090] In some implementations, the system output 404 for performing the data item processing task includes the representation output from the trained encoder neural network 112. In some implementations, the representation output from the trained encoder neural network 112 is further processed by the system head 402 to generate the system output 404 for the task.

[0091] As an example, the representation output from the trained encoder neural network 112 includes feature vector representations, which can be used to evaluate the similarity between two data items, such as two images. This might involve sequentially feeding each data item to the encoder neural network 112 to generate a corresponding feature vector representation, and then using a similarity metric to compare these representations, such as a distance metric like the L-norm, or a dot product or cosine similarity metric. The similarity metric can be used, for example, to detect when a data item might have been duplicated for duplicate or near-duplicate detection. This can also be used for data item validation. As another example, the feature vector representation output from the trained encoder neural network 112 can be used to evaluate the similarity between a target data item (e.g., a target image or video) and each of multiple data items (e.g., images or videos) in a database. The closest or one of the closest data items can be selected to retrieve one or more data items from the database that are similar to the target data item.

[0092] As another example, one or more final output layers of the trained encoder neural network 112 may be discarded after training, and the representation output from the trained encoder neural network 112 may include feature map outputs generated from intermediate layers (previously) of the encoder neural network 112. Such feature maps may have the utility of, for example, recognizing features of input data items such as input images, and the data item processing task may be the task of generating such feature maps from input data items, such as from one or more input images.

[0093] In some example implementations, the data item is an image and the representation output from the trained encoder neural network 112 is further processed by a system head 402, where the neural network system 400 is an image processing system. For image classification or scene recognition tasks, the system head 402 may include a classifier, such as a linear classifier. For semantic segmentation tasks, the representation output may be the output from intermediate layers or the "backbone" of the encoder neural network 112, such as the output from a ResNet convolutional layer, and the system head 402 may include a semantic segmentation decoder neural network, such as a convolutional neural network with a final 1×1 convolution for per-pixel classification. For object detection tasks, the representation output may also be the output from intermediate layers or the "backbone" of the encoder neural network 112, and the system head 402 may include a bounding box prediction neural network head. For depth estimation tasks, the representation output may be the output from intermediate layers or the "backbone" of the encoder neural network 112, and the system head 402 may include a convolutional neural network with one or more upsampled blocks. In these examples, the system output 404 may be the output of the task as described above.

[0094] As another example, the output can be the output from an intermediate layer or "backbone" of the encoder neural network 112, the system head 402 can include a reinforcement learning system, and the system output 404 can be an action selection output for selecting actions to be performed by an entity (e.g., a machine in a real-world environment).

[0095] A data item processing neural network system 400 can be trained to perform a data item processing task. In some implementations, during this training period, the parameters of the trained encoder neural network 112 in system 400 can be frozen while the parameters of the system head 402 are trained simultaneously. In some implementations, during this training period, the parameters of the trained encoder neural network 112 and the parameters of the system head 402 can be trained jointly to fine-tune the parameters of the encoder neural network 112 for the task, optionally making some adjustments.

[0096] Figure 4b The process for training and using a data item processing neural network system 400 to perform data item processing tasks is illustrated. This process can be implemented as one or more computer programs on one or more computers at one or more locations.

[0097] Figure 4b An online neural network 110, including an encoder neural network 112, has been previously illustrated, for example, by... Figure 2 The bootstrap process is trained (step 400). In step 402, this process trains system 400 using any machine learning technique, such as backpropagation of the gradient of the object function, to perform a data item processing task using the same training data items as before or new training data items. Step 402 may, but does not need to, include further training of the parameters of encoder neural network 112 (or a portion thereof included in system 400). Then, in step 404, the trained system 400 can be used to perform the data item processing task.

[0098] Figure 5 This involves image classification from the ImageNet database (Russakovsky et al., arXiv:1409.0575). The y-axis shows the top 1 percentile accuracy, i.e., the accuracy of the prediction with the highest probability; the x-axis shows the number of parameters in the neural network that performs the classification.

[0099] exist Figure 5In the diagram, curve 510 is used for a ResNet-200 image classifier with supervised training. Curve 520 is used for an image classification system including a ResNet-50 encoder neural network, followed by a linear classifier trained on feature vector representations, where the encoder neural network parameters are frozen. The ResNet-50 encoder neural network is encoder neural network 112 that has been trained as described above. The trained encoder neural network 112 is compared to using contrastive loss (… Figure 5 Other self-supervised methods (not shown in the figure) perform better and are close to the performance of supervised training.

[0100] Data items can typically be of any type, including image and video frames as previously described. For example, a data item can be an audio data item, i.e., a data item comprising a representation of a digitized audio waveform (e.g., a speech waveform). This representation can include samples representing digitized amplitude values ​​of the waveform, or, for example, a time-frequency domain representation of the waveform, such as an STFT (Short-Term Fourier Transform) or MFCC (Mel Frequency Cepstral Coefficients) representation. In this case, a transformed or enhanced “view” of the data item can also include random cropping, but in the time or frequency domain rather than the spatial domain, such as selecting portions of the audio data item with random start and end times or with randomly selected higher and lower frequencies. Other transformed or enhanced “views” of the data item can include modifications to the amplitude of the data item, such as by randomly increasing or decreasing the amplitude of the audio; or, for example, by modifying the frequency characteristics of the audio by randomly filtering the audio.

[0101] Instead of representing an audio waveform, the data item can represent the waveform of any signal, such as a signal from a sensor, which is, for example, a sensor that senses an object or a property of the real world. The transformed view of the data item can then correspond to the views described above for the audio waveform.

[0102] When the data item represents a waveform, such as an audio waveform, the data item processing task may include, for example, a recognition or classification task, such as a speech or sound recognition task, a telephone or speaker classification task, or an audio tagging task, in which case the output may be a category score or tag of the data item or a fragment of the data item; or a similarity determination task, such as an audio copy detection or search task, in which case the output may be a similarity score.

[0103] In some implementations, the data item can be a text data item, and the transformed or enhanced "view" of the data item can include cropping or distortion of the data item, such as grammatical or spelling distortion. Data item processing tasks can include identification or classification tasks, or similarity determination tasks, such as generating category scores, similarity scores, or tags as described above; or machine translation tasks. The data item can also represent observations, such as ad impressions or click counts or ratios, for example, combined with other data such as text data. The transformed view can then similarly include distortion of the data item and can perform similar tasks.

[0104] For a computer system configured to perform a specific operation or action, this means that the system has software, firmware, hardware, or a combination thereof installed thereon, which, in operation, causes the system to perform those operations or actions. For one or more computer programs to be configured to perform a specific operation or action, this means that the one or more programs include instructions that, when executed by a data processing device, cause that device to perform those operations or actions.

[0105] The embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, including the structures disclosed in this specification and their structural equivalents, or combinations thereof.

[0106] Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier, for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of these. However, the computer storage medium is not a propagating signal.

[0107] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines used for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0108] A computer program (also referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but does not need to, correspond to a file in a file system. A program may be stored as a portion of a file that holds other programs or data, for example, as one or more scripts stored in a markup language document, as a single file dedicated to the program in question, or as multiple collaborative files, for example, as a file storing one or more modules, subroutines, or code portions. A computer program can be deployed to execute on a single computer or on multiple computers located in one place or distributed across multiple locations and interconnected through a communication network.

[0109] As used herein, "engine" or "software engine" refers to a software-implemented input / output system that provides outputs different from the inputs. An engine can be a coded functional block, such as a library, platform, software development kit ("SDK"), or object. Each engine can be implemented on any suitable type of computing device including one or more processors and computer-readable media, such as a server, mobile phone, tablet computer, laptop computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other fixed or portable device. Furthermore, two or more engines can be implemented on the same computing device or on different computing devices.

[0110] The processes and logic flows described in this specification can be executed by one or more programmable computers, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special-purpose logic circuits, and the device can be implemented as special-purpose logic circuits, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, the processes and logic flows can be executed by a graphics processing unit (GPU) or a tensor processing unit (TPU), and the device can also be implemented using a GPU or a TPU.

[0111] For example, a computer suitable for executing computer programs may, by way of example, be based on a general-purpose or special-purpose microprocessor or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory or both. Typical components of a computer are the central processing unit for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, to receive data from or transfer data to, or both. However, a computer does not need to have such devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name just a few examples.

[0112] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented or incorporated therein by dedicated logic circuitry.

[0113] To provide interaction with the user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, that the user can use to provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0114] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes backend components, such as a data server, or middleware components, such as an application server, or frontend components, such as a client computer with a graphical user interface or a web browser through which a user can interact with the implementation of the subject matter described in this specification, or any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), such as the Internet.

[0115] A computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact through a communication network. The client-server relationship arises from computer programs running on their respective computers that have a client-server relationship with each other.

[0116] While this specification contains numerous specific implementation details, these should not be construed as limiting any invention or the scope of the claims, but rather as descriptions of features characteristic of particular embodiments of a particular invention. Certain features described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations, and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and the claimed combination may be for sub-combinations or variations thereof.

[0117] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown, or requiring all shown operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0118] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims can be performed in a different order and still achieve the desired result. As an example, the processes depicted in the figures do not necessarily require the specific order or sequence shown to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

Claims

1. A method executed by one or more computers, the method comprising: The target neural network is trained by the one or more computers using machine learning techniques, the training comprising: The target neural network is used to process a first transformed view of the training data items to generate a target output that includes a first representation of the training data items; A second transformation view of the training data items is processed using an online neural network to generate a prediction of the target output, wherein the online neural network includes: (i) an encoder subnetwork having the same architecture as the target neural network but with different parameter values, and (ii) a prediction subnetwork including one or more neural network layers and being separate from the encoder subnetwork; The second transformation view of processing the training data items using the online neural network includes: The encoder subnetwork of the online neural network is used to process a second transformed view of the training data item to generate a second representation of the training data item; and The second representation of the training data item is processed using the prediction subnetwork of the online neural network to generate a prediction of the target output; Update one or more parameters of the online neural network to minimize the error between the predicted target output and the target output; and Update one or more parameters of the target neural network based on the parameters of the encoder subnetwork of the online neural network. The training data items include at least one of image data, video data, audio data, or text data.

2. The method according to claim 1, wherein, Updating one or more parameters of the target neural network includes: according to To determine the update values ​​of one or more parameters of the target neural network, wherein, θ represents the parameters of the target neural network, θ represents the parameters of the online neural network, and τ is the decay rate.

3. The method according to any one of claims 1-2, wherein, The online neural network and the target neural network each include a corresponding projection neural network, and the method further includes each projection neural network performing the following operations: Receive the representation of the training data item; and The representation of the training data items is processed to reduce the dimensionality of the representation.

4. The method according to claim 3, wherein, Each projection neural network consists of multiple perceptrons.

5. The method according to claim 1, wherein, The predictive neural network includes a multilayer perceptron.

6. The method of claim 1, further comprising: The parameters of the online neural network and / or the target neural network are initialized to random values.

7. The method of claim 1, further comprising: A first data item transformation is applied to the training data item to generate the first transformation view of the training data item; and A second data item transformation is applied to the training data item to generate a second transformed view of the training data item, wherein the second data item transformation is different from the first data item transformation.

8. The method according to claim 1, wherein, Updating one or more parameters of the online neural network includes: The prediction of the normalized target output; and The squared error between the predicted normalized target output and the target output is minimized.

9. The method of claim 1, further comprising: For each training data item in a batch comprising multiple training data items, the operations of processing the training data item using the target neural network in a first transformed view and processing the training data item using the online neural network are performed iteratively; then one or more parameters of the online neural network are updated, and one or more parameters of the target neural network are updated.

10. A processing system comprising a trained target neural network, wherein, The target neural network has been trained using the following method: A first transformed view of training data items is processed using a target neural network to generate a target output that includes a first representation of the training data items; A second transformation view of the training data items is processed using an online neural network to generate a prediction of the target output, wherein the online neural network includes: (i) an encoder subnetwork having the same architecture as the target neural network but with different parameter values, and (ii) a prediction subnetwork including one or more neural network layers and being separate from the encoder subnetwork; The second transformation view of processing the training data items using the online neural network includes: The encoder subnetwork of the online neural network is used to process a second transformed view of the training data item to generate a second representation of the training data item; and The second representation of the training data item is processed using the prediction subnetwork of the online neural network to generate a prediction of the target output; Update one or more parameters of the online neural network to minimize the error between the predicted target output and the target output; and Update one or more parameters of the target neural network based on the parameters of the encoder subnetwork of the online neural network. The training data items include at least one of image data, video data, audio data, or text data.

11. A computer-implemented system comprising one or more computers and one or more storage devices storing instructions, which, when executed by said one or more computers, cause said one or more computers to perform operations of a corresponding method according to any one of claims 1-9.

12. A computer-readable instruction, or one or more computer storage media storing the instruction, which, when executed by one or more computers, causes the one or more computers to perform the operation of a corresponding method according to any one of claims 1-9.

Images