Self-supervised representation learning using bootstrapped latent representations

Abstract

A computer implementation method for training a neural network. The method comprises the steps of processing a first transformed view of a training data item (e.g., an image) with a target neural network to generate a target output, processing a second transformed view of a training data item (e.g., an image) with an online neural network to generate a prediction of the target output, updating parameters of the online neural network to minimize the error between the prediction of the target output and the target output, and updating parameters of the target neural network based on the parameters of the online neural network. The method can effectively train an encoder neural network without using labeled training data items and without using contrast loss, that is, without requiring "negative examples" containing transformed views of different data items.

Description

Technology Field

[0001] This application claims priority to U.S. Provisional Application No. 63 / 035,583 filed June 5, 2020 and U.S. Provisional Application No. 63 / 039,343 filed June 15, 2020. The disclosure of this application is construed as part of and incorporated by reference to the disclosure of this application.

[0002] This specification relates to image processing using machine learning models. Background Technology

[0003] A neural network is a machine learning model that uses one or more non-linear unit layers to predict an output for a received input. Some neural networks include one or more hidden layers in addition to the output layer. The output of each hidden layer is used as the input for the next layer of the network, that is, the next hidden layer or output layer. Each layer of the network generates an output from the received input based on the current values ​​of an individual set of parameters.

[0004] This specification describes a system implemented as a computer program on one or more computers at one or more locations configured to learn a representation of a data item, such as an image, through a self-supervised learning process.

[0005] A first aspect of the present disclosure provides a computer-implemented method for training a neural network. The method comprises the steps of processing a first transformed view of a training data item (e.g., an image) with a target neural network to generate a target output; processing a second transformed view of a training data item (e.g., an image) with an online neural network to generate a prediction of the target output; updating parameters of the online neural network to minimize an error between the prediction of the target output and the target output; and updating parameters of the target neural network based on the parameters of the online neural network.

[0006] The term "transformed view" refers to a transformed version of a training data item (e.g., an image) and is used to distinguish training data items after they have undergone transformation (e.g., image transformation) from the original (untransformed) training data item.

[0007] The parameters of a neural network may include the weights of the neural network, and the step of updating the parameters of the neural network may include the step of adjusting the values ​​of the weights.

[0008] The step of updating one or more parameters of the target neural network may include the step of updating one or more parameters of the target neural network using the moving average of the parameters of the online neural network. The moving average may be an exponential moving average.

[0009] The step of updating one or more parameters of the target neural network It may include a step of determining an updated value of one or more parameters of a target neural network according to, where represents the parameters of the target neural network, θ represents the parameters of the online neural network, and τ is the damping rate. The damping rate can be a value between 0 and 1.

[0010] The online neural network and the target neural network may include individual encoder neural networks. Accordingly, the method may further include the steps of receiving a transformed view of a training data item (e.g., an image) by each encoder neural network, and processing the transformed view of the training data item (e.g., an image) to generate a representation of the training data item.

[0011] Optionally, each encoder neural network may include a residual neural network, that is, a neural network having one or more residual or skip connections around one or more layers of the neural network.

[0012] When an online neural network is trained, the encoder neural network of the online neural network can be used to generate a representation of any appropriate input data item, for example, a representation of an input image (pixel). Thus, the generated image representation can be used by other downstream tasks.

[0013] For example, an image classification operation can be performed on an image representation. As another example, an image segmentation operation can be performed on an image representation. Other image processing tasks may be performed alternatively or additionally.

[0014] The online neural network and the target neural network may each include an individual projection neural network. The method may further include steps of performing operations by each projection neural network, including receiving a representation of a training data item (e.g., an image) and processing the representation of the training data item (e.g., an image) to reduce the dimensionality of the representation. Optionally, each projection neural network may include a multilayer perceptron.

[0015] The use of projective neural networks is optional because online neural networks can directly predict the representation of training data items (e.g., images) generated by the target neural network (rather than predicting the projection of the representation). However, the use of projective neural networks can provide improved performance.

[0016] The online neural network may include a predictive neural network. The method may further include steps of performing operations, such as receiving a representation of a training data item (e.g., an image) by the predictive neural network, and processing the representation of the training data item using a regression model implemented by the parameters of the predictive neural network to generate a prediction of the target output. Optionally, the predictive neural network may include a multilayer perceptron. The use of a predictive neural network is not mandatory but can help improve the stability of the training. In the implementation, the target neural network does not include a predictive neural network.

[0017] Therefore, in the implementation, the online neural network and the target neural network have the same neural network architecture except for one or more additional processing steps in one of the neural networks, specifically in the online neural network, but may have different parameter values.

[0018] The target neural network can have a stop gradient ("sg"). The stop gradient prevents backpropagation to the target neural network, ensuring that the parameters of the target neural network are not updated when the error is minimized.

[0019] The method may further include a step of initializing the parameters of the online neural network and / or the target neural network to random values.

[0020] The method may further include the step of applying a first data item transformation (e.g., image transformation) to a training data item (e.g., image) to generate a first transformation view of the training data item, and the step of applying a second data item transformation (e.g., image transformation) to the training data item to generate a second transformation view of the training data item (e.g., image). The second data item transformation (e.g., image transformation) is different from the first data item transformation (e.g., image transformation).

[0021] If the training data item includes a training image, the first image transformation and the second image transformation may include a combination of one or more of cropping, flipping along the horizontal and / or vertical axes, color jittering; conversion to grayscale; Gaussian blurring; or solarization. Other image transformations may be used alternatively or additionally. By using transformed views of the training image, the online neural network learns based on the target network's representation of different transformed views of the same training image.

[0022] Although the present disclosure focuses on an example where two different transformed views of a training data item (e.g., an image) are input to an online neural network and a target neural network, in other examples, the transformed views of a training data item (e.g., an image) may be input to only one of the online neural network and the target neural network. In this example, the original training data item (e.g., an image) (untransformed) is input to the other of the online neural network and the target neural network.

[0023] The step of updating one or more parameters of an online neural network may include the step of using a machine learning optimizer based, for example, stochastic gradient descent, to minimize the aforementioned error. The step of updating one or more parameters of an online neural network includes the step of normalizing a prediction of a target output; and the step of minimizing the squared error between the normalized prediction of a target output and the target output.

[0024] The steps of processing a first transformation view of a training data item with a target neural network, processing a second transformation view of a training data item with an online neural network, and updating one or more parameters of the target neural network may be performed repeatedly for each training data item in a batch containing multiple training data items. The parameters of the online neural network and the target neural network may be updated after each training data item in the batch is processed.

[0025] Further aspects of the present disclosure provide a computer-implemented method for processing a data item, e.g., an image. The method comprises the steps of: providing an input data item (e.g., an image) to an online neural network, i.e., to a part of an online neural network trained according to the present disclosure; processing the input data item (e.g., an image) by the online neural network, i.e., using the part of the online neural network trained; outputting a representation of the input data item (e.g., an image) from the online neural network, i.e., from the part of the online neural network trained; and processing the representation of the input data item (e.g., an image).

[0026] An online neural network may include a residual neural network configured to generate a representation of an input data item (e.g., an image).

[0027] If the input data item includes an image, the step of processing the representation of the input image may include a step of classifying the input image using the representation of the input image. Alternatively, or instead, the step of processing the representation of the input image may include a step of segmenting the input image using the representation of the input image. Alternatively, or instead, other image processing tasks may be performed.

[0028] A further aspect of the present disclosure provides a system comprising one or more computers and one or more storage devices that store instructions that, when executed by one or more computers, cause one or more computers to perform any method disclosed herein.

[0029] A further aspect of the present disclosure provides one or more computer storage media that store instructions that, when executed by one or more computers, cause one or more computers to perform any method disclosed herein.

[0030] The subject matter described in this specification may be implemented in specific embodiments to realize one or more of the following advantages.

[0031] Online neural networks are trained by a self-supervised learning process, thereby learning how to represent images from unlabeled raw training data. Unlike approaches to self-supervised learning that use contrastive loss functions, the technique disclosed herein avoids the need for negative pairs of training examples (e.g., images) (i.e., different pairs of training examples such as images). This, in turn, avoids the difficulties associated with the use of negative pairs, such as large batch sizes, memory banks consisting of sample representations from training datasets, and the careful selection of negative pairs. Nevertheless, when applied, for example, to image classification tasks, image representations generated by an online neural network trained as disclosed herein have been found to achieve classification accuracy equivalent to training using labeled examples and superior to some contrastive methods.

[0032] The technology disclosed in this specification is also computationally efficient as it avoids the high computational requirements of some methods.

[0033] The inputs to the online neural network and the target neural network may include appropriate types of image data, such as video image data. The image data may include color or grayscale pixel value data. Such image data may be captured from an image sensor, such as a camera or a LIDAR sensor.

[0034] Image representations generated by the online neural network disclosed herein may be used for various image processing tasks. For example, image representations may be used for image classification, thereby allowing the classification system to output one or more class labels for a given input image representation. Continuing with this example, the classification system may process the image representation and output a score for each set of object categories, whereby each score represents the estimated likelihood that the image contains an object belonging to the category.

[0035] As another example of an image processing task, image representations can be used for image segmentation, whereby the segmentation system uses the image representation to label individual pixels of an input image as belonging to one of several different classes. One exemplary use case of image segmentation is object detection, whereby the segmentation system labels pixels of an input image according to the type of object represented by each pixel.

[0036] Object detection can be used as input for mechanical agents, such as robots or vehicles, that can operate in a real-world environment. The detected objects may be, for example, obstacles (e.g., people, other mechanical agents, walls) and / or paths (e.g., roads or other surfaces on which the mechanical agent can move). Thus, the detected objects can be used by the mechanical agent's control system to make decisions regarding how to perform mechanical tasks, such as controlling the mechanical agent's direction of movement and / or speed.

[0037] Another exemplary use case for image segmentation is segmenting medical images, whereby the segmentation system labels pixels in the input medical image based on whether they represent areas of the human or animal body where a specific medical condition exists. These are purely non-limiting examples of image segmentation, and there are many other practical applications of image segmentation.

[0038] Throughout this specification, processing an image using a neural network refers to processing intensity values ​​associated with pixels of an image using a neural network.

[0039] Details of one or more embodiments of the subject matter described in this specification are described in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Brief explanation of the drawing

[0040] Figure 1 illustrates an exemplary neural network system including an encoder neural network. Figure 2 illustrates an exemplary process for training the system of Figure 1. Figure 3 illustrates an exemplary process for using a trained encoder neural network. FIGS. 4a and 4b illustrate an exemplary neural network system including a trained encoder neural network, and an exemplary process for training and using the system. Figure 5 illustrates the performance of a trained encoder neural network for an image classification task. In drawings, similar reference numbers indicate similar elements. Specific details for implementing the invention

[0041] This specification describes a method for training an encoder neural network to generate representations of data items without using labeled training data items and without using contrast loss. A method using contrast loss learns to distinguish between positive examples, that is, two transformed or "augmented" views of the same data item, and negative examples, which include augmented transformed or "augmented" views of different data items. Implementations of the described method require only positive examples for their objective function and do not require negative examples, which can be difficult to generate. This also allows for achieving performance approaching that of a neural network using labeled training data.

[0042] In some of the examples described, data items include images, but generally, all types of data items can be processed. Several examples of other types of data items will be explained later. This method can be used to train encoder neural networks to perform all types of tasks involving the processing of the same types of data items (e.g., images) used for training.

[0043] Where the image data items used in this specification include video data items, the task may include any type of image processing or vision task, such as an image classification or scene recognition task, an image segmentation mask (e.g., a semantic segmentation mask), an object localization or detection task, or a depth estimation task. When performing these tasks, the input may include or be derived from pixels of an image. In the case of an image classification or scene recognition task, the output may include a classification output providing scores for each of a plurality of image or scene categories, representing the likelihood that, for example, an input data item, an object or element of an input data item, or an action within a video data item will belong to a category. In the case of an image segmentation task, the output may include, for each pixel, an assigned segmentation category or the probability that the pixel belongs to a segmentation category, for example, an object or action represented in the image or video. In the case of an object localization or detection task, the output may include data defining the coordinates of a bounding box or region for one or more objects represented in the image. For depth estimation tasks, the output may include, for each pixel, an estimated depth value such that the output pixel defines a (3D) depth map for the image. Such tasks can also contribute to higher-level tasks, such as object tracking across video frames; or gesture recognition, that is, the recognition of gestures performed by entities depicted in the video.

[0044] Another exemplary image processing task is an image keypoint detection task, where the output includes the coordinates of one or more image keypoints, such as landmarks of objects represented in the image. For example, this may include a human pose estimation task where keypoints define the locations of body joints. An additional example is an image similarity determination task, where the output may include a value indicating similarity between two images, for example, as part of an image search task.

[0045] This method is used to train a neural network system in which an encoder neural network forms part of the system. The rest of the system is not required after training. The trained encoder neural network can be used to perform a task without additional training, for example, without parameters being fixed or additional training to perform a specific task. Generally, the trained encoder neural network can be integrated into a larger system to perform a specific task, such as a system configured to perform image classification, image segmentation, object localization, or depth estimation tasks. Therefore, an image processing system that integrates the trained neural network is also provided.

[0046] For image data items, the transformed or "augmented" view may be a transformed view of the same image. For example, an image may be randomly cropped or distorted in other ways to generate two views. In a broad sense, this method learns how to generate a representation that is robust against such transformations.

[0047] FIG. 1 illustrates a neural network system (100) for implementing an example of a method. The system (100) of FIG. 1 may be implemented as one or more computer programs on one or more computers located at one or more locations.

[0048] The system (100) includes an online neural network (110) and a target neural network (120).

[0049] The target neural network (120) is configured to receive a first transformed (transformed) view (v') of a data item (102). The online neural network (110) is configured to receive a second transformed view (v) of the same data item (102). Transformed views are generated by applying respective transformations (T', T) to the data item (102), as will be further described later. In some implementations, the data item (102) includes an image containing an image frame of a video, as used herein.

[0050] The target neural network (120) is configured to process a first transformed view (v') of a data item to generate a target output (126(z')). The online neural network (110) is configured to process a second transformed view (v) of a data item to generate a prediction (118)(q(z)) of the target output. The online neural network (110) is trained, for example by a training engine, by updating the parameters of the online neural network using a machine learning optimizer to minimize the difference or error between the prediction (118) of the target output (126) and the target output (126).

[0051] The online neural network (110) includes an encoder neural network (112), an optional projection neural network (114), and a prediction neural network (116), and is defined by a set of parameters of these neural networks. The target neural network (120) includes an encoder neural network (122) and an optional projection neural network (124), and is defined by a set of parameters of these neural networks.

[0052] In some implementations, although not mandatory, the target neural network (120) has the same architecture as the online neural network (110) (excluding the prediction neural network) but has different parameters (weights). That is, the encoder neural network (112, 122) may be the same neural network but have different parameters, and similarly, the projection neural network (114, 124) may be the same neural network but have different parameters.

[0053] Each encoder neural network (112, 122) receives a transformed view of a data item (102) and processes it to generate a representation of each transformed view of the data item (102), i.e., a high-dimensional feature vector. This can be received and processed by subsequent individual projection neural networks (114, 124) to generate a reduced-dimensional representation of each transformed view of the data item (102), i.e., reduced-dimensional feature vectors of z and z', respectively. The reduced-dimensional representation from the projection neural network (124) provides a target output (126).

[0054] The prediction neural network (116) receives a reduced-dimensional representation from the projection neural network (114) (or a representation from the encoder neural network (112)) and processes it to generate a prediction (118). In this implementation, the prediction (118) includes a vector having the same dimensions as the target output (126). The input to the prediction neural network (116) may include a feature vector representation of a data item identical to that of the target output (126), excluding the values ​​of the feature vector. The target output (126) may be the output of the projection neural network (124) as previously described, or in some implementations, a representation from the encoder neural network (122).

[0055] The encoder neural network (112, 122) may have any architecture suitable for encoding data items (102). The result of the training method described herein is the trained encoder neural network (112). The encoder neural network (112) may be any neural network trained to generate a representation of an input data item, configured to receive one of the data items as input, and to generate a feature vector representation of the received data item.

[0056] For example, if the data item includes an image or video, the encoder neural network (112, 122) includes any type of image or video encoding neural network configured to generate a representation such as a feature vector representation of the input image or video. Such image or video encoding neural networks may include one or more convolutional neural network layers or have any other architecture suitable for image or vision processing. In an implementation, such image or video encoder neural networks (112, 122) may each include a residual neural network, that is, a neural network having one or more residual or skip connections. As merely an exemplary example, a convolutional neural network having one or more residual or skip connections may be used, for example, based on ResNet (He et al., arXiv:1512.03385), but it is not necessary to use a convolutional neural network.

[0057] Generally, the encoder neural network (112, 122) is configured to process the type of data item to be trained on, such as images, videos, audio, text, or others.

[0058] The projection neural network (114, 124) may be any neural network configured to reduce the dimensionality of the feature representation. For example, each may be a multilayer perceptron having an output space smaller than the input space, for example, containing one or more fully connected layers, each of which is optionally followed by a batch normalization layer (Ioffe and Szegedy, arXiv:1502.03167), followed by an optionally fully connected linear output layer.

[0059] Just as an example, for a 224×224 pixel image, the feature vector representation generated by each encoder neural network can have 4096 dimensions and the reduced dimension representation can have 256 dimensions.

[0060] In the implementation, the system can train the encoder neural network (112) to ignore transformations applied to data items. For example, in the case of image data items, if the transformation on the training image involves a color change, the feature representation can be trained to ignore the color. Projecting the feature representation onto a reduced-dimensional representation can help retain useful information in that feature representation.

[0061] In an implementation, the prediction neural network (116) may be a relatively small neural network that learns how to predict a target output, for example, from a reduced-dimensional representation from a projection neural network (124) to a reduced-dimensional representation from a projection neural network (114). Thus, the parameters of the prediction neural network (116) implement a regression model. In an exemplary implementation, the prediction neural network (116) includes other multilayer perceptrons.

[0062] Optionally, the output vectors of the target output (126) and the prediction (118), respectively, may be normalized using, for example, an L2 norm. For example, the target output (126)(z') is It can be normalized to, and the prediction (118)(q(z)) is It can be normalized to. Here represents the prediction neural network (116) and z is a reduced dimension representation from the projection neural network (114).

[0063] The neural network system (100) also includes a training engine (130) configured to implement a training process for the system.

[0064] FIG. 2 illustrates an exemplary process for training the neural network system (100) of FIG. 1, in particular the encoder neural network (112). The process of FIG. 2 may be implemented as one or more computer programs on one or more computers located at one or more locations.

[0065] The parameters of the online neural network (110) and the target neural network (120) may be initialized to random values. In step (200), a training data item, for example, a training image, is acquired. This is then processed to acquire a first and second view that is differently transformed or "augmented" (step 202). This may involve applying one or a series of transformations to the data item to generate each transformed view. For example, each transformed view may be acquired by applying each of the transformation sets with a predetermined probability.

[0066] For example, if the training data items include images, the transformation may include one or more of random cropping, image flipping, color jittering, color dropping, Gaussian blurring, and solarization. Random cropping may include selecting a random patch of an image and then expanding it to the original size of the image. Image flipping may include applying a horizontal or vertical flip to the image. Color jittering may include changing one or more of the brightness, contrast, saturation, and hue of some or all pixels of the image by a random offset. Color dropping may include converting the image to grayscale. Gaussian blurring may include applying a Gaussian blurring kernel to the image; other types of kernels may be used for other types of filtering. Solarization may involve applying a solarizing color transformation to an image. Other color transformations may be used. Other variations such as rotation or cutting out parts of the image (setting pixels of a random patch to uniform values) are possible.

[0067] For example, the transformation of the image may include an adversarial perturbation, that is, a perturbation selected to increase the likelihood that the encoder neural network (112) will generate a false representation. For example, an adversarial attack may be implemented in one of the transformed views of the transformed view pair using the technique described in Madry et al., arXiv: 1706.06083 to maximize the error between the prediction (118) and the target output (126).

[0068] There are various transformations that can be used to obtain the transformed view. The specific type of transformation used may vary, for example, depending on the type of variation expected between the task and data items that the encoder neural network (112) is training. Optionally, the transformed view may be normalized for color channels, for example, in the case of an image. Normalization may include subtracting the mean and dividing by the standard deviation.

[0069] A first transformed view of the data item is processed by the target neural network (120) to generate a target output (126) (step 204), and a second transformed view of the data item is processed by the online neural network (110) to generate a prediction (118) of the target output (step 206). Then, a prediction error between the prediction (118) and the target output (126), specifically a prediction loss based on the prediction (118) and the target output (126), is determined (step 208). The prediction loss may include any measure of the difference between the prediction (118) and the target output (126) and is optionally normalized as previously described. For example, the prediction loss or error may include (mean) squared error, negative cosine or dot product similarity, or cross-entropy loss (where the feature vector values ​​are interpreted as probabilities and normalized). For example, the prediction loss or error (L) It can be determined as an L-norm such as, where represents the L2 norm.

[0070] In some implementations, the method determines an additional prediction loss or error (L') with swapped first and second transformation views, namely, a first transformation view of a data item processed by the online neural network (110) and a second transformation view of a data item processed by the target neural network (120). The target neural network (120) then determines the symmetric prediction loss or error (L'). TOTAL ) can be determined by summing the losses. That is, L TOTAL = L + L'.

[0071] The process can accumulate prediction loss or error for a batch of training data items before proceeding.

[0072] In steps (210 and 212), the parameters of the online neural network (110) and the target neural network (120) are updated. However, only the parameters of the online neural network (110), not the parameters of the target neural network (120), are updated by gradient descent, that is, by backpropagating the gradient of the prediction loss or error. This can be expressed by providing a "stop gradient" to the target neural network (120), that is, by stopping the gradient flowing back to the target neural network (120). However, in practice, this can be achieved by training only the online neural network (110) using the prediction loss or error.

[0073] Accordingly, in step (210), the parameters of the online neural network (110) are updated to minimize the prediction loss or error, while in step (212), the parameters of the target neural network (120) are updated based on the parameters of the online neural network. In some implementations, minimizing the error may involve maximizing the similarity, the cosine similarity between the prediction (118) and the target output (126).

[0074] In the implementation, the parameters of the online neural network (110) are updated using a stochastic optimization step implemented with a machine learning optimizer, such as a gradient descent-based optimizer, to minimize the prediction loss or error. Other types of machine learning optimizers may be used. In the implementation, the optimizer predicts the loss or error (i.e., L or L) only for the parameters of the online neural network (110) (i.e., not for the parameters of the target neural network (120)). TOTAL Minimizes ). For example, the parameters of the updated online neural network (110) can be updated by backpropagating the gradient of the prediction loss or error through the prediction neural network (116), the projection neural network (114) (if present), and the encoder neural network (112).

[0075] In the implementation, the parameters of the target neural network (120) are updated based on the parameters of the corresponding part of the online neural network (110), that is, not based on prediction loss or error. For example, the parameters of the target neural network (120) may be determined, for example, as a copy or moving average of the parameters of the online neural network (110), for example, as a weighted or exponential moving average. Generally, the parameters of the target neural network (120) include a delayed (and more stable) version of the parameters of the online neural network (110).

[0076] In some implementations, the parameters of the target neural network (120) are updated It can be determined using, where τ is the target damping rate in the range [0, 1], and τ is a parameter of the target neural network (120), and θ is a set of parameters of the online neural network (110), excluding the parameters of the prediction neural network, i.e., the parameters of the encoder neural network (112) and, if present, the parameters of the projection neural network. Only as an example, τ can be greater than 0.99 and can be increased during training.

[0077] Therefore, the implementation of the method uses bootstrapping, as the updated, i.e., partially trained online neural network (110) is used to update the target neural network (120) to generate a new target output for further training of the online neural network (110). Training the online neural network (110) with the new target improves the representation from the encoder neural network (112).

[0078] For example, a representation of one augmented view of an image, for example, a random crop, can predict a representation of another augmented view of the same image, that is, a neighbor crop. However, surprisingly, the training described above does not result in reducing the feature representation of the encoder neural network (112) to a constant vector, and this will be the same for all augmentations. Instead, the target neural network helps stabilize the training. The target neural network (120) does not need to be updated at the same time or frequency as the online neural network.

[0079] After the neural network system (100) is trained, all systems except the encoder neural network (112) (and its trained parameters) can be discarded. That is, the result of the process of FIG. 2 is a trained version of the encoder neural network (112).

[0080] FIG. 3 illustrates a process using a trained encoder neural network (112) to process data items such as images. The process may be implemented as one or more computer programs on one or more computers located at one or more locations.

[0081] In step (300), an input data item, e.g., an image or video, is provided to a trained encoder neural network portion of the trained online neural network (110). The input data item is processed using part or all of the trained encoder neural network (112) (step 302) to output a representation of the input data item (step 304). This is then further processed to perform a task, e.g., the previously described image processing task (step 306). The trained encoder neural network (112) may be used to perform any processing task, for example, processing the same type of data item used in system training.

[0082] Depending on the task, the entire trained encoder neural network (112) may not be required to process the data items. Therefore, the representation output from the trained encoder neural network (112) may be the feature vector representation described above, or the representation output may be, for example, an output from the "backbone" or intermediate layer of the encoder neural network (112) rather than an output from the final fully connected layer. For example, if the encoder neural network has a ResNet architecture, the representation output may be an output from an intermediate convolutional neural network layer.

[0083] FIG. 4a illustrates a computer implementation of a data item processing neural network system (400) comprising a trained encoder neural network (112) (or part thereof) adapted to a data item processing task to be performed and an optional system head (402). The system (400) is configured to receive a data item as input and to process the data item using the trained encoder neural network (112) (or part thereof) to output a representation of the input data item.

[0084] In some implementations, the system output (404) for performing a data item processing task includes a representation output from a trained encoder neural network (112). In some implementations, the representation output from the trained encoder neural network (112) is further processed by a system head (402) to generate a system output (404) for the task.

[0085] As an example, the representation output from the trained encoder neural network (112) includes a feature vector representation that can be used to evaluate the similarity between two data items, for example, two images. This may include presenting each data item to the encoder neural network (112) in turn to generate individual feature vector representations, and then comparing the representations using a distance metric, such as a similarity measure, for example, an L-norm, inner product, or cosine similarity measure. The similarity measure may be used, for example, to detect cases where data items are likely to have been copied for duplicate or near-duplicate detection. It may also be used for data item verification. As another example, the feature vector representation output from the trained encoder neural network (112) may be used to evaluate the similarity between a target data item (e.g., a target image or video) and each of multiple data items (e.g., images or videos) in a database. The closest or one of the closest may be selected to retrieve data items or items similar to the target data item from the database.

[0086] As an additional example, the final output layer or layers of the trained encoder neural network (112) may be discarded after training, and the representation output from the trained encoder neural network (112) may include a feature map output generated from an intermediate layer (previously present) of the encoder neural network (112). This feature map may have utility for identifying features of input data items, such as input images, for example, and the data item processing task may be a task of generating this feature map from input data items, for example, input images or images.

[0087] In some exemplary implementations where the data item is an image and the representation output from the trained encoder neural network (112) is further processed by the system head (402), the neural network system (400) is an image processing system. For an image classification or scene recognition task, the system head (402) may include a classifier, for example, a linear classifier. For a semantic segmentation task, the representation output may be the output of an intermediate layer or "backbone" of the encoder neural network (112), for example, a ResNet convolution layer, and the system head (402) may include a semantic segmentation decoder neural network, for example, a convolution neural network having a final 1×1 convolution for classification per pixel. For an object detection task, the representation output may also be the output from the "backbone" or intermediate layer of the encoder neural network (112), and the system head (402) may include a bounding box prediction neural network head. For a depth estimation task, the representation output may be an output from the "backbone" or intermediate layer of the encoder neural network (112) and the system head (402), and the system head (402) may include a convolutional neural network having one or more upsampling blocks. In this example, the system output (404) may be an output for the previously described task.

[0088] As another example, the representation output may be an output from an intermediate layer or "backbone" of the encoder neural network (112), the system head (402) may include a reinforcement learning system, and the system output (404) may be an action selection output for selecting an action to be performed by an agent (a mechanical agent in the real environment).

[0089] A data item processing neural network system (400) can be trained to perform a data item processing task. In some implementations, during this training, the parameters of the trained encoder neural network (112) in the system (400) may be fixed while the parameters of the system head (402) are being trained. In some implementations, during this training, the parameters of the trained encoder neural network (112) and the parameters of the system head (402) may be jointly trained to fine-tune the parameters of the encoder neural network (112) for the task, optionally with some normalization.

[0090] FIG. 4b illustrates a process of training and using a data item processing neural network system (400) to perform a data item processing task. The process may be implemented as one or more computer programs on one or more computers located at one or more locations.

[0091] FIG. 4b illustrates that an online neural network (110) including an encoder neural network (112) has been previously trained, for example, by the bootstrapping process (step 400) of FIG. 2. In step (402), the process trains the system (400) using any machine learning technique, for example, backpropagation of the objective function gradient, to perform a data item processing task using the same training data item as the previous training data item or the new training data item. Step (402) may include, but is not required to include, additional training of the parameters of the encoder neural network (112) (or a part thereof included in the system (400)). Then, in step (404), the trained system (400) can be used to perform the data item processing task.

[0092] Figure 5 relates to image classification from the ImageNet database (Russakovsky et al., arXiv:1409.0575). The y-axis represents the top-1 percentage accuracy, i.e., the accuracy of the prediction with the highest probability, and the x-axis represents the number of parameters of the neural network performing the classification.

[0093] In FIG. 5, curve (510) is for a ResNet-200 image classifier with supervised training. Curve (520) is for an image classification system including a ResNet-50 encoder neural network, followed by a linear classifier trained on a feature vector representation with fixed encoder neural network parameters. The ResNet-50 encoder neural network is an encoder neural network (112) trained as described above. The trained encoder neural network (112) provides better performance than other self-supervised approaches using contrast loss (not shown in FIG. 5) and approaches the performance of supervised training.

[0094] The data item may generally be any type of data item, including the previously described image and video frames. For example, the data item may be an audio data item, that is, a data item containing a representation of a digitized audio waveform (e.g., a speech waveform). Such a representation may include samples representing the digitized amplitude values ​​of the waveform, or time-frequency domain representations of the waveform, such as a Short Fourier Transform (STFT) or Mel-Frequency Cepstral Coefficient (MFCC) representation. In this case, the transformed or augmented "view" of the data item may also include random excerpts, but may include a selection of parts of the audio data item having random start and end times or randomly selected upper and lower frequencies, in the time or frequency domain rather than the spatial domain. Other transformed or augmented "views" of the data item may include modifying the amplitude of the data item, for example, by randomly increasing or decreasing the amplitude of the audio, and modifying the frequency characteristics of the audio, for example, by randomly filtering the audio.

[0095] Instead of a data item representing an audio waveform, the data item can represent the waveform of an arbitrary signal, for example, a signal from a sensor that detects objects or characteristics of the real world. Then, the transformed view of the data item can correspond to what was described above for the audio waveform.

[0096] Here, a data item represents a waveform, for example, an audio waveform, and a data item processing task may include an identification or classification task, such as a speech or sound recognition task, for example, an audio tagging task (in which case the output may be a category score or tag for a data item or a segment of a data item), or a similarity determination task, for example, an audio copy detection or search task, in which case the output may be a similarity score.

[0097] In some implementations, the data item may be a text data item, and the transformed or augmented "view" of the data item may include distortions of the data item, such as excerpts of the data item or grammatical or spelling distortions. Data item processing tasks may include identification or classification tasks, or similarity determination tasks for generating category scores, similarity scores, or tags as described above, or machine translation tasks. The data item may also be combined with other data, such as text data, to represent observations such as ad impressions, click counts, or click-through rates. The transformed view may then similarly include distortions of the data item, and similar tasks may be performed.

[0098] That a system composed of one or more computers is configured to perform a specific operation or action means that software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action during operation is installed on the system. That one or more computer programs are configured to perform a specific operation or action means that one or more programs include instructions that cause the device to perform the operation or action when executed by the data processing device.

[0099] Embodiments of the subjects and functional operations described in this specification may be implemented in digital electronic circuits, tangibly implemented computer software or firmware, computer hardware including structures disclosed in this specification and structural equivalents thereof, or a combination of one or more of these.

[0100] Embodiments of the subject matter described herein may be implemented as one or more computer programs, namely, one or more modules of computer program instructions encoded in a tangible non-transient storage medium to be executed by a data processing device or to control the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded in an artificially generated radio signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, created to encode information to be transmitted to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these. However, the computer storage medium is not a propagated signal

[0101] The term “data processing device” includes all kinds of devices, devices, and machines for processing data, including, for example, programmable processors, computers, or multiprocessors or computers. A device may include special-purpose logic circuits, for example, a Field Programmable Gate Array (FPGA) or an Application-Specific Integrated Circuit (ASIC). A device may optionally include code that, in addition to the hardware, creates an execution environment for the corresponding computer program, for example, processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0102] A computer program (which may also be referred to as a program, software, software application, app, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be distributed in any form, including standalone programs or modules, components, subroutines, or other units suitable for use in a computing environment. A computer program may correspond to a file in a file system, but is not necessarily so. A program may be stored, for example, in one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in part of a file containing other programs or data stored in multiple coordination files (for example, files storing one or more modules, subprograms, or parts of code). A computer program may be distributed to run on a single computer or at a single site, or on multiple computers distributed across multiple sites and interconnected by a data communication network.

[0103] As used herein, “engine” or “software engine” refers to a software-implemented input / output system that provides an input and other outputs. An engine may be an encoded functional block such as a library, platform, software development kit (“SDK”), or object. Each engine may be implemented in any suitable type of computing device, such as a server, mobile phone, tablet computer, notebook computer, music player, e-book reader, laptop or desktop computer, PDA, smartphone, or other stationary or portable device, comprising one or more processors and a computer-readable medium. Additionally, two or more engines may be implemented in the same computing device or in different computing devices.

[0104] The processes and logic flows described herein may be performed by one or more programmable computers that execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows may also be performed by special-purpose logic circuits such as FPGAs or ASICs, and devices may also be implemented. For example, the processes and logic flows may be performed by a graphics processing unit (GPU) or a tensor processing unit (TPU), and devices may also be implemented using these.

[0105] A computer suitable for executing computer programs may be based, for example, on a general-purpose or special-purpose microprocessor, or both, or on any type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random access memory, or both. Typical elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes or is operablely connected to one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, to receive data from, transmit data to, or both of these. However, a computer does not require these devices. Furthermore, a computer may be embedded in portable storage devices such as mobile phones, PDAs, mobile audio or video players, game consoles, Global Positioning System (GPS) receivers, or Universal Serial Bus (USB) flash drives.

[0106] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks or removable disks); magneto-optical disks; and all forms of non-volatile memory, media, and memory devices, including CD-ROM and DVD-ROM disks. Processors and memory may be complemented or integrated by special-purpose logic circuits.

[0107] To provide interaction with a user, embodiments of the subject matter described herein may be implemented in a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user and a keyboard and pointing device (e.g., a mouse or trackball) for the user to provide input to the computer. Interaction with a user may also be provided using other types of devices. For example, feedback provided to a user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from a user may be received in any form, including sound, voice, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from the device used by the user, for example, by transmitting a web page to a web browser on the user device in response to a request received from a web browser.

[0108] Embodiments of the subject matter described herein may be implemented in a computing system that includes, for example, a backend component such as a data server, a middleware component such as an application server, a frontend component (for example, a client computer with a graphical user interface, a web browser, or an app through which a user can interact with the implementation of the subject matter described herein), or a combination of one or more backend, middleware, or frontend components. Components of the system may be interconnected by any form or medium of digital data communication, such as a communication network. Examples of communication networks include a Local Area Network (LAN) and a Wide Area Network (WAN), for example, the Internet.

[0109] A computing system may include clients and servers. Clients and servers are typically located far apart from each other and usually interact through a communication network. The relationship between clients and servers arises thanks to computer programs running on each computer that have a client-server relationship with one another.

[0110] Although this specification contains many specific implementation details, they should not be interpreted as a limitation on the scope of any invention or claimable scope, but rather as a description of features that may be specific to a particular embodiment of a particular invention. Specific features described in this specification in relation to separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. Furthermore, while features may be described above as operating in a specific combination and may even be initially claimed as such, one or more features of the claimed combination may be removed from the combination in some cases, and the claimed combination may relate to a sub-combination or a variation of the sub-combination.

[0111] Similarly, although operations are illustrated in the drawings and cited in a specific order in the claims, this should not be understood as requiring that these operations be performed in the specific order illustrated or sequential order, or that all illustrated operations be performed, in order to achieve a desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems are generally packaged into a single software product or multiple software products.

[0112] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the operations mentioned in the claims may be performed in a different order and still obtain a desirable result. As an example, the process illustrated in the accompanying drawings does not necessarily require the specific order or sequential order illustrated to achieve a desirable result. In some cases, multitasking and parallel processing may be advantageous.

Claims

Claim 1 A computer implementation method for training a neural network, wherein the method comprises: a step of processing a first transformed view of a training data item with a target neural network to generate a target output; a step of processing a second transformed view of a training data item with an online neural network trained by a self-supervised learning process to generate a prediction of the target output; a step of updating one or more parameters of the online neural network to minimize an error between a prediction of the target output and the target output, wherein the parameters of the online neural network are updated by backpropagating the gradient of the error between the prediction of the target output and the target output—wherein the parameters of the target neural network are not updated because backpropagation to the target neural network is prevented by the stop gradient—and a step of updating one or more parameters of the target neural network based on the parameters of the corresponding part of the updated online neural network. Claim 2 A computer-implemented method for training a neural network, characterized in that, in claim 1, the step of updating one or more parameters of the target neural network includes the step of updating one or more parameters of the target neural network using the moving average of the parameters of the online neural network. Claim 3 In claim 1, the step of updating one or more parameters of the target neural network is, The method includes a step of determining an updated value of one or more parameters of a target neural network according to, wherein A computer implementation method for training a neural network characterized in that represents the parameters of the target neural network, θ represents the parameters of the online neural network, and τ is the decay rate. Claim 4 A computer-implemented method for training a neural network according to claim 1, wherein the online neural network and the target neural network each include an individual encoder neural network, and the method further comprises the step of performing an operation including receiving a transformed view of a training data item by each encoder neural network; and processing the transformed view of a training data item to generate a representation of the training data item. Claim 5 A computer implementation method for training a neural network, characterized in that, in paragraph 4, each encoder neural network includes a residual neural network. Claim 6 A computer implementation method for training a neural network, characterized in that, in claim 4, the online neural network and the target neural network each include an individual projection neural network, and the method further includes the step of performing operations, such as receiving a representation of a training data item by each projection neural network; and processing the representation of the training data item to reduce the dimensionality of the representation. Claim 7 A computer implementation method for training a neural network, characterized in that, in claim 6, each projection neural network comprises a multi-layer perceptron. Claim 8 A computer-implemented method for training a neural network according to claim 4, wherein the online neural network and the target neural network have the same architecture except that the online neural network further includes a prediction neural network not included in the target neural network, and the method further comprises the step of performing operations including receiving a representation of a training data item by the prediction neural network; and processing the representation of the training data item using a regression model implemented with parameters of the prediction neural network to generate a prediction of a target output. Claim 9 A computer implementation method for training a neural network, characterized in that, in claim 8, the predictive neural network includes a multilayer perceptron. Claim 10 A computer implementation method for training a neural network according to claim 1, further comprising the step of initializing parameters of an online neural network and / or a target neural network to random values. Claim 11 A computer-implemented method for training a neural network according to claim 1, further comprising the step of applying a first data item transformation to a training data item to generate a first transformation view of the training data item; and the step of applying a second data item transformation to a training data item to generate a second transformation view of the training data item, wherein the second data item transformation is different from the first data item transformation. Claim 12 A computer-implemented method for training a neural network according to claim 1, wherein the step of updating one or more parameters of the online neural network comprises: a step of normalizing a prediction of a target output; and a step of minimizing a squared error between the normalized prediction of a target output and the target output. Claim 13 A computer-implemented method for training a neural network, characterized in that, in claim 1, it further comprises the steps of: processing a first transformation view of a training data item with a target neural network and, for each training data item in a batch containing a plurality of training data items, processing a second transformation view of a training data item with an online neural network; and repeatedly performing the operations of updating one or more parameters of an online neural network and updating one or more parameters of a target neural network. Claim 14 A computer-implemented method for processing a data item according to claim 1, wherein the training data item includes an image, and the method further comprises the step of generating a pixel representation of an input image using an encoder neural network of a trained online neural network; and the step of using the representation to perform an image processing task. Claim 15 A system comprising one or more computers and one or more storage devices that store instructions that, when executed by one or more computers, cause one or more computers to perform the operations of any one of the individual methods of claims 1 through 14. Claim 16 A computer-readable instruction or one or more computer storage media storing instructions that, when executed by one or more computers, cause one or more computers to perform the operations of any one of the individual methods of claims 1 through 14. Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images