Pre-training object detection models

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training an object detection model In particular, a system performs detection-oriented pre-training of the object detection model by pre-training at least a set of detection heads that output level-specific detection embeddings on image-text pairs.

Description

PRE-TRAINING OBJECT DETECTION MODELSCROSS-REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 586,328, filed on September 28, 2023, the disclosure of which is hereby incorporated by reference in its entirety.BACKGROUND

[0002] This specification relates to training neural networks, and to the use of trained neural networks.

[0003] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current value inputs of a respective set of parameters.SUMMARY

[0004] This specification describes a system implemented as computer programs on one or more computers in one or more locations that trains an object detection model for detecting objects from an input image. The object detection model includes (i) an image encoder neural network, which is configured to generate image embeddings of the input image and (ii) a set of one or more detection neural networks (e.g., multiple detection neural networks), one for each of a set of feature levels (of an image), configured to generate detection outputs.

[0005] A “feature level” as used in this specification can refer to a specific spatial resolution or scale at which features in an image are extracted and processed. For example, a “higher” feature level can refer to a level that corresponds to coarser image features representing more global information, while a “lower” feature level can refer to a level that corresponds to image features with finer spatial details capturing more localized textures and patterns. Thus a feature level of an image can refer to a resolution of a feature map of the image, and the set of one or more feature levels, can define a set of one or more feature mapsof the image at one or more different resolutions. Each detection neural network can process the image embeddings on a particular feature level, e.g. at a particular resolution of a feature map corresponding to the feature level. The image embeddings at a particular feature level, i.e. the level-specific image embeddings, may be defined by, e.g. comprise, features of the feature map at the particular feature level.

[0006] An “embedding” as used in this specification is a vector of numeric values, e.g., floating point values or other values, having a predetermined dimensionality. The space of possible vectors having the predetermined dimensionality is referred to as the “embedding space.”

[0007] In this specification, a “patch” of an image is a region of an image, e.g., so that the image is divided into non-overlapping regions called “patches.”

[0008] In one aspect, this specification describes a training method for training an object detection model. The method is implemented by a system including one or more computers.

[0009] The object detection model includes (i) an image encoder neural network having a set of image encoder parameters and (ii) for each of a set of one or more feature levels, a respective detection neural network having a respective set of detection parameters, the method including performing a pre-training of the image encoder neural network and the detection neural networks. The object detection model can be configured to process an image input to generate an output, e.g. a set of object detection embeddings, that can be used for performing an object detection task, e.g. by computing a similarity between the embeddings and a query embedding corresponding to a category of object to be detected.

[0010] The system obtains a batch of training examples, where each training example includes a respective training image and a respective text segment.

[0011] For each training example, the system processes, using the image encoder neural network and in accordance with current values of the image encoder parameters, the training image in the training example to generate, at each respective feature level of the set of feature levels, a respective set of level-specific image embeddings for the training image at the respective feature level; for each respective feature level of the set of feature levels, processes, using the respective detection neural network in accordance with current values of the respective set of detection parameters, the respective set of level-specific imageembeddings to generate a respective level-specific detection embedding; and processes the respective text segment in the training example using a text encoder neural network to generate a text embedding of the text segment.

[0012] For each respective feature level of the set of feature levels, the system computes a respective contrastive loss using (i) the respective level-specific detection embeddings generated for the batch of training examples at the respective feature level and (ii) the respective text embeddings of the text segments in the training examples. A contrastive loss can be a loss that is dependent upon a positive example and one or more negative examples. A positive example can comprise embeddings for an image and a text segment that are from the same training example (training pair), e.g. a training image and the respective text segment; a negative example can comprise embeddings for an image and a text segment that are from the different training examples. More specifically the contrastive loss can minimize a similarity, e.g. distance, between the embeddings in a positive example and maximize a similarity, e.g. distance, between the embeddings in a negative example (e.g. by having respective terms dependent on these). Many different contrastive losses are known.

[0013] The system updates at least the sets of detection parameters by training the detection neural networks on the contrastive losses for the set of feature levels.

[0014] In some implementations of the training method, to train the detection neural networks on the contrastive losses, the system jointly trains at least the image encoder neural network and the detection neural networks on the contrastive losses to update the set of image encoder parameters and the sets of detection parameters.

[0015] In some implementations of the training method, to process the training image to generate, at each respective feature level of the set of feature levels, the respective set of level-specific image embeddings for the training image at the respective feature level, the system processes, using the image encoder neural network and in accordance with current values of the image encoder parameters, the training image to generate a training feature map including a set of image embeddings of the training image; and for each respective feature level of the set of feature levels, the system processes the training feature map to generate the level-specific image embeddings for the training image at the respective feature level.

[0016] In some implementations of the training method, processing the training image to generate the training feature map includes processing the training image using a vision Transformer neural network.

[0017] In some implementations of the training method, processing the training feature map to generate the respective set of level-specific image embeddings includes: performing a convolution or deconvolution operation on one or more input maps generated from the training feature map to generate a respective set of first embeddings; and generating the respective set of level-specific image embeddings from the respective set of first embeddings, wherein a stride of the convolution or deconvolution operation depends on the respective feature level.

[0018] In some cases, the one or more input maps can be the same as the training feature map. In some cases, the respective set of first embeddings can form a respective levelspecific feature map.

[0019] In some cases, processing the training feature map to generate the level-specific image embeddings further includes: identifying, a respective set of regions, here termed regions of interest (Rols), of the level-specific feature map for the respective feature level.

[0020] In some cases, identifying the respective set of Rols of the training feature map includes: randomly sampling the respective set of Rols from the respective level-specific feature map. In some cases, the number of the respective set of Rols depends on the respective feature level.

[0021] In some cases, processing the training feature map to generate the level-specific image embeddings further includes: extracting a respective set of Rol features from the respective level-specific feature map according to the respective set of sampled Rols, and performing a feature alignment operation on the respective set of Rol features to generate the respective set of level-specific image embeddings.

[0022] In some implementations of the training method, processing the respective set of level-specific image embeddings to generate the respective level-specific detection embedding includes: processing, using the respective detection neural network in accordance with current values of the respective set of detection parameters, the respective set of levelspecific image embeddings to generate a respective set of detector outputs; and performingmax pooling on the respective set of detector outputs to generate the level-specific detection embedding.

[0023] In some implementations of the training method, processing the training image to generate the training feature map includes: dividing the training image into a plurality of patches; processing the plurality of patches to generate a first feature map including a plurality of patch embeddings; generating a second feature map by shifting the patch embeddings in the first feature map, along each of a width dimension and a height dimension of the first feature map, by a predefined number of pixels; processing the first feature map using a set of layers including one or more attention layers of the image encoder neural network to generate a third feature map; processing the second feature map using the set of layers of the image encoder neural network to generate a fourth feature map; generating a fifth feature map by shifting the patch embeddings in the fourth feature map, along each of a width dimension and a height dimension of the fourth feature map, in directions that are opposite of the shifting directions when generating the second feature map; generating a combined feature map by combining the third feature map and the fifth feature map; and generating the training feature map from the combined feature map.

[0024] In some cases shifting the patch embeddings in the first feature map along each of the width dimension and the height dimension of the first feature map includes: performing circular shifting along each of the width dimension and the height dimension of the first feature map by the predefined number of pixels.

[0025] In some cases, generating the combined feature map includes averaging the third feature map and the fifth feature map.

[0026] In some cases, the set of layers of the image encoder neural network includes one or more windowed-attention layers that operate on a predefined window size, and the predefined number of pixels for the shifting is a fraction of the predefined window size.

[0027] In some implementations of the training method, training the detection neural networks on the contrastive losses includes: jointly training the image encoder neural network, the text encoder neural network, and the detection neural networks on the contrastive losses to update the set of image encoder parameters, a set of text encoder parameters of the text encoder neural network, and the sets of detection parameters.

[0028] In some cases, the pre-training is a second pre-training, the contrastive losses are second contrastive losses, and the method further includes, before performing the second pretraining, performing a first pre-training including: training at least a portion of the image encoder neural network on a first contrastive loss.

[0029] In some cases, the first pre-training does not update the detection parameters of the detection neural networks.

[0030] In some implementations of the training method, the method further includes: after performing the pre-training of the image encoder neural network and the detection neural networks, training the object detection model to perform an object detection task using a set of labeled training examples.

[0031] In some cases, the object detection task is open vocabulary object detection. The object detection task can be to detect an object defined by a text query. Open vocabulary object detection can be object detection in which the text queries are not limited to text queries (text segments) used in training or, put differently, an object detection task in which an object be to detected is defined by a text query that is not from a pre-defined vocabulary of text queries.

[0032] In some implementations of the training method, the set of feature levels includes a plurality of feature levels.

[0033] In another aspect, this specification describes an object detection method. The method is implemented by a system including one or more computers.

[0034] The system obtains (i) an input image and (ii) a set of one or more query embeddings, wherein each query embedding represents a respective category of object. The system processes the input image using the image encoder neural network and the detection neural networks that have been trained using the method of any of the preceding claims, to generate a set of object detection embeddings. For each respective category of object, the system computes a respective detection score based on similarities between the set of object detection embeddings and the respective query embedding corresponding to the respective category of object.

[0035] In another aspect, this specification describes a system including one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the training method or the object detection method described above.

[0036] In another aspect, this specification describes one or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the training method or the object detection method described above.

[0037] In another aspect, this specification describes a computer-implemented system for performing object detection. The system includes an image encoder neural network configured to receive a first input specifying an input image (e.g. comprising an image, e.g. defined by values of pixels of the image), and process the first input to generate an image feature map including a set of image embeddings of the input image; a neural network, that may be referred to as a (simple) feature pyramid network, configured to, for each respective feature level of a set of feature levels, process a respective second input generated from the image feature map to generate a respective set of level-specific image embeddings for the input image; a set of detection neural networks that include, for each respective feature level of the set of feature levels, a respective detection neural network configured to process a respective third input generated from the respective set of level-specific image embeddings to generate a respective set of level-specific detection embeddings; one or more pooling layers configured to combine each respective set of level-specific object detection embeddings to generate a respective level-specific combined detection embedding; a text encoder neural network configured to process a set of one or more text queries to generate a set of one or more query embeddings, wherein each query embedding represents a respective category of object; and a detection unit configured to, for each respective category of object, compute a respective detection score based on similarities between the level-specific combined detection embeddings and the respective query embedding corresponding to the respective category of object. The simple feature pyramid network can generate each respective feature level of the set of feature levels from the same common (or main) feature map, e.g. by applying convolutions (to obtain a reduced resolution feature map) and or deconvolutions (to obtain an increased resolution feature map).

[0038] In some implementations of the computer-implemented system, the respective third input is generated by: identifying, a respective set of regions of interest (Rols) of a respective level-specific feature map formed by the respective set of level-specific image embeddings; extracting a respective set of Rol features from the respective level-specific feature map according to the respective set of sampled Rols; and performing a feature alignment operation on the respective set of Rol features.

[0039] In some implementations of the computer-implemented system, the image encoder neural network configured to: divide the input image into a plurality of patches; process the plurality of patches to generate a first feature map including a plurality of patch embeddings; generate a second feature map by shifting the patch embeddings in the first feature map, along each of a width dimension and a height dimension of the first feature map, by a predefined number of pixels; process the first feature map using a set of layers of the image encoder neural network to generate a third feature map; process the second feature map using the set of layers of the image encoder neural network to generate a fourth feature map; generate a fifth feature map by shifting the patch embeddings in the fourth feature map, along each of a width dimension and a height dimension of the fourth feature map, in directions that are opposite of the shifting directions when generating the second feature map; generate a combined feature map by combining the third feature map and the fifth feature map; and generate the image feature map from the combined feature map.

[0040] In some cases, the set of layers of the image encoder neural network includes one or more windowed-attention layers that operate on a predefined window size, and the predefined number of pixels for the shifting is a fraction of the predefined window size.

[0041] In some implementations of the computer-implemented system, the image encoder neural network and the detection neural networks have been trained using the training method described above.

[0042] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0043] Detecting and localizing objects and entities in the visual world has a wide range of applications, including, for example, self-driving vehicles, robotic systems, and search and recommendation systems. Conventional techniques for training object detectors typically rely on human-annotated regions and class labels. These annotations are costly to collect andunscalable in terms of the number of categories and the number of images. On the other hand, contrastive learning techniques using a large image-text corpus can be used to learn representations of images that yield significant improvements when the representations are used for downstream tasks, e.g., image classification, image captioning, text-to-image search, and so on.

[0044] However, in the conventional techniques, although the image encoding network is pre-trained using contrastive learning techniques, the detector heads are typically trained from scratch using supervised training. This often leads to suboptimal performance since the detector heads are trained on a limited vocabulary of the training dataset without gaining the open-vocabulary knowledge provided by contrastive learning.

[0045] To bridge this gap, this specification provides techniques for image-language pretraining tailored to open-vocabulary object detection. Unlike standard pre-training, the provided techniques pre-train the image encoding network and the detector heads jointly using the contrastive learning approach. In some implementations, the detector heads receive text supervision at multiple feature levels by pooling over randomly generated Rols, facilitating image-text representation across different semantic levels.

[0046] In addition, in some implementations, the provided techniques use a Shifted- Window Learning (SWL) technique to enhance the performance of ViT using windowed attention. By shifting patch embeddings with smaller strides than the window size, the technique results in a more shift-invariant representation, reducing bias from window attention patterns.

[0047] These techniques lead to enhanced performance in object detection tasks by pretraining the detector heads through contrastive learning. Additionally, the image embeddings produced by the image encoding network are less adversely affected by window attention patterns, further contributing to the overall improvement in performance.

[0048] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0049] FIG. 1 shows an example machine learning system for object detection.

[0050] FIG. 2 shows an example neural network system.

[0051] FIG. 3 illustrates an example process for pre-training an object detection model.

[0052] FIG. 4 illustrates an example of a Shifted-Window Learning (SWL) process.

[0053] FIG. 5 is a flow diagram of an example process for training an object detection model.

[0054] FIG. 6 is a flow diagram of an example process for performing object detection using an object detection model.

[0055] FIG. 7 shows a performance comparison of object detection models trained using different methods.

[0056] Like reference numbers and designations in the various drawings indicate like elements.DETAILED DESCRIPTION

[0057] FIG. 1 shows an example machine learning system 100. The machine learning system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0058] The system 100 includes an object detection model 140 configured to detect objects from an input image 150a. The system further includes a training system 110 configured to perform training of the detection model 140, i.e., update values of model parameters 145 of the detection model 140.

[0059] In general, the object detection model 140 is configured to identify and localize objects within one or more input images 150a. That is, the object detection model 140 is configured to process the input image 150a to generate the detection output 160 that identifies one or more objects in the input image 150a, such as by determining the class of a detected object. In some instances, the detection output 160 also includes spatial location information (e.g., as a bounding box) that indicates the position of the detected objects within the input image 150a.

[0060] In some implementations, the object detection model 140 can be configured to perform open-vocabulary object detection. Open-vocabulary object detection is a type of object detection where the model is capable of detecting objects belonging to object categories that the model has not been explicitly trained on. In other words, the model can generalize to new or unseen object categories. Unlike traditional object detection models, which are trained and constrained to recognize a fixed set of categories, an open-vocabulary object detection model can generalize to new or unseen categories by leveraging information such as textual descriptions, embeddings, or semantic relationships between objects. This allows the model to identify and localize objects across a broad and potentially unlimited range of categories, including those not present in the labeled training data.

[0061] For example, the object detection model 140 can receive the input image 150a as well as a set of one or more query input(s) 150b, e.g., text queries, representing a set of object categories. The object detection model 140 processes the input image 150a and the set of query inputs 150b to determine, for each particular object category, a respective detection score characterizing the likelihood or confidence that the particular object category is present in the input image 150a. One example of performing open-vocabulary object detection will be described below with reference to FIG. 6.

[0062] Examples of the model architecture of the object detection model 140 will be described in further detail below with references to FIGs. 2-4. In general, the object detection model 140 includes an image encoder neural network and a set of detection neural networks (also referred to as detection heads in this Specification). The image encoder processes the input image to generate image embeddings, while the detection neural networks operate on the image embeddings at multiple feature levels to produce detection embeddings.

[0063] In the context of this Specification, a feature level can refer to a specific scale or resolution at which an image is represented or analyzed. For example, a “higher” feature level can be defined as a level that corresponds to coarser, higher-level image features. Each detection neural network processes the image embeddings on a particular feature level, enabling the model 140 to capture image information at various scales and detect objects of different sizes. The detection embeddings from these networks are then combined and processed to generate the detection output 160.

[0064] The techniques and architecture described in this specification, while primarily focused on open-vocabulary object detection, can also be adapted and applied to other computer vision tasks. As an example, the task can be semantic segmentation. In semantic segmentation, the input is an image or a set of multiple images and the output assigns each of a plurality of pixels in the input image(s) to a respective object category from a set of object categories.

[0065] In another example, the task can be instance segmentation. In instance segmentation, the input is an image or a set of multiple images and the output assigns each of a plurality of pixels in the input image(s) to a respective object instance, with two pixels that are assigned the same instance depicting the same object instances and two pixels that are assigned different instances depicting different object instances.

[0066] In another example, the task can be panoptic segmentation. In panoptic segmentation, the input is an image or a set of multiple images and the output assigns each of a plurality of pixels in the input image(s) to a respective object instance and to a respective object category.

[0067] The training system 110 is configured to perform training of the detection model 140, i.e., by updating values of model parameters 145 of the detection model 140. For example, these model parameters 145 can include network parameters of the image encoder neural network and network parameters of the set of detection neural networks.

[0068] The training system 110 includes a pre-training engine 120 configured to perform pre-training of the object detection model 140 on pre-training data, which includes a set of training examples 125. Each training example 125 includes an image-text pair, i.e., a respective training image 125a and a respective text segment 125b.

[0069] In general, the pre-training engine 120 can use a large dataset of image-text pairs. The dataset can provide a rich variety of visual and textual information, enabling the model 140 to learn generalized representations of images and their associated descriptions.

[0070] The pre-training engine 120 performs contrastive learning to update the parameters of the model 140. Unlike conventional methods where the image encoder network is pretrained separately and the detection heads are trained from scratch with supervised learning, the pre-training engine 120 trains both the image encoder network and the detection heads, e.g., jointly, using contrastive learning on the pre-training dataset. These techniques lead toenhanced performance in object detection tasks by pre-training the detector heads through contrastive learning.

[0071] In some implementations, the training system 110 further includes a supervised training engine 130 configured to, following the pre-training, perform supervised training of the object detection model 140 on labeled data, which includes a set of training examples 135. Each training example 135 includes a respective training image 135a and a respective detection label 135b.

[0072] The labeled dataset used by the supervised training engine 130 is typically smaller (i.e., has fewer training examples) than the pre-training dataset. In some cases, the labeled dataset can be labeled for a particular task. For example, for an object detection task, the labeled dataset can be annotated with detection labels that define object classes, and optionally, spatial locations of objects within the images.

[0073] The operations of the training system 110 will be described in more detail with references to FIGs. 2 and 5 below.

[0074] FIG. 2 shows an example neural network system 200. The neural network system 200 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented. The neural network system 200 is a system that performs pre-training of an object detection model, e.g., the object detection model 140 described with reference to FIG. 1.

[0075] The object detection model includes (i) an image encoder neural network 210 and (ii) a set of detection heads 240. The neural network system 200 can also include a text encoder network 230 that generates text embeddings 235 from text segments. The system 200 trains the object detection model on contrastive losses 250 computed using the detection embeddings 245 generated by the detection heads 240 and the text embeddings 235 generated by the text encoder network 230.

[0076] To pre-train the object detection model, the system receives training data including a set of training examples 225. Each training example 225 includes a training input image 225a and a corresponding input text segment 225b. That is, training data 120 for the pretraining can include multiple image-text training pairs.

[0077] In particular, the input text segment 225b in a given pair has been determined by the system 200 or an external source to describe the contents of the image 225a in the given pair or otherwise be relevant to the image 225a in the given pair. In other words, the image 225a and the input text segment 225b in each pair have been determined to be semantically similar to one another.

[0078] For example, within a given training pair, the text segment 225b can be a text annotation of the image 225a from a set of manually or automatically generated image annotations or can be alt text associated with the image 225a in a set of alt-text data. Alt text is text that is displayed in place of an image on a web page, e.g., if the image cannot be rendered properly or otherwise fails to load. For example, the system 200 can obtain the alt- text data from data maintained by an Internet search engine or other software that automatically crawls web pages on the Internet.

[0079] The image encoder neural network 210 is a neural network that receives an input image 225a and processes the input image 225a to generate sets of image embeddings 215 of the input image 225a in an embedding space. In particular, the sets of image embeddings 215 are level-specific embeddings for a set of feature levels. The set of feature levels can represent multiple scales or resolutions at which an input image is represented or analyzed. For each of the set of feature levels, the image encoder 210 generates a respective set of level-specific image embeddings 215 for the training image 225a.

[0080] The set of detection heads 240 are level-specific. That is, each detection head 240 corresponds to a different feature level and processes the set of level-specific image embeddings 215 for that particular level to generate a level-specific detection embedding 245 for the corresponding level.

[0081] The text encoder network 230 is a neural network that processes each text segment 225b in the training examples 225 to generate a corresponding text embedding 235. In general, the text embedding 235 generated by the text encoder network 230 is in the same embedding space as the detection embeddings 245. That is, each detection embedding 245 and the text embedding 235 have the same dimensionality.

[0082] The text encoder neural network 230 can have any appropriate architecture that allows the text encoder neural network 230 to map a text sequence to a text embedding. In a particular example, the text encoder neural network 230 can have an attention-basedarchitecture, e.g., the architecture of an encoder-only, encoder-decoder, or decoder-only Transformer neural network. In this example, the text encoder neural network 230 can include a sequence of layers that includes one or more self-attention layers, where each attention layer is configured to receive as input a respective current representation of each of the text tokens in the current text sequence and to process the respective current representations to generate as output a respective updated representation of each of the text tokens in the current text sequence by applying self-attention mechanism over the respective current representations.

[0083] For each batch of training examples 225, the system 200 calculates a set of contrastive losses 250, one for each feature level. In general, a contrastive loss is one that encourages embeddings for an image and a text that are from the same pair to be closer (more similar) than embeddings for an image and a text that are from different pairs. The set of contrastive losses 250 are based on the level-specific detection embeddings 245 (generated from the input images 225a) and the text embeddings 235 (generated from the text segments 225b). Each contrastive loss quantifies the similarity between the detection embeddings at a specific feature level and the corresponding text embeddings.

[0084] In some implementations, the contrastive loss 250 at each feature level is computed using the dot products of the level-specific detection embeddings {v} and the text embeddings {£} for a training batch of size ‘B’. A specific example of such a loss (in this example, an InfoNCE loss) is:where T is a temperature that scales the logits.

[0085] Minimizing this contrastive loss LConencourages the embeddings of matching image-text pairs ( t and It) to become closer in the embedding space while pushing them further apart from the embeddings of other non-matching pairs within the batch. This process achieves the objective of contrastive learning, which is to learn representations that capture the semantic relationships between images and text.

[0086] The system 200 can use these contrastive losses to train at least the detection heads 240. This training process involves backpropagating the gradients of the contrastive loss through the corresponding detection head at each feature level, and then updating the parameters of that detection head using an optimizer, such as the stochastic gradient descent (SGD) or AdamW,

[0087] In some cases, before using the contrastive losses described above to train the object detection model, the system 200 can separately train at least a portion of the image encoder neural network 210 using another contrastive loss. This other contrastive loss is different from the contrastive losses 250 computed using the detection embeddings 245. For example, this other contrastive loss can characterize the similarities between the text embeddings 235 and embeddings generated using the image encoder 210. The gradients of this loss are used to update the parameters of the relevant portion of the image encoder 210.

[0088] In some other cases, the system 200 jointly trains the image encoder neural network 210 and the detection heads on the contrastive losses 250 to update the parameters of the image encoder network 210 and parameters of the detection heads 240. The system 200 can use backpropagation to compute the gradients of the contrastive losses 250 with respect to the parameters of both the image encoder 210 and the detection heads 230. In some cases, a total contrastive loss can be used. The total contrastive loss can be a weighted sum of the levelspecific contrastive losses 250. The system then uses an optimizer to update the parameters of both the image encoder 210 and the detection heads 240 based on the computed gradients.

[0089] In some cases, the joint training can be further extended to include the text encoder neural network 230. This joint training updates the parameters of the image encoder 210, the text encoder 230, and the detection heads 240 based on the contrastive losses 250.

[0090] The image encoder neural network 210 can have any appropriate architecture that allows the neural network 210 to map the input image 225a to the level-specific image embeddings 215. These embeddings capture the visual information of the input image at different scales or levels of detail, enabling effective object detection of objects of different sizes.

[0091] In some cases, the image encoder 210 can include (i) a main neural network that processes the input image 225a to generate a main feature map that includes a set of image embeddings of the input image at a single feature level, and (ii) additional components thatprocess the main feature map to generate the level-specific image embeddings 215 at various feature levels.

[0092] The main neural network can have any appropriate architecture. For example, the image encoder neural network 210 can include a vision Transformer (ViT) neural network as the main network for generating the main feature map. The ViT has one or more selfattention layer blocks that each include one or more self-attention layers. Each self-attention layer receives a respective input embedding for an image patch and applies a self-attention mechanism over the respective input embeddings to update the input embeddings.

[0093] As another example, the image encoder neural network 210 can include a convolutional neural network (CNN) as the main network. As yet another example, the image encoder neural network 210 can include a neural network that has a mix of both convolutional and self-attention layers.

[0094] As a particular example of the additional components, the image encoder 210 can include a simple feature pyramid network (FPN) that processes the main feature map to generate the level-specific image embeddings 215. Detailed descriptions of the simple FPN can be found in Li, et al.,” Exploring plain vision transformer backbones for object detection,” European Conference on Computer Vision, October 23-27, 2022, Proceedings, Part IX, Pages 280 - 296. In general, the simple FPN performs a set of convolution and / or deconvolution operations on the main feature map to generate the level-specific image embeddings 215 at different feature levels. The stride of each convolution / deconvolution operation is adjusted based on the desired feature level, allowing for the extraction of features at different scales. For instance, the simple FPN can use convolutional layers with a stride greater than 1 to reduce the spatial resolution of the main feature map and generate a levelspecific feature map (which corresponds to a set of level-specific image embeddings) that captures more global context and high-level semantic information. A larger stride of the convolution operation generates coarser, higher-level features. The simple FPN can use deconvolutional layers to increase the spatial resolution of the main feature map, and generate level-specific image embeddings that capture finer details and lower-level features. By combining these operations with varying strides, the Simple FPN can generate a pyramid of level-specific feature maps at multiple scales. The system 200 generates each set of levelspecific embeddings from the corresponding level-specific feature map.

[0095] Each detection head 240 can be a neural network with any appropriate architecture. For example, the detection heads 240 can be mask R-CNN heads, faster R-CNN heads, or other appropriate detector heads that can be used as part of an object detection neural network. Examples of object detection neural networks with faster R-CNN heads can be found in Ren, et al., “Faster r-cnn: Towards real-time object detection with region proposal networks,” NeurlPS, 2015.

[0096] In some cases, the level-specific image embeddings 215 are Region of Interest (Rol) embeddings. In these cases, for each feature level, the system randomly samples a respective set of box regions (potential object locations) as Rols across the feature map. The size and aspect ratio of these boxes can be sampled from a predefined distribution, e.g., a uniform distribution, ensuring diversity in the regions considered. The number of regions sampled for each level can depend on the corresponding feature level. For example, the system can sample fewer regions in the feature map of a higher feature level. The system 200 can extract embeddings from these sampled Rols as the Rol embeddings. The image encoder neural network can include an Rol-Align layer that operates on the extracted Rol embeddings to align the embeddings to features from the portions of the feature map that correspond to the original Rols. Details of the Rol-Align operations can be found in He, et al., “Mask R-CNN,” 2017 IEEE International Conference on Computer Vision (ICCV).

[0097] Each detection head 240 can process the set of Rol embeddings (after the Rol-Align operation) for the set of Rols of the corresponding feature level to generate a set of detector outputs, and perform max pooling on the set of detector outputs to generate the level-specific detection embedding 245.

[0098] FIG. 3 illustrates a particular example of the system’s operation during pre-training. In this example, the image encoder includes a ViT encoder 310 and a simple FPN 312.

[0099] The ViT encoder 310 processes the input image 325a to generate a main feature map that captures visual information across the input image 325. The simple FPN 312 then performs a set of convolution and deconvolution operations on the main feature map to produce a set of level-specific feature maps 314, each corresponding to different feature level. The system randomly samples a respective set of Regions of Interest (Rols) from each of these level-specific feature maps 314. Next, the system applies Rol Align operations on the Rol features extracted from the sampled Rols to produce aligned Rol features, which serve asinputs to the detection heads 340. Each detection head 340 processes its respective aligned Rol features to generate a set of detector outputs, which are then passed through a maxpooling layer to produce the level-specific detection embedding 345.

[0100] The system also processes the input text segment 325b using the text encoder 330 to generate the text embedding 335, and computes the level-specific contrastive losses using the text embedding 335 and the level-specific detection embedding 345.

[0101] The system also processes the input text segment 325b using the text encoder 330, generating a corresponding text embedding 335. The system then computes the level-specific contrastive losses 350 by comparing the text embedding 335 with the level-specific detection embeddings 345. The level-specific contrastive losses 350 are then used to update the parameters of at least a portion of the object detection model, as described above with reference to FIG. 2.

[0102] FIG. 4 illustrates a particular implementation of the image encoder described with reference to FIG. 2 and FIG. 3, showing a Shifted-Window Learning (SWL) technique implemented within the ViT encoder 410. In this scenario, the image encoder includes both the ViT encoder 410 and a Simple FPN 412, similar to the architecture shown FIG. 3.

[0103] The input image 425 is divided into multiple patches. These patches are then processed to generate a first feature map 426a, which includes a collection of patch embeddings. Next, a second feature map 426b is generated by shifting the patch embeddings in the first feature map 426a along both its width and height dimensions by a predefined number of pixels. This shifting can be implemented, for example, as a circular shift, where elements that move beyond the boundary are reintroduced from the opposite side.

[0104] The first feature map 426a is then processed using a set of window attention layers within the ViT encoder 410, resulting in a third feature map 411a. These windowed-attention layers operate on a predefined window size, and the predefined number of pixels used for the shifting is a fraction of this window size.

[0105] Similarly, the second feature map 426b is processed using the same window attention layers to generate a fourth feature map. Subsequently, a fifth feature map 41 lb is generated by shifting the patch embeddings in the fourth feature map in the opposite direction compared to the shift applied when generating the second feature map 426b. Next, the thirdfeature map 411a and the fifth feature map 411b are combined (e.g., by averaging) to produce the main feature map. This main feature map is then further processed by the Simple FPN 412 and the detection heads 440 to generate the detection embeddings 445, as described in detail with reference to FIG. 2 and FIG. 3.

[0106] The SWL technique shifts patch embeddings with smaller strides than the window size. This results in a more shift-invariant representation and reduces bias resulted from window attention patterns.

[0107] FIG. 5 is a flow diagram of an example process 500 for training an object detection model, e.g., the object detection model described with references to FIGs. 1-4.

[0108] For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a computer- implemented system, e.g., the training system 110 depicted in FIG. 1, or the neural network system 200 depicted in FIG. 2, appropriately programmed in accordance with this specification, can perform the process 500.

[0109] The object detection model includes (i) an image encoder neural network having a set of image encoder parameters and (ii) for each of a set of one or more feature levels, a respective detection neural network having a respective set of detection parameters. In some cases, the set of feature levels includes a plurality of feature levels.

[0110] At 510, the system obtains a batch of training examples. Each training example includes a respective training image and a respective text segment.

[0111] As described with reference to FIG. 2, the training image and the text segment in a given training example have been determined to be semantically similar to one another. For example, the text segment can describe the contents of the training image or otherwise be relevant to the training image in the given training example.

[0112] The system then proceeds to pre-train the object detection model using the training examples, as outlined in steps 520 through 560.

[0113] Steps 520 to 540 are executed for each training example within the batch.

[0114] At 520, the system processes, using the image encoder neural network and in accordance with current values of the image encoder parameters, the training image in the training example to generate, at each respective feature level of the set of feature levels, a respective set of level-specific image embeddings for the training image at the respective feature level.

[0115] As described with references to FIG. 2 and FIG. 3, to generate the sets of levelspecific image embeddings, in some implementations, the system uses the image encoder neural network to process the training image to generate a training feature map (e.g., the main feature map) that includes a set of image embeddings of the training image. For example, the system can use a vision Transformer neural network to generate the training feature map.

[0116] The system then processes the training feature map to generate the sets of levelspecific image embeddings. For example, the system can use a simple FPN to perform convolution or deconvolution operations on the training feature map to generate the sets of level-specific feature map, and generate the sets of level-specific image embeddings from the level-specific feature maps. A stride of the convolution or deconvolution operation depends on the respective feature level.

[0117] As described with references to FIG. 2 and FIG. 3, in some cases, the system can identify regions of interest (Rols) within each level-specific feature map (e.g., via random sampling). Rol features are then extracted and aligned to produce the final level-specific image embeddings.

[0118] As described with reference to FIG. 4, in some cases, the system implements a Shifted-Window Learning (SWL) technique within the image encoder. In this approach, the system divides the training image into multiple patches, processes these patches to generate a first feature map including patch embeddings, and then shifts the patch embeddings along both the width and height dimensions by a predefined number of pixels to generate a second feature map. The first feature map is processed using a set of layers, including attention layers of the image encoder, to generate a third feature map. Similarly, the second feature map is processed using the same set of layers to generate a fourth feature map. The system then shifts the patch embeddings in the fourth feature map in the opposite direction to generate a fifth feature map. The third and fifth feature maps are combined to generate a combined feature map, which is then used to generate the training feature map.

[0119] At 530, for each feature level, the system uses the corresponding detection neural network, in accordance with current values of the detection parameters, to process the levelspecific image embeddings to generate a level-specific detection embedding.

[0120] As described with references to FIG. 2 and FIG. 3, in some cases, the detection neural network generates a respective set of detector outputs from the respective set of levelspecific image embeddings, and performs max pooling on the respective set of detector outputs to generate the level-specific detection embedding.

[0121] At 540, the system processes the text segment in the training example using a text encoder neural network to generate a text embedding of the text segment. As described with reference to FIG. 2, the text embedding is in the same embedding space as the detection embeddings. That is, each detection embedding and the text embedding have the same dimensionality.

[0122] At 550, for each feature level, the system computes a respective contrastive loss using (i) the respective level-specific detection embeddings generated for the batch of training examples at the respective feature level and (ii) the respective text embeddings of the text segments in the training examples.

[0123] As described with reference to FIG. 2, each contrastive loss quantifies the similarity between the detection embeddings at a specific feature level and the corresponding text embeddings in the embedding space.

[0124] At 560, the system updates at least the sets of detection parameters by training the detection neural networks on the contrastive losses for the set of feature levels. This process involves backpropagating the gradients of the contrastive loss through the corresponding detection head at each feature level, and then updating the parameters of that detection head using an optimizer, such as the stochastic gradient descent (SGD) or AdamW,

[0125] As described with reference to FIG. 2, in some implementations, the system jointly trains at least the image encoder neural network and the detection neural networks on the contrastive losses to update the set of image encoder parameters and the sets of detection parameters. In some cases, the system jointly trains the image encoder neural network, the text encoder neural network, and the detection neural networks on the contrastive losses to update the set of image encoder parameters, a set of text encoder parameters of the text 1encoder neural network, and the sets of detection parameters. The system can repeat steps 510-560 for multiple batches of training examples. That is, the system can repeatedly perform the process 500 on different batches of training examples sampled from a larger training set to repeatedly update the model parameters.

[0126] In some implementations, before training the detection neural networks using the contrastive loss computed at 550, the system separately trains at least a portion of the image encoder neural network on a different contrastive loss, e.g., computed using the text embedding and an embedding generated by the image encoder. In this separate training, the system does not update the detection parameters of the detection neural networks.

[0127] In some implementations, after performing the pre-training described above, the system further performs supervised training of the object detection model for performing an object detection task, e.g., an open vocabulary object detection task.

[0128] The supervised training is performed using a set of labeled training examples. Each labeled training example includes a respective training image and a respective detection label. For example, for an object detection task, the labeled dataset can be annotated with detection labels that define object classes, and optionally, spatial locations of objects within the images.

[0129] FIG. 6 is a flow diagram of an example process 600 for performing object detection using an object detection model that has been trained, e.g., using a process described with references to FIG. 2 and FIG. 5. In particular, the object detection model includes (i) an image encoder neural network and (ii) for each of a set of one or more feature levels, a respective detection neural network. In some cases, the object detection model further includes a text encoder neural network, e.g., the text encoder neural network described with reference to FIG. 2.

[0130] For convenience, the process 600 will be described as being performed by a system of one or more computers located in one or more locations. For example, a machine learning system, e.g., the machine learning system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 600.

[0131] At 610, the system obtains an input image. For example, the system can receive the input image via a user input or other means.

[0132] At 620, the system obtains a set of one or more query embeddings, where each query embedding represents a respective category of object. For example, the system can receive, e.g., via user input, a text query that specifies a set of object categories, and use the text encoder neural network to process the text query to generate the set of query embeddings.

[0133] At 630, the system processes the input image using the image encoder neural network and the detection neural networks that have been trained, to generate, for each of a set of feature levels, a respective level-specific object detection embedding.

[0134] At 630a, the system uses the trained image encoder neural network to process an input specifying the input image to generate, for each feature level, a respective set of levelspecific image embeddings.

[0135] Implementation examples of the image encoder neural network have been further described with references to FIGs. 2-4,

[0136] In some cases, the system uses the image encoder, e.g., a ViT encoder, to generate an image feature map that includes a set of image embeddings of the input image. The system uses a simple feature pyramid network (FPN) to process the image feature map to generate, for each feature level, a respective level-specific feature map that includes the respective set of level-specific image embeddings.

[0137] In some cases, regions of interest (Rols) can be identified within each level-specific feature map (e.g., through random sampling), and Rol features can be extracted and aligned to produce the level-specific image embeddings (as described with references to FIG. 2 and FIG. 3).

[0138] As described with reference to FIG. 4, in some cases, the system implements a Shifted-Window Learning (SWL) technique within the image encoder. In this approach, the system divides the training image into multiple patches, processes these patches to generate a first feature map including patch embeddings, and then shifts the patch embeddings along both the width and height dimensions by a predefined number of pixels to generate a second feature map. The first feature map is processed using a set of layers, including attention layers of the image encoder, to generate a third feature map. Similarly, the second feature map is processed using the same set of layers to generate a fourth feature map. The systemthen shifts the patch embeddings in the fourth feature map in the opposite direction to generate a fifth feature map. The third and fifth feature maps are combined to generate a combined feature map, which is then used to generate the training feature map.

[0139] At 630b, for each feature level, the corresponding detection neural network processes the level-specific image embeddings to generate the level-specific detection embedding. In some cases, this includes generating a set of detector outputs from the levelspecific image embeddings and then using one or more pooling layers to apply pooling (e.g., max pooling) to generate the level-specific detection embedding.

[0140] At 640, for each object category, the system computes a detection score based on similarities between the embeddings generated by the object detection model and the query embedding corresponding to the object category.

[0141] In some cases, the system calculates a level-specific similarity, such as cosine similarity, between each level-specific detection embedding and the query embedding. The similarity score (n) is then computed by combining these level-specific similarities. This combination can be done through various methods, such as taking the maximum or average of the level-specific similarities.

[0142] In some cases, the system can compute a similarity score n that measures the similarity between the query embedding and an overall detection embedding that represents the combined detection features across all feature levels. To generate this overall detection embedding, the system can apply pooling layers (e.g., average pooling or max pooling) to the level-specific detection embeddings.

[0143] The system can convert these similarity scores n into detection scores (pi) for the object categories. This conversion can be achieved using a softmax operation, which normalizes the scores into probabilities, ensuring they sum to 1 and represent the relative likelihood of each category being present in the image.

[0144] In some cases, in addition to the detection embeddings, the system also computes a VLM (Vision-Language Model) embedding from an image feature map outputted by the ViT backbone. The VLM score (z0 is calculated as the similarity (e.g., cosine similarity) between this VLM embedding and the query embedding. In some cases, the VLM score is computed using a “frozen” instance of the ViT backbone, where the parameters are fixed after pre-training. This helps preserve the pre-trained image-text knowledge, which might otherwise be attenuated during supervised training.

[0145] The system can then compute the final open-vocabulary detection scores (Siens) by combining the detection scores (pi) and the VLM scores (z0. One approach is to use the geometric mean for this combination. In one specific example, the final open-vocabulary detection score Siensfor the i-th object category is computed as:

[0146] Here, CBrepresents the base categories (object categories seen during supervised training), CNrepresents the novel categories (object categories unseen during supervised training), and a and P are parameters that control the weighting of base versus novel categories in the ensemble.

[0147] At 650, the system outputs a detection result, e.g., the detection scores, to an output device or a user device.

[0148] FIG. 7 shows an example 700 of a comparison of the performances of an object detection model trained using the described techniques (“DITO”) and other techniques on an open-vocabulary object detection task.

[0149] In particular, FIG. 7 shows results of the models on the Large Vocabulary Instance Segmentation (LVIS) benchmark dataset. The ‘frequent’ and ‘common’ classes in LVIS are considered base categories (seen during training), while the ‘rare’ classes are the novel categories (unseen during training). The main performance metric is the mask AP (average precision) on these rare classes, denoted as mask APr.

[0150] The best DITO model achieves a mask APr of 38.4, significantly outperforming existing state-of-the-art methods like RO-ViT and CFM-ViT, which use the same ViT-L backbone. This improvement is achieved while using the same pre-training data (ALIGN dataset) as these other methods. Furthermore, by replacing the ALIGN dataset with the DataComp-lB dataset during pre-training, DITO’s performance further increases to 40.4 for mask APr.

[0151] Even with a smaller ViT-B backbone, DITO maintains a considerable lead of around +4 APr over other ViT-B based approaches, demonstrating its effectiveness across different model sizes. The results highlight the improved performance of DITO in openvocabulary object detection, particularly its ability to generalize to novel object categories.

[0152] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0153] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0154] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an executionenvironment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0155] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0156] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[0157] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0158] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0159] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0160] Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0161] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0162] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0163] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework.

[0164] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0165] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0166] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations andeven initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0167] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0168] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

CLAIMS1. A computer-implemented method for training an object detection model, the object detection model comprising (i) an image encoder neural network having a set of image encoder parameters and (ii) for each of a set of one or more feature levels, a respective detection neural network having a respective set of detection parameters, the method comprising performing a pre-training of the image encoder neural network and the detection neural networks, the pre-training comprising: obtaining a batch of training examples, each training example comprising a respective training image and a respective text segment; for each training example: processing, using the image encoder neural network and in accordance with current values of the image encoder parameters, the training image in the training example to generate, at each respective feature level of the set of feature levels, a respective set of levelspecific image embeddings for the training image at the respective feature level; for each respective feature level of the set of feature levels, processing, using the respective detection neural network in accordance with current values of the respective set of detection parameters, the respective set of level-specific image embeddings to generate a respective level-specific detection embedding; and processing the respective text segment in the training example using a text encoder neural network to generate a text embedding of the text segment; and for each respective feature level of the set of feature levels, computing a respective contrastive loss using (i) the respective level-specific detection embeddings generated for the batch of training examples at the respective feature level and (ii) the respective text embeddings of the text segments in the training examples; and updating the sets of detection parameters by training the detection neural networks on the contrastive losses for the set of feature levels.

2. The method of claim 1, wherein training the detection neural networks on the contrastive losses comprises: jointly training at least the image encoder neural network and the detection neural networks on the contrastive losses to update the set of image encoder parameters and the sets of detection parameters.

3. The method of claim 1 or claim 2, wherein processing the training image to generate, at each respective feature level of the set of feature levels, the respective set of level-specific image embeddings for the training image at the respective feature level comprises: processing, using the image encoder neural network and in accordance with current values of the image encoder parameters, the training image to generate a training feature map comprising a set of image embeddings of the training image; and for each respective feature level of the set of feature levels, processing the training feature map to generate the level-specific image embeddings for the training image at the respective feature level.

4. The method of claim 3, wherein processing, using the image encoder neural network and in accordance with current values of the image encoder parameters, the training image to generate the training feature map comprises processing the training image using a vision Transformer neural network.

5. The method of claim 3 or claim 4, wherein processing the training feature map to generate the respective set of level-specific image embeddings comprises: performing a convolution or deconvolution operation on one or more input maps generated from the training feature map to generate a respective set of first embeddings; and generating the respective set of level-specific image embeddings from the respective set of first embeddings, wherein a stride of the convolution or deconvolution operation depends on the respective feature level.

6. The method of claim 5, wherein processing the training feature map to generate the levelspecific image embeddings further comprises: identifying, a respective set of regions of interest (Rols) of the level-specific feature map for the respective feature level.

7. The method of claim 6, wherein identifying the respective set of Rols of the training feature map comprises: randomly sampling the respective set of Rols from the respective level-specific feature map.

8. The method of claim 6 or claim 7, wherein a number of the respective set of Rols depends on the respective feature level.

9. The method of any of claims 5-8, wherein processing the training feature map to generate the level-specific image embeddings further comprises: extracting a respective set of Rol features from the respective level-specific feature map according to the respective set of sampled Rols, and performing a feature alignment operation on the respective set of Rol features to generate the respective set of level-specific image embeddings.

10. The method of any of claims 3-9, wherein processing the respective set of level-specific image embeddings to generate the respective level-specific detection embedding comprises: processing, using the respective detection neural network in accordance with current values of the respective set of detection parameters, the respective set of level-specific image embeddings to generate a respective set of detector outputs; and performing max pooling on the respective set of detector outputs to generate the levelspecific detection embedding.

11. The method of any of claims 3-9, wherein processing the training image to generate the training feature map comprises: dividing the training image into a plurality of patches; processing the plurality of patches to generate a first feature map comprising a plurality of patch embeddings; generating a second feature map by shifting the patch embeddings in the first feature map, along each of a width dimension and a height dimension of the first feature map, by a predefined number of pixels; processing the first feature map using a set of layers comprising one or more attention layers of the image encoder neural network to generate a third feature map; processing the second feature map using the set of layers of the image encoder neural network to generate a fourth feature map; generating a fifth feature map by shifting the patch embeddings in the fourth feature map, along each of a width dimension and a height dimension of the fourth feature map, in directions that are opposite of the shifting directions when generating the second feature map;generating a combined feature map by combining the third feature map and the fifth feature map; and generating the training feature map from the combined feature map.

12. The method of claim 11, wherein shifting the patch embeddings in the first feature map along each of the width dimension and the height dimension of the first feature map comprises: performing circular shifting along each of the width dimension and the height dimension of the first feature map by the predefined number of pixels.

13. The method of claim 11 or claim 12, wherein generating the combined feature map comprises: averaging the third feature map and the fifth feature map.

14. The method of any of claims 11-13, wherein the set of layers of the image encoder neural network comprises one or more windowed-attention layers that operate on a predefined window size, and the predefined number of pixels for the shifting is a fraction of the predefined window size.

15. The method of any of the preceding claims, wherein training the detection neural networks on the contrastive losses comprises: jointly training the image encoder neural network, the text encoder neural network, and the detection neural networks on the contrastive losses to update the set of image encoder parameters, a set of text encoder parameters of the text encoder neural network, and the sets of detection parameters.

16. The method of claim 15, wherein the pre-training is a second pre-training, the contrastive losses are second contrastive losses, and the method further comprises, before performing the second pre-training, performing a first pre-training comprising: training at least a portion of the image encoder neural network on a first contrastive loss.

17. The method of claim 16, wherein the first pre-training does not update the detection parameters of the detection neural networks.

18. The method of any of the preceding claims, the method further comprising: after performing the pre-training of the image encoder neural network and the detection neural networks, training the object detection model to perform an object detection task using a set of labeled training examples.

19. The method of claim 18, wherein the object detection task is open vocabulary object detection.

20. The method any of the preceding claims, wherein the set of feature levels comprise a plurality of feature levels.

21. A computer-implemented detection method for performing an object detection task, the detection method comprising: obtaining (i) an input image and (ii) a set of one or more query embeddings, wherein each query embedding represents a respective category of object; processing the input image using the image encoder neural network and the detection neural networks that have been trained using the method of any of the preceding claims, to generate a set of object detection embeddings; and for each respective category of object, computing a respective detection score based on similarities between the set of object detection embeddings and the respective query embedding corresponding to the respective category of object.

22. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-21.

23. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-21.

24. A computer-implemented system comprising:an image encoder neural network configured to receive a first input specifying an input image and process the first input to generate an image feature map comprising a set of image embeddings of the input image; a simple feature pyramid network configured to, for each respective feature level of a set of feature levels, process a respective second input generated from the image feature map to generate a respective set of level-specific image embeddings for the input image; a set of detection neural networks that comprise, for each respective feature level of the set of feature levels, a respective detection neural network configured to process a respective third input generated from the respective set of level-specific image embeddings to generate a respective set of level-specific detection embeddings; one or more pooling layers configured to combine each respective set of level-specific object detection embeddings to generate a respective level-specific combined detection embedding; a text encoder neural network configured to process a set of one or more text queries to generate a set of one or more query embeddings, wherein each query embedding represents a respective category of object; and a detection unit configured to, for each respective category of object, compute a respective detection score based on similarities between the level-specific combined detection embeddings and the respective query embedding corresponding to the respective category of object.

25. The system of claim 24, wherein the respective third input is generated by: identifying, a respective set of regions of interest (Rols) of a respective level-specific feature map formed by the respective set of level-specific image embeddings; extracting a respective set of Rol features from the respective level-specific feature map according to the respective set of sampled Rols; and performing a feature alignment operation on the respective set of Rol features.

26. The system of claim 24 or claim 25, wherein the image encoder neural network configured to: divide the input image into a plurality of patches; process the plurality of patches to generate a first feature map comprising a plurality of patch embeddings;generate a second feature map by shifting the patch embeddings in the first feature map, along each of a width dimension and a height dimension of the first feature map, by a predefined number of pixels; process the first feature map using a set of layers of the image encoder neural network to generate a third feature map; process the second feature map using the set of layers of the image encoder neural network to generate a fourth feature map; generate a fifth feature map by shifting the patch embeddings in the fourth feature map, along each of a width dimension and a height dimension of the fourth feature map, in directions that are opposite of the shifting directions when generating the second feature map; generate a combined feature map by combining the third feature map and the fifth feature map; and generate the image feature map from the combined feature map.

27. The system of claim 26, wherein the set of layers of the image encoder neural network comprises one or more windowed-attention layers that operate on a predefined window size, and the predefined number of pixels for the shifting is a fraction of the predefined window size.

28. The system of any of claims 24-27, wherein the image encoder neural network and the detection neural networks have been trained using the method of any of claims 1-20.

Images