Detection of open vocabulary objects in images

Abstract

A method, system, and apparatus for object detection, comprising a computer program encoded on a computer storage medium. In one aspect, the method includes: (i) obtaining an image and (ii) a set of one or more query embeddings, each query embedding representing a respective category of an object; processing the image and the set of query embeddings using an object detection neural network to generate object detection data for the image, including processing the image using an image encoding sub-network of the object detection neural network to generate a set of object embeddings, processing each object embedding using a localization sub-network to generate localization data defining a corresponding region of the image, and processing (i) the set of object embeddings and (ii) the set of query embeddings using a classification sub-network to generate, for each object embedding, a respective classification score distribution across the set of query embeddings.

Description

Background Art

[0001] This specification relates to the processing of data using a machine learning model.

[0002] A machine learning model receives an input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate an output based on the received input and the values of the model's parameters.

[0003] Some machine learning models are deep models that use multiple layers of the model to generate an output for the received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, and each layer applies a non-linear transformation to the received input to generate an output.

Summary of the Invention

[0004] This specification describes a system for performing object detection in an image, implemented as a computer program on one or more computers located in one or more locations.

[0005] Throughout this specification, "embedding" may refer to an ordered set of numbers, such as a vector of numbers, a matrix, or another tensor.

[0006] According to one embodiment, a method is provided that is performed by one or more computers, the method comprising: (i) an image, and (ii) a set of one or more query embeddings, each query embedding, comprising: obtaining a set of one or more query embeddings, each representing a respective category of objects; processing the image and the set of query embeddings using an object detection neural network to generate object detection data for the image, comprising: processing the image using an image encoding subnetwork of the object detection neural network to generate a set of object embeddings; processing each object embedding using a localization subnetwork of the object detection neural network to generate localization data defining a corresponding region of the image; and processing (i) the set of object embeddings and (ii) the set of query embeddings using a classification subnetwork of the object detection neural network to generate a respective classification score distribution for each object embedding, each classification score distribution for each object embedding, comprising generating a set of image and query embeddings, each defining the possibility that the region of the image corresponding to the object embedding depicts an object belonging to the category represented by the query embedding.

[0007] In some embodiments, for one or more of the query embeddings, obtaining the query embeddings includes obtaining a text sequence describing a category of objects and processing the text sequence using a text encoding subnetwork of an object detection neural network to generate the query embeddings.

[0008] In some embodiments, the image encoding subnetwork and the text encoding subnetwork are pre-trained, and the pre-training includes repeatedly performing operations that include: (i) obtaining a training image, (ii) a positive text sequence that characterizes the training image, and (iii) one or more negative text sequences that do not characterize the training image; generating an embedding of the training image using the image encoding subnetwork; generating embeddings for the positive text sequence and the negative text sequence, respectively, using the text encoding subnetwork; and co-training the image encoding subnetwork and the text encoding subnetwork to facilitate (i) increasing the similarity between the embedding of the training image and the embedding of the positive text sequence, and (ii) decreasing the similarity between the embedding of the training image and the embedding of the negative text sequence.

[0009] In some embodiments, generating embeddings for a training image using an image encoding subnetwork includes processing the training image using an image encoding subnetwork to generate a set of object embeddings for the training image, and processing the object embeddings using an embedding neural network to generate embeddings for the training image.

[0010] In some embodiments, the embedded neural network is co-trained with image encoding subnetworks and text encoding subnetworks.

[0011] In some embodiments, co-training an image encoding subnetwork and a text encoding subnetwork involves co-training the image encoding subnetwork and the text encoding subnetwork to optimize an objective function that includes a control loss term.

[0012] In some embodiments, after pre-training of image encoding subnetworks and text encoding subnetworks, the object detection neural network is trained to optimize an objective function that measures the performance of the object detection neural network on the task of detecting objects in an image.

[0013] In some embodiments, the objective function used to measure the performance of an object detection neural network on the task of detecting objects in an image includes a bipartite matching loss term.

[0014] In some embodiments, using a classification subnetwork of an object detection neural network to process (i) a set of object embeddings and (ii) a set of query embeddings to generate a classification score distribution for each object embedding across the set of query embeddings includes processing each object embedding using one or more neural network layers of the classification neural network to generate a corresponding classification embedding, and for each object embedding, generating a classification score distribution using (i) a classification embedding corresponding to the object embedding and (ii) a query embedding to generate a classification score distribution, which includes generating a similarity measure between the classification embedding and the query embedding, each of which is a similarity measure between the classification embedding and each query embedding, defining the likelihood that the region of the image corresponding to the object embedding depicts an object belonging to the category represented by the query embedding.

[0015] In some embodiments, processing each object embedding using one or more neural network layers of a classification neural network to generate a corresponding classification embedding includes generating each classification embedding by projecting the corresponding object embedding into a latent space containing query embeddings.

[0016] In some embodiments, generating a measure of similarity between a classification embedding and each query embedding involves calculating the inner product between the classification embedding and the query embedding for each query embedding.

[0017] In some embodiments, for each object embedding, processing the object embedding using a localization subnetwork to generate localization data that defines the corresponding region of the image includes processing the object embedding using a localization subnetwork to generate localization data that defines a bounding box within the image.

[0018] In some embodiments, processing an image using an image encoding subnetwork to generate a set of object embeddings includes generating an initial set of object embeddings by an embedding layer of the image encoding subnetwork, where each initial object embedding is at least partially derived from a corresponding patch in the image, and processing the initial set of object embeddings by a plurality of neural network layers, including one or more self-attention neural network layers, to generate a final set of object embeddings.

[0019] In some embodiments, processing object embeddings using a localization subnetwork to generate localization data that defines the corresponding region of the image includes generating a set of offset coordinates, where the offset coordinates define the offset of the corresponding region of the image from the location of the image patch corresponding to the object embedding.

[0020] In some embodiments, the text encoding subnetwork includes one or more self-attention neural network layers.

[0021] In some embodiments, the method includes determining, for one or more object embeddings, that the region of the image corresponding to the object embedding depicts an object that belongs to a category represented by a query embedding based on a classification score distribution for the object embedding.

[0022] In this embodiment, obtaining a query embedding for one or more query embeddings includes obtaining one or more query images, each query image containing a target region that depicts an example of a target object; generating an embedding for each target object in each query image; and generating a query embedding by combining the embeddings of the target objects in the query images.

[0023] In some embodiments, for each query image, generating an embedding of a target object within the query image includes: processing the query image using an image encoding subnetwork to generate a set of object embeddings for the query image; processing each object embedding for the query image using a localization subnetwork to generate localization data that defines the corresponding region of the query image; determining, for each object embedding for the query image, a measurement of the respective overlap between (i) a target region of the query image depicting an example of the target object and (ii) a region of the query image corresponding to the object embedding for the query image; and selecting an object embedding from the set of object embeddings for the query image as the embedding of the target object within the query image based on the overlap measurement.

[0024] In some embodiments, generating query embeddings by combining the embeddings of target objects in the query image includes averaging the embeddings of target objects in the query image.

[0025] In another embodiment, a system is provided, which includes one or more computers and one or more storage devices communicably coupled to the one or more computers, the storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations described herein.

[0026] In another embodiment, one or more non-temporary computer-readable media are provided that, when executed by one or more computers, store instructions causing one or more computers to perform the operations described herein.

[0027] The subject matter described herein can be implemented in certain embodiments to achieve one or more of the following advantages:

[0028] The system described in this specification can perform "open-vocabulary" object detection. That is, the system can detect objects of any object category. In contrast, conventional object detection systems may be limited to detecting objects within a small, fixed set of object categories. This system enables open-vocabulary object detection by making it possible to specify object categories in the form of query embeddings. Each query embedding represents a respective object category and can be generated in any suitable way, for example, from a text sequence that describes the object category (e.g., "a bird perched on a tree") or from an image that shows an example of an object within the object category. In particular, this system enables image-conditioned object detection. That is, a query embedding is derived from an image that shows an example of an object, thereby enabling the detection of objects that are difficult to describe in text but easy to capture in an image (e.g., special technical parts).

[0029] In the case of text sequences, the system can perform object detection by comparing a set of query embeddings generated by processing the text sequence using a text-encoded neural network with a set of object embeddings generated by processing the image using an image-encoded neural network. The text-encoded and image-encoded neural networks can operate independently, which can dramatically improve inference efficiency. For example, in a system where text-encoded and image-encoded neural networks are fused, encoding queries requires a forward pass through the image-encoded neural network, which must be repeated for each image-query combination. In contrast, the systems described herein can process an image once using the image-encoded neural network and then generate any number of query embeddings without reprocessing the image. The same applies to other query modalities. The image-encoded neural network and the query embedding generator are independent entities. Query embeddings may be generated separately (and on different systems) and provided to the system to perform object detection (and vice versa).

[0030] The system can pre-train a text encoding neural network and an image encoding neural network "in contrast" to learn the representations of images and text in a shared embedding space, such that, for example, embeddings of semantically similar images and text tend to be closer in the embedding space with respect to the object categories depicted in the images and defined in the text. Training data for the contrastive pre-training of the text encoding neural network and the image encoding neural network is abundantly available. By pre-training the text encoding neural network and the image encoding neural network in contrast, the downstream performance of the text encoding neural network and the image encoding neural network on the object detection task can be significantly improved, for which relatively little training data may be available.

[0031] Details of one or more embodiments of the subject matter of this specification are described in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

Brief Description of the Drawings

[0032] [Figure 1] It is a block diagram of an exemplary object detection system. [Figure 2] It is a flowchart of an exemplary process for detecting and locating an object. [Figure 3] It is an exemplary architecture of the object detection system of FIG. 1. [Figure 4] It is a flowchart of an exemplary process for pre-training and fine-tuning an image encoding sub-network and a text encoding sub-network. [Figure 5] It shows the pre-training of an image encoding sub-network and a text encoding sub-network. [Modes for carrying out the invention]

[0033] Similar reference numbers and symbols in various drawings refer to the same elements.

[0034] Figure 1 shows an exemplary object detection system 100. The object detection system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, where the system, components, and techniques described below are implemented.

[0035] The object detection system 100 is configured to process an image 105 and a set of one or more query embeddings 110 to generate an object detection output 145.

[0036] Image 105 can be generated using any suitable imaging device, such as a camera, microscope imaging device, telescope imaging device, or medical imaging device (e.g., computed tomography (CT) imaging device, X-ray imaging device, or ultrasound (US) imaging device). Image 105 may show (depict) one or more objects. For example, an image captured by a camera may show objects such as road signs, vehicles, pedestrians, or buildings. As another example, an image captured by a medical imaging device may show anatomical structures (e.g., organs, tumors, etc.), tissues, regions of the human or animal body, etc. Image 105 may include pixel intensity values.

[0037] Each query embedding in a set of one or more query embeddings 110 represents a respective object category and can be generated in any suitable way, for example, from a text sequence describing the object category (e.g., "birds perched on a tree") or from an image showing an example of an object within that object category. In particular, this object detection system 100 enables image-conditional object detection. That is, query embeddings are derived from images showing examples of objects, thereby enabling the detection of objects that are difficult to describe in text but easy to capture in images (e.g., specialized technical parts). For example, exemplary techniques for generating query embeddings 110 representing object categories from text sequences or images are described in more detail below with reference to Figures 2 to 4.

[0038] The object detection output may include one or more object detection instances. Each object detection instance may be associated with (i) localization data that identifies a region of the image, and (ii) object identification data that characterizes the type of object shown in the region of the image. In some cases, the object identification data may include, for each query embedding, a classification score distribution that defines the likelihood that a region of the image represents an object within the object category represented by the query embedding. In some cases, the object identification data may define that a region of the image represents an object that belongs to the object category associated with a particular query embedding from a set of query embeddings. Thus, the object detection output can characterize the location and type of objects shown in the image.

[0039] The object detection system 100 can receive images 105 and query embeddings 110 from a user or another source, for example, via an application programming interface (API) made available by the system. The object detection system 100 can provide object detection outputs 145 for, for example, transmission over a data communication network (e.g., the Internet), display on an interface (e.g., a graphical user interface), or storage in memory.

[0040] The object detection system 100 can be implemented in any suitable location, for example, within a data center, on a user device (for example, a smartphone, tablet, or laptop), or in a distributed manner, for example, with certain parts of the system implemented in the data center and other parts of the system implemented on the user device.

[0041] The object detection system 100 may include an object detection neural network 150, which itself includes an image encoding subnetwork 115, a localization subnetwork 125, and a classification subnetwork 135, each of which will be described in more detail later in this specification.

[0042] The image encoding subnetwork 115 processes the image 105 to generate a set of object embeddings 120. One or more of the object embeddings may each correspond to an object shown in the image.

[0043] The set of object embeddings 120 can include any appropriate number of object embeddings, for example, a static (predefined) number of object embeddings, or a number of object embeddings that depend on the image 105. The image encoding subnetwork 115 can have any appropriate neural network architecture, for example, including any appropriate number (e.g., 5, 10, or 20) of any appropriate type of neural network layers (e.g., attention layers, convolutional layers, fully connected layers, etc.) connected in any appropriate configuration (e.g., as a directed graph of layers). A specific exemplary architecture of the image encoding subnetwork 115 is described in more detail below with reference to Figure 3.

[0044] The localization subnetwork 125 processes each object embedding in the set of object embeddings 120 to generate localization data 130 that defines the corresponding region of the image. The localization subnetwork can have any suitable neural network architecture, for example, including any suitable number (e.g., 5, 10, or 20) of any suitable type of neural network layers (e.g., attention layers, convolutional layers, fully connected layers, etc.) connected in any suitable configuration (e.g., as a directed graph of layers).

[0045] In some examples, localization data can define bounding boxes within an image that correspond to each embedded object. For instance, if image 105 depicts several different objects, the localization subnetwork 125 can define a corresponding bounding box for each object in the image.

[0046] The classification subnetwork 135 processes a set of object embeddings 120 and a set of query embeddings 110 to generate a classification score distribution 140 for each object embedding across the set of query embeddings. The classification score distribution 140 for each object embedding 120 defines, for each query embedding 110, the likelihood that the region of the image corresponding to the object embedding (i.e., according to localization data) depicts an object that belongs to the category represented by the query embedding.

[0047] The classification subnetwork 135 can have any suitable neural network architecture, for example, including any suitable number (e.g., 5, 10, or 20) of any suitable type of neural network layers (e.g., attention layers, convolutional layers, fully connected layers, etc.) connected in any suitable configuration (e.g., as a directed graph of layers). A specific exemplary architecture of the classification subnetwork is described in more detail below with reference to Figure 3.

[0048] The object detection system 100 can process localization data 130 and classification score distribution 140 to generate an object detection output 145. The object detection output includes one or more object detection instances, each of which may be associated with (i) localization data identifying a region of an image, and (ii) object identification data characterizing the type of object shown in the region of the image (as described above).

[0049] In particular, the object detection system 100 can generate an object detection instance for each of one or more object embeddings. For example, for each of one or more object embeddings, the object detection system 100 can generate an object detection instance associated with (i) localization data generated by a localization subnetwork for the object embedding, and (ii) object identification data based on a classification score distribution generated by a classification subnetwork for the object embedding. In some cases, the object identification data includes a classification score distribution. In some cases, the object identification data identifies a query embedding associated with the highest score under the classification score distribution.

[0050] In some cases, the object detection system 100 may refrain from generating object detection instances for object embeddings based, for example, on the classification score distribution for object embeddings. For example, the object detection system 100 may refrain from generating object detection instances for object embeddings if the maximum score included in the classification score distribution for object embeddings is less than a threshold, for example, 0.001, or 0.01, or 0.1.

[0051] The training of the object detection neural network will be explained in more detail below, referring to Figures 4 and 5.

[0052] Figure 2 is a flowchart of an exemplary process 200 for object detection and locating. For convenience, process 200 is described as being performed by one or more computer systems located in one or more locations. For example, an object detection system, e.g., object detection system 100 in Figure 1, is appropriately programmed according to this specification and is capable of performing process 200.

[0053] The system retrieves a set of images and query embeddings (step 202). Each query embedding represents a category of the object.

[0054] An image can depict one or more objects. For example, an image captured by a camera may show objects such as road signs, vehicles, pedestrians, or buildings. As another example, an image captured by a medical imaging device may show anatomical structures (e.g., organs, tumors, etc.), tissues, or regions of the human or animal body.

[0055] The system can obtain query embeddings in one of several possible ways. The following describes some exemplary techniques that the system can use to generate query embeddings.

[0056] In some examples, the system generates one or more query embeddings by obtaining a text sequence that describes the category of an object. The system then processes the text sequence using a text encoding subnetwork to generate query embeddings for the given text sequence. For example, a given text sequence could be a short description of the object type (e.g., Monarch butterfly) or "full moon".

[0057] In some examples, the system obtains one or more query embeddings by first obtaining one or more query images. Each query image contains a target region that shows an example of the target object. For example, a query image could depict a bird on a tree branch in front of a blue background. If the target object is a bird, the target region could be the region depicting the bird. The system generates the respective embeddings of the target object in each query image and combines these embeddings to generate the query embeddings. In some examples, the system generates the query embeddings by averaging the embeddings of the target object in the query images and combining them.

[0058] The system can generate embeddings for target objects for each query image by processing the query image using an image encoding subnetwork to generate a set of object embeddings for the query image. The system can process each object embedding for the query image using a localization subnetwork to generate localization data that defines the corresponding region of the query image. For each object embedding for the query image, the system can determine the respective overlap measurement using, for example, an intersection over union (IoU) measurement between the target region of the query image depicting an example of the target object and the region of the query image corresponding to the object embedding for the query image. Based on the overlap measurement, the system can select an object embedding from the set of object embeddings for the query image as the embedding of the target object in the query image. For example, the system can select the object embedding with the best overlap measurement.

[0059] For example, a query image may depict a tennis ball and a racket. The target region may depict the tennis ball. A set of object embeddings may include an object embedding corresponding to the tennis ball and an object embedding corresponding to the racket. The system can determine two overlap measurements: one between the object embedding corresponding to the tennis ball and the target region depicting the tennis ball, and the other between the object embedding corresponding to the rack and the target region depicting the tennis ball. The system may determine that there is a higher overlap measurement between the object embedding corresponding to the tennis ball and the target region depicting the tennis ball, and select this object embedding as the target object embedding. The overlap measurement may be a percentage or score on a given scale, or an intersection over union (IoU) measurement.

[0060] The image encoding subnetwork, text encoding subnetwork, and localization subnetwork can have any suitable neural network architecture, for example, comprising any suitable number (e.g., 5, 10, or 20) of any suitable type of neural network layer (e.g., self-attention layers, convolutional layers, fully connected layers, etc.) connected in any suitable configuration (e.g., as a directed graph of layers).

[0061] The training of the image encoding subnetwork, text encoding subnetwork, and localization subnetwork will be described in detail below with reference to Figures 4 and 5.

[0062] The system processes the image using an image encoding subnetwork to generate a set of object embeddings (step 204). For example, the system can generate an initial set of object embeddings by an embedding layer of an image encoding subnetwork. In particular, the system can divide the image into a grid of patches. Each initial object embedding can be derived from a corresponding patch in the image. For example, each initial object embedding can be generated by concatenating pixels in the corresponding patch in the image into a vector. The system can process the initial set of object embeddings by one or more neural network layers, for example, one or more self-attention neural network layers, to generate a final set of object embeddings. The final set of object embeddings may include one or more embeddings for each of the one or more objects depicted in the image.

[0063] In some embodiments, the system can generate a predefined number of object embeddings, such as 500 for each input image. An image may contain fewer objects than the predefined number of object embeddings. For example, an image may contain only two objects. In this case, the system may generate two object embeddings corresponding to the objects in the input image, and 498 object embeddings that do not correspond to any objects in the input image, which may, for example, correspond to embeddings generated from the background of the image. Alternatively, there may be multiple object embeddings generated for each object, and in particular, if the object embeddings are derived from image patches, the object may span multiple patches. When object embeddings are derived from image patches, they can be thought of as image region embeddings.

[0064] The system processes each object embedding and generates localization data that defines the corresponding region of the image using a localization subnetwork (step 206).

[0065] The system can use a localization subnetwork to process each object embedding and generate localization data that defines regions of the image, such as bounding boxes within the image. For example, if an image depicts several different objects, the localization data can define a corresponding bounding box for each object in the image.

[0066] In some examples, the system can generate a set of offset coordinates as localization data. These offset coordinates can be expressed as (x,y) displacements. The offset coordinates can define the offset of the corresponding region in an image from the image patch corresponding to the object embedding.

[0067] The system uses a classification subnetwork to process sets of object embeddings and query embeddings to generate a classification score distribution for each object embedding across the set of query embeddings (step 208). The classification score distribution for object embeddings defines, for each query embedding, the likelihood that the region of the image corresponding to the object embedding (i.e., according to localization data) depicts an object that belongs to the category represented by the query embedding.

[0068] The system can first generate a set of classification score distributions by processing each object embedding using one or more neural network layers of a classification neural network to generate a corresponding classification embedding. Next, for each object embedding, the system can generate a classification score distribution across the set of query embeddings using the classification embedding corresponding to the object embedding and the query embedding.

[0069] In some examples, the system generates each classification embedding by projecting the corresponding object embedding into a latent space that includes query embeddings.

[0070] In some examples, the system determines the respective measures of similarity between the classification embedding and each query embedding (e.g., L1 norm, L2 norm, cosine similarity, dot product, etc.). The measures of similarity between the classification embedding and the query embedding can define the likelihood that the image region corresponding to the object embedding depicts an object that belongs to the category represented by the query embedding.

[0071] The system generates an object detection output from localization data and classification score distribution (step 210). The object detection output includes one or more object detection instances, each of which may be associated with (i) localization data identifying a region of the image and (ii) object identification data characterizing the type of object shown in the region of the image. In particular, the system can generate a separate object detection instance for each of one or more object embeddings. For example, for each of one or more object embeddings, the system can generate an object detection instance associated with (i) localization data generated by a localization subnetwork for the object embedding and (ii) object identification data based on a classification score distribution generated by a classification subnetwork for the object embedding. In some cases, the object identification data includes a classification score distribution. In some cases, the object identification data identifies a query embedding associated with the highest score under the classification score distribution.

[0072] In some cases, the system may refrain from generating object detection instances for object embeddings based on, for example, the classification score distribution for object embeddings. For instance, the system may refrain from generating object detection instances for object embeddings if the maximum score included in the classification score distribution for object embeddings is below a threshold, for example, 0.001, or 0.01, or 0.1.

[0073] For example, the system can receive query embeddings representing the category "duck" and query embeddings representing the category "dog". The system can receive an input image depicting a duck and a cat. The set of object embeddings for the image includes embeddings representing a duck and embeddings representing a cat. For the object embedding representing a duck, the classification score distribution may include a score of 0.93 for the category "duck" and a score of 0.01 for the category "dog". For the object embedding representing a cat, the classification score distribution may include a score of 0.02 for the category "duck" and a score of 0.01 for the category "dog". The system can determine not to generate an object detection instance if the maximum score included in the classification score distribution for the object embedding is less than the threshold of 0.1. Therefore, the system does not generate an object detection instance for the object embedding representing a cat, and the system generates an object detection instance for the object embedding representing a duck.

[0074] In some cases, a set of query embeddings may include query embeddings designated as "background" query embeddings. Background query embeddings can represent the background portion of an image, for example, a portion of the image that does not show a particular object. In response to determining that a background embedding is associated with the highest score under the classification score distribution for object embeddings, the system may refrain from generating object detection instances for the object embedding.

[0075] The object detection output includes one object detection instance associated with localization data that identifies the region of the image depicting the duck, and object identification data that characterizes the duck. For example, the localization data could be a bounded box surrounding the region of the image depicting the duck. The object identification data can identify that for an object embedding representing a duck, the query embedding representing the category "duck" is associated with the highest score under the classification score distribution across all query embeddings.

[0076] In some examples, the system can be implemented on a robot. The robot can receive natural language instructions to perform tasks. Natural language instructions may ask the robot to identify a specific object (for example, "clean the kitchen using X"). The software installed on the robot can identify the object specified by the natural language instruction using the open vocabulary object detection capabilities of an object detection neural network.

[0077] In some examples, the system may be implemented on a robot that has previously seen one or more objects. The robot can receive commands to identify objects similar to those it has previously seen. The object detection neural network can use the previously seen objects to generate a set of query embeddings. The robot can then use the object detection neural network to detect similar objects in the future.

[0078] Figure 3 shows an exemplary architecture 300 of the object detection system shown in Figure 1. This architecture includes a text encoding neural network 310 and an image encoding neural network 330.

[0079] Image encoding subnetworks and text encoding subnetworks can have any suitable neural network architecture, for example, comprising any suitable number (e.g., 5, 10, or 20) of any suitable type of neural network layers (e.g., self-attention layers, convolutional layers, fully connected layers, etc.) connected in any suitable configuration (e.g., as a directed graph of layers). An exemplary architecture includes a visual transformation encoder.

[0080] The text encoding subnetwork 310 receives a text sequence 305 that describes categories of objects, such as "giraffe," "tree," and "car." The text sequence 305 may be the name of a category or other text description. The text encoding subnetwork processes the text sequence 305 to generate a set of query embeddings 315 that correspond to the text sequence. Each query embedding consists of a separate token sequence representing an individual object description and is processed individually by the text encoder.

[0081] The image encoding neural network 330 receives an input image 325 which is divided into patches. The image encoding neural network 330 generates a set of object embeddings 355. The set of object embeddings includes object-specific embeddings for each object shown in the input image 325. The input image 325 shows two giraffes and one tree. In this example, there may be object embeddings for each giraffe and tree.

[0082] The localization subnetwork 335 processes the object embeddings 355 to generate localization data that defines a set of bounding boxes 340 within the image. The bounding boxes are represented as box coordinates. For each object embedding, the localization subnetwork generates a predicted box that represents the location of the object.

[0083] The classification subnetwork 350 processes a set of object embeddings and a set of query embeddings to generate a classification score distribution 360 across the set of query embeddings for each object embedding. This may generate a prediction class 320 for each object embedding.

[0084] Query embedding 315 consists of "giraffe," "tree," and "car" from left to right. The classification score distribution corresponding to the top object embedding indicates a probability of 0.9 for giraffe, 0.1 for tree, and 0.1 for car. Therefore, classification subnetwork 350 predicts that the object embedding belongs to the class "giraffe." The classification score distribution corresponding to the bottom object embedding indicates a probability of 0.1 for giraffe, 0 for tree, and 0.1 for car. Therefore, classification subnetwork 350 predicts that the object embedding does not belong to any of the query classes.

[0085] Architecture 300 does not involve fusion between the image encoder 330 and the text encoder 310. That is, the image encoding neural network 330 and the text encoding neural network 310 are separate neural networks that perform their processing independently of each other. While early fusion seems intuitively beneficial, encoding queries 315 requires a forward pass through the entire image model, which must be repeated for each image / query combination, thus reducing inference efficiency. In this architecture, system 100 can compute query embeddings independently of images, making it possible to use thousands of queries per image, and far more queries than would be possible with early fusion.

[0086] The object detection system 100 processes the prediction class 320 and the prediction box 340 to generate an object detection instance for each of one or more object embeddings. The object detection instance can, for example, define a bounding box and a classification score distribution for the bounding box.

[0087] Figure 4 is a flowchart of an exemplary process 400 for pre-training and fine-tuning the image encoding subnetwork and text encoding subnetwork included in the object detection neural network described with reference to Figure 1, for example. For convenience, process 400 is described as being performed by one or more computer systems located in one or more locations, for example, the object detection system described with reference to Figure 1.

[0088] The system pre-trains image encoding subnetworks and text encoding subnetworks (step 405). Pre-training involves repeatedly performing a series of operations on a set of training images. For example, for each training image in the set of training images, the system may obtain a positive text sequence and one or more negative text sequences. A positive text sequence may characterize at least one of the objects depicted in the training image, while a negative text sequence may not characterize any of the objects depicted in the training image. For example, the training image may be a monarch butterfly. The positive text sequence may be "monarch butterfly," and the negative text sequences may be "swallowtail butterfly," "moth," "caterpillar," and "starfish." The system may obtain positive and negative text sequences for training images from a variety of possible sources. For example, the system may scrap positive text sequences from an image database to train images as image captions. Negative text sequences may be, for example, randomly sampled text sequences.

[0089] The system can generate embeddings for training images using an image encoding subnetwork. For example, the system can process training images using an image encoding subnetwork to generate a set of object embeddings for the training images (as illustrated, for example, with reference to Figures 1 and 2). The system can then process the object embeddings using an embedding neural network to generate embeddings for the training images. The embedding neural network can have any suitable neural network architecture, for example, comprising any suitable number (e.g., 1, 5, or 10) of any suitable type of neural network layers (e.g., pooling layers, fully connected layers, attention layers, etc.) connected in any suitable configuration (e.g., as a directed graph of layers). A specific exemplary architecture of the embedding neural network is illustrated with reference to Figure 5. In some embodiments, the embedding neural network is co-trained with an image encoding subnetwork and a text encoding subnetwork, as will be described in more detail below.

[0090] The system uses a text encoding subnetwork to generate separate embeddings for positive and negative text sequences. That is, for each positive and negative text sequence, the system processes the text sequence using the text encoding subnetwork and generates an embedding for the text sequence.

[0091] The system co-trains an image encoding subnetwork, a text encoding subnetwork, and optionally an embedding neural network to facilitate (i) increasing the similarity between the embedding of training images and the embedding of positive text sequences, and (ii) decreasing the similarity between the embedding of training images and the embedding of negative text sequences. For example, the system can co-train the image encoding subnetwork, the text encoding subnetwork, and the embedding neural network to optimize an objective function that includes a contrast loss term or a triplet loss term. The objective function can measure the similarity between embeddings using, for example, the dot product, or L1 similarity measure, or L2 similarity measure, or other appropriate similarity measure. For each training image, the system can determine the gradient of the objective function with respect to the parameters of the image encoding subnetwork, the text encoding subnetwork, and optionally an embedding neural network. The system can then use the gradient to tune the parameter values ​​of the image encoding subnetwork, the text encoding subnetwork, and optionally an embedding neural network using, for example, an update rule for an appropriate gradient descent optimization algorithm (e.g., RMSprop or Adam).

[0092] In other words, the system can pre-train text-encoded neural networks and image-encoded neural networks "in contrast" to learn representations of images and text in a shared embedding space, for example, so that embeddings of images and text with common objects tend to be closer together in the embedding space. By pre-training text-encoded neural networks and image-encoded neural networks in contrast, the downstream performance of text-encoded neural networks and image-encoded neural networks on object detection tasks can be significantly improved.

[0093] In some cases, the embedded neural network is discarded after pre-training is complete.

[0094] After pre-training the image encoding subnetwork and the text encoding subnetwork, the system then fine-tunes the image encoding subnetwork and the text encoding subnetwork for the object detection task (step 410). The system fine-tunes both the image encoding subnetwork and the text encoding subnetwork from start to finish.

[0095] When fine-tuning, the system repeatedly performs a series of operations on a set of training examples. For example, each training example may include a training image, one or more positive text sequences, the bounding box associated with each positive text sequence, and one or more negative text sequences. Each positive text sequence can characterize the respective object shown within the corresponding bounding box in the training image. Negative text sequences do not characterize the objects shown in the training image.

[0096] The system can process training images using an image encoding subnetwork to generate a set of object embeddings for the training images (as illustrated, for example, with reference to Figures 1 and 2).

[0097] The system uses a text encoding subnetwork to generate separate embeddings for positive and negative text sequences. That is, for each positive and negative text sequence, the system processes the text sequence using the text encoding subnetwork and generates an embedding for the text sequence.

[0098] The system uses a localization subnetwork to generate localization data that defines a set of bounding boxes in the image. For each object embedding, the localization subnetwork generates a predicted box that represents the object's location.

[0099] The system trains an object detection neural network to optimize an objective function that measures the performance of the object detection neural network on the task of object detection in an image. In some examples, the objective function includes a loss term that helps the object detection neural network produce accurate object classifications; for example, the objective function may include a bipartite matching loss term. In some examples, the objective function includes a loss term that helps a localization subnetwork perform accurate localization. For example, the objective function may include a loss term that measures, for each positive text sequence in the training example, the error between (i) the bounding box associated with the positive text sequence in the training example and (ii) the bounding box generated by the object detection neural network for the corresponding object embedding.

[0100] Figure 5 shows the pre-training of the image encoding subnetwork 525 and the text encoding subnetwork 510 included in the object detection neural network described, for example, with reference to Figure 1.

[0101] The text encoding subnetwork 525 receives a positive text sequence 505, such as "bird perched on a tree." The image encoding subnetwork receives a training image 520 that matches the positive text sequence, such as an image of a bird perched on a tree.

[0102] The text encoding subnetwork 510 processes the positive text sequence 505 to generate text embeddings 515 corresponding to the positive text sequence. The image encoding subnetwork 525 processes the input image 520 to generate embeddings for the training image 535.

[0103] The input image 520 can be divided into patches. The image encoding subnetwork 525 generates a set of object embeddings 545.

[0104] The embedded neural network 530 processes the object embeddings 545 to generate embeddings of the training image 535. In this example, the embedded neural network includes pooling layers (e.g., mean pooling layers or max pooling layers) and projection layers (e.g., fully connected layers).

[0105] The object detection system can train text encoding subnetworks and image encoding subnetworks as batches, one at a time. A batch can contain one or more training images, each with its own positive and negative text sequence. For example, a batch could contain five training images with appropriate descriptions: one depicting a bird, one depicting a cat, one depicting a dog, one depicting a squirrel, and one depicting a giraffe. In the case of the bird image, the positive text sequence could be "bird perched on a tree," and the negative text sequence could be "giraffe standing on grass."

[0106] Text embedding 515 is added to the set of text embeddings 550 that correspond to other text descriptions in the batch. The embedding of training image 535 is added to the set of image embeddings that correspond to other training images in the batch.

[0107] The object detection system 100 optimizes an objective function that includes a control loss term across all images in batch 540. The objective function promotes increased similarity between the embeddings of the training image 535 and the embeddings of the positive text sequence 515, and decreased similarity between the embeddings of the training image and the embeddings of the negative text sequence. An example in Figure 5 shows a grid with text embeddings 550 on one axis and training image embeddings 555 on the other axis. The intersection of each text embedding and each image embedding is represented by either "+" or "-", where "+" indicates that similarity between the text embedding and the image embedding is promoted, and "-" indicates that dissimilarity between the text embedding and the image embedding is promoted.

[0108] The term “configured” is used herein in relation to systems and computer program components. A system of one or more computers being configured to perform a particular operation or action means that the system has installed software, firmware, hardware, or a combination thereof that causes the operating system to perform that operation or action. One or more computer programs being configured to perform a particular operation or action means that one or more programs, when executed by a data processing device, contain instructions that cause the device to perform that operation or action.

[0109] The subject matter and functional embodiments described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware, including structures disclosed herein and their structural equivalents, or one or more combinations thereof. Embodiments of the subject matter described herein may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded in a tangible, non-temporary storage medium, which are executed by or control the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or one or more combinations thereof. Alternatively, or in addition, the program instructions may be encoded in artificially generated transmitted signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by a data processing device.

[0110] The term "data processing device" refers to data processing hardware and encompasses all kinds of devices, machines, and equipment for data processing, including, for example, programmable processors, computers, or multiple processors or multiple computers. A device may be, or further may be, a special-purpose logic circuit, such as an FPGA (Field-Programmable Gate Array) or ASIC (Application-Specific Integrated Circuit). Optionally, in addition to hardware, a device may include code that generates an execution environment for computer programs (e.g., code that constitutes processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof).

[0111] Computer programs, which may be called or described as programs, software, software applications, apps, modules, software modules, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as standalone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. A program may, but may not, correspond to a file in a file system. A program may be part of a file that holds one or more scripts stored in a document in a markup language, in a single file dedicated to the program of interest, or in multiple collaborative files, such as a file that holds one or more modules, subprograms, or parts of code. A computer program may be deployed to run on one computer, or it may be deployed to run on multiple computers located in one place or distributed across multiple locations and interconnected by a data communication network.

[0112] In this specification, the term “engine” is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine is implemented as one or more software modules or components and installed on one or more computers in one or more locations. In some cases, one or more computers are dedicated to a particular engine, while in other cases, multiple engines can be installed and run on the same one or more computers.

[0113] The processes and logic flows described herein can be performed by one or more programmable computers executing one or more computer programs to act on input data and produce outputs, thereby performing their functions. Alternatively, the processes and logic flows can be performed by special-purpose logic circuits (e.g., FPGAs or ASICs) or by a combination of special-purpose logic circuits and one or more programmed computers.

[0114] A computer suitable for running computer programs can be based on a general-purpose or dedicated microprocessor, or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from read-only memory, random-access memory, or both. The basic components of a computer are the central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be complemented by or integrated into special-purpose logic circuits. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or is operablely coupled to receive data from them, transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be integrated into other devices, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, a Universal Serial Bus (USB) flash drive, etc.

[0115] Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROMs and DVD-ROM disks, and all forms of non-volatile memory, media, and memory devices.

[0116] To interact with a user, embodiments of the subject matter described herein can be implemented in a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal screen) monitor, and a keyboard and pointing device, such as a mouse or trackball, on which the user can input to the computer. Other types of devices can also be used to interact with a user. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer may interact with the user by sending a document to a device used by the user and receiving a document from the device used by the user (for example, by sending a web page to a web browser on the user's device in response to a request received from a web browser). The computer may also interact with the user by sending a text message or other form of message to a personal device (for example, a smartphone running a messaging application) and then receiving a response message from the user.

[0117] Data processing equipment for implementing machine learning models may include, for example, dedicated hardware accelerator units for processing the general and numerical computation portions (i.e., inference, workloads) of machine learning training or production.

[0118] Machine learning models can be implemented and deployed using machine learning frameworks, such as the TensorFlow framework or the Jax framework.

[0119] Embodiments of the subject matter described herein can be implemented in a computing system including, for example, a backend component as a data server, or a computing system including a middleware component, for example, an application server, or a frontend component, for example, a client computer having a graphical user interface, a web browser, or an application that allows a user to interact with embodiments of the subject matter described herein, or in a computing system including any combination of one or more such backend, middleware, or frontend components. The components of the system can be interconnected by digital data communications of any form or medium, for example, a communication network. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and for example, the Internet.

[0120] A computing system can include clients and servers. Clients and servers are generally remote to each other and typically interact via a communication network. The client-server relationship arises from computer programs that run on each computer and have a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from that user. Data generated on the user device (e.g., the results of user interactions) can be received by the server from the device.

[0121] This specification includes details of many specific embodiments, but these should not be construed as limiting the scope of any invention or claimable scope, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described herein as separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described as a single embodiment may also be implemented in multiple embodiments, individually or in any preferred secondary combination. Furthermore, even if several features are described above as functioning in a particular combination and are initially described as such in the claims, one or more features may be removed from the combination described in the claims, and furthermore, the combination described in the claims may cover secondary combinations or variations of secondary combinations.

[0122] Similarly, while operations are shown in the drawings and described in a specific order in the claims, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all shown operations be performed, in order to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0123] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims may be performed in a different order, and this may still yield desirable results. As an example, the process shown in the accompanying figures does not necessarily require that the actions be performed in the specific order or sequence shown to obtain the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method that is performed by one or more computers, (i) an image, and (ii) a set of one or more query embeddings, each query embedding obtaining a set of one or more query embeddings that represent each category of the object, To generate object detection data for the aforementioned image, an object detection neural network is used to process the image and the set of query embeddings, The image is processed using the image encoding subnetwork of the object detection neural network to generate a set of object embeddings. The object detection neural network's localization subnetwork is used to process each object embedding to generate localization data that defines the corresponding region of the image. Using the classification subnetwork of the object detection neural network, (i) the set of object embeddings and (ii) the set of query embeddings are processed to generate a classification score distribution for each object embedding across the set of query embeddings, Includes, The respective classification score distribution for each of the object embeddings defines, for each query embedding, the possibility that the region of the image corresponding to the object embedding depicts an object that belongs to the category represented by the query embedding. The set of the aforementioned image and the aforementioned query embedding is processed as described above, For one or more of the aforementioned object embeddings, Determining that the region of the image corresponding to the object embedding depicts an object included in the category represented by the query embedding based on the classification score distribution for the object embedding, The method, including the method described above.

2. To obtain one or more of the aforementioned query embeddings, Obtaining a text sequence that describes the category of an object, The text sequence is processed using the text encoding subnetwork of the object detection neural network to generate the query embeddings, The method according to claim 1, including the method described in claim 1.

3. The image encoding subnetwork and the text encoding subnetwork are pre-trained, and the pre-training is performed as follows: (i) a training image, (ii) a positive text sequence which characterizes the training image, and (iii) one or more negative text sequences which characterize the training image. The image encoding subnetwork is used to generate the embedding of the training image, Using the text encoding subnetwork, the embeddings of the positive text sequence and the negative text sequence are generated, (i) to increase the similarity between the embedding of the training image and the embedding of the positive text sequence, and (ii) to decrease the similarity between the embedding of the training image and the embedding of the negative text sequence, The method according to claim 2, comprising repeatedly performing an operation including the operation.

4. Using the image encoding subnetwork to generate the embedding of the training image is, The process involves using the image encoding subnetwork to process the training image and generate a set of object embeddings for the training image. The process involves using an embedded neural network to process the object embeddings and generate the embeddings in the training images, The method according to claim 3, including the method described in claim 3.

5. The method according to claim 4, wherein the embedded neural network is co-trained with the image encoding subnetwork and the text encoding subnetwork.

6. Joint training of the image encoding subnetwork and the text encoding subnetwork is To optimize the objective function including the control loss term, the image encoding subnetwork and the text encoding subnetwork are jointly trained. The method according to claim 3, including the method described in claim 3.

7. The method according to claim 3, wherein, after the pre-training of the image encoding subnetwork and the text encoding subnetwork, the object detection neural network is trained to optimize an objective function that measures the performance of the object detection neural network with respect to the task of detecting objects in an image.

8. The method according to claim 7, wherein the objective function for measuring the performance of the object detection neural network with respect to the task of detecting objects in an image includes a bipartite matching loss term.

9. Using the classification subnetwork of the object detection neural network, processing (i) the set of object embeddings and (ii) the set of query embeddings to generate a classification score distribution for each object embedding across the set of query embeddings is: This involves using one or more neural network layers of a classification neural network to process each object embedding and generate the corresponding classification embedding, For each object embedding, generate the classification score distribution across the set of query embeddings using (i) the classification embedding corresponding to the object embedding and (ii) the query embedding, A measure of similarity between the classification embedding and each query embedding, which defines the possibility that the region of the image corresponding to the object embedding depicts an object included in the category represented by the query embedding, thereby generating the measure of similarity between the classification embedding and the query embedding. The generation of the classification score distribution, which includes, The method according to claim 1, including the method described in claim 1.

10. Using one or more neural network layers of the aforementioned classification neural network to process each object embedding and generate the corresponding classification embedding is: Each classification embedding is generated by projecting the corresponding object embedding onto the latent space containing the query embedding. The method according to claim 9, including the method described in claim 9.

11. The generation of the respective similarity measurements between the classification embedding and each query embedding is performed for each query embedding. Calculate the inner product between the classification embedding and the query embedding. The method according to claim 9, including the method described in claim 9.

12. For each object embedding, processing the object embedding using the localization subnetwork to generate localization data that defines the corresponding region of the image is: The object embedding is processed using the aforementioned localization subnetwork to generate localization data that defines a bounding box within the image. The method according to claim 1, including the method described in claim 1.

13. Using the aforementioned image encoding subnetwork to process the image and generate the set of object embeddings is, The embedding layer of the image encoding subnetwork generates a set of initial object embeddings, each initial object embedding being generated, at least partially derived from a corresponding patch in the image. The initial set of object embeddings is processed by multiple neural network layers, including one or more self-attention neural network layers, to generate the final set of object embeddings. The method according to claim 1, including the method described in claim 1.

14. Using the aforementioned localization subnetwork to process object embedding and generate localization data that defines the corresponding region of the image, The method according to claim 13, comprising generating a set of offset coordinates, wherein the offset coordinates define an offset of the corresponding region of the image from the location of the image patch corresponding to the object embedding.

15. The method according to claim 2, wherein the text encoding subnetwork includes one or more self-attention neural network layers.

16. The image encoding subnetwork includes one or more self-attention neural network layers. The method according to claim 1.

17. To obtain one or more of the aforementioned query embeddings, The acquisition of one or more query images, each query image including a target region that depicts an example of the target object, To generate the respective embeddings of the target object in each query image, The query embedding is generated by combining the embedding of the target object in the query image, The method according to claim 1, including the method described in claim 1.

18. For each query image, generating the embedding of the target object within the query image is: The process involves using the aforementioned image encoding subnetwork to process the query image and generate a set of object embeddings for the query image, Using the aforementioned localization subnetwork, process the embedding of each object into the query image to generate localization data that defines the corresponding region of the image. For each object embedding in the query image, (i) determine the measured value of the overlap between the target region of the query image that depicts the example of the target object and (ii) the region of the query image that corresponds to the object embedding in the query image; Selecting an object embedding from the set of object embeddings for the query image as the embedding for the target object in the query image, based on the overlap measurement, The method according to claim 17, including the method described in claim 17.

19. Generating the query embedding by combining the embedding of the target object in the query image is, Averaging the embedding of the target object in the query image. The method according to claim 18, including the method described in claim 18.

20. One or more computers, One or more storage devices communicably coupled to one or more computers, the one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the operation of each of the methods described in any one of claims 1 to 19, A system that includes this.

21. One or more computer-readable storage media that, when executed by one or more computers, stores instructions causing one or more computers to perform the operation of each of the methods described in any one of claims 1 to 19.

Images