Mapping images to words for combined image retrieval

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for combined image retrieval. In one aspect, a method performed by one or more computers is described. The method includes receiving a query including: (i) an image depicting a scene, and (ii) a textual cue describing a context of the scene; processing the image using a visual encoder to generate a visual embedding of the image; processing the visual embedding of the image using a mapping neural network to generate one or more linguistic lems of the image; generating a plurality of language lexical elements of the text prompt; processing the language lemma of the image and the textual cue using a language encoder to generate a language embedding of the query; and selecting one or more of the candidate images from a number of candidate images using the language embedding of the query.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 439,446, filed on January 17, 2023. The disclosure of the prior application is considered part of and incorporated by reference into the disclosure of the present application. Background Art

[0003] This specification relates to using machine learning models to process data.

[0004] A machine learning model receives input and generates an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and generate an output based on the received input and the values ​​of the model parameters.

[0005] Some machine learning models are deep models, which use multiple layers to generate outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to generate an output. Summary of the Invention

[0006] This specification describes an image retrieval system implemented as a computer program on one or more computers in one or more locations that can retrieve images in response to received queries, eg, to perform composed image retrieval (CIR).

[0007] In the described example of an image retrieval system, a query may include: (i) a query image, and (ii) a query text prompt. The query may be provided by a user, an automated system (e.g., a computer program), or both. For example, a user may wish to obtain a target image that depicts a scene (e.g., including a particular object) in a desired context (e.g., with one or more additional objects present in the scene). The user may provide a query that includes: (i) a query image that depicts the scene, and (ii) a query text prompt that describes the context of the scene. The image retrieval system may process the query and retrieve one or more images that include the target image and / or variants of the target image.

[0008] Image retrieval systems can retrieve highly aligned images in response to a query, i.e., retrieved images that accurately produce the desired target image specified by the query. This is because image retrieval systems can be trained and subsequently implemented in a zero-shot fashion. "Zero-shot" generally refers to the type of inference that an image retrieval system can perform after training. Specifically, image retrieval systems can be trained in a weakly supervised and / or unsupervised manner, such that at inference, the image retrieval system can process the query and retrieve images that were not included in the training dataset.

[0009] An image retrieval system uses a visual encoder, a mapping neural network, and a language encoder to accomplish such CIR tasks. The visual encoder is configured to: receive an image; and process the image to generate a visual embedding of the image, e.g., representing the semantic content of a scene depicted in the image. The mapping neural network is configured to: receive the visual embedding of the image; and process the visual embedding to generate one or more language tokens of the image. The mapping neural network can be optimized so that the language tokens of the image faithfully represent the visual embedding of the image in a tokenized form. In this way, the image retrieval system can map the scene depicted in the image to language tokens, which can then be seamlessly combined with a textual prompt describing the context of the scene. Specifically, the language encoder is configured to: receive a sequence of language tokens, the sequence of language tokens including: (i) language tokens of the image, and (ii) language tokens of the textual prompt; and process the sequence of language tokens to generate a language embedding of the sequence, e.g., representing the semantic content of the scene depicted in the image within the context described by the textual prompt.

[0010] Typically, the visual encoder and the language encoder encode entities into the same embedding space, for example, the visual encoder and the language encoder are pre-trained to maximize the similarity between the visual embedding of an image and the language embedding of the textual description of the image. As a result, the image retrieval system can retrieve candidate images with visual embeddings that align with the language embedding of the query, representing a combination of the query image modified by the query text hint. This promotes high-fidelity CIR because the image retrieval system can flexibly compose query image features with textual modifications to isolate candidate images that reliably produce the intended target image.

[0011] As used herein, the term "scene" generally refers to any collection of one or more objects or generic "things" that may or may not interact in some way. For example, a scene can include multiple objects interacting with each other in an environment, e.g., a strawberry in a cup under a starry sky, or a brain flying to the moon on a rocket ship, or a strawberry cup filled with white sesame seeds floating in a sea of ​​dark chocolate. A scene can include a single object with no background or backdrop or with a solid color background or backdrop, e.g., a minimal kinetic sculpture of a bird on a white background. A scene can include text or abstract art, such as colors, shapes, lines, etc., e.g., a blue flame forming text or a watercolor-based painting.

[0012] These and other aspects of the subject matter described in this specification are summarized below.

[0013] In one aspect, a method performed by one or more computers is provided. The method includes receiving a query comprising: (i) an image depicting a scene, and (ii) a textual prompt describing the context of the scene; processing the image using a visual encoder to generate a visual embedding for the image; processing the visual embedding for the image using a mapping neural network to generate one or more language tokens for the image; generating a plurality of language tokens for the textual prompt; processing the image and the language tokens for the textual prompt using a language encoder to generate a language embedding for the query; and selecting one or more candidate images from a plurality of candidate images using the language embedding for the query.

[0014] In some implementations of the method, processing the language tokens of the image and the text prompt using the language encoder to generate the language embedding for the query includes: appending the one or more language tokens of the image to the language tokens of the text prompt to obtain a sequence of language tokens for the query; and processing the sequence of language tokens for the query using the language encoder to generate the language embedding for the query.

[0015] In some implementations of the method, selecting one or more candidate images from a number of candidate images using a language embedding of a query includes: for each candidate image from the number of candidate images: processing the candidate image using a visual encoder to generate a visual embedding of the candidate image; and selecting one or more candidate images from the number of candidate images based on (i) the visual embedding of the candidate image and (ii) the language embedding of the query.

[0016] In some implementations of the method, selecting one or more candidate images from a number of candidate images based on (i) a visual embedding of the candidate image and (ii) a language embedding of the query includes: for each candidate image from the number of candidate images: calculating a similarity measure between: (i) the visual embedding of the candidate image, and (ii) the language embedding of the query; and determining a score for the candidate image based on the similarity measure; and selecting one or more candidate images from the number of candidate images based on the scores of the number of candidate images.

[0017] In some implementations of the method, selecting one or more candidate images from a number of candidate images based on the scores of the many candidate images includes: sorting each of the many candidate images according to the scores of the many candidate images; and selecting one or more candidate images with the highest scores from the many candidate images.

[0018] In some implementations of the method, the context described by the text prompt includes one or more of: (i) domain conversion, (ii) object and scene composition, or (iii) sentence specification.

[0019] In some implementations of the method, a mapping neural network has been trained on training data comprising a number of training images each depicting a corresponding scene.

[0020] In some implementations of the method, the plurality of training images includes three million or more training images.

[0021] In some implementations of the method, the visual encoder and the language encoder are pre-trained neural networks and are kept frozen during training of the mapping neural network.

[0022] In some implementations of the method, the visual encoder and the language encoder are pre-trained on 400 million or more image-caption pairs.

[0023] In some implementations of the method, training the mapping neural network on the training data includes: for each training image in a number of training images: processing the training image using a visual encoder to generate a visual embedding for the training image; processing the visual embedding of the training image using the mapping neural network to generate one or more language tokens for the training image; providing a training text prompt for the training image; generating a number of language tokens for the training text prompts; and processing the language tokens of the training images and the training text prompts using a language encoder to generate a language embedding for the training image; determining a loss that is a loss function that depends on the visual embedding and the language embedding of each training image; and updating a set of network parameters of the mapping neural network based on the loss.

[0024] In some implementations of the method, for each training image in a plurality of training images, processing the training image and the language tokens of the training text prompt using the language encoder to generate the language embedding for the training image includes: appending the one or more language tokens of the training image to the language tokens of the training text prompt to obtain a sequence of language tokens for the training image; and processing the sequence of language tokens for the training image using the language encoder to generate the language embedding for the training image.

[0025] In some implementations of the method, the training text prompt is the same for each training image in the plurality of training images.

[0026] In some implementations of the method, the training text prompt includes: "A photo of".

[0027] In some implementations of the method, the loss function depends on normalized versions of the visual embeddings and the language embeddings of the training images.

[0028] In some implementations of the method, the loss function includes one or more contrastive loss terms that, for each training image, represent a similarity measure between: (i) a visual embedding of the training image, and (ii) a language embedding of the training image.

[0029] In some implementations of the method, the loss function includes a symmetric contrastive loss term.

[0030] In some implementations of the method, updating network parameters of the mapping neural network based on the loss includes: determining values ​​of the network parameters that minimize the loss; and updating the network parameters of the mapping neural network using the values ​​that minimize the loss.

[0031] In a second aspect, a system is provided, comprising one or more computers and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of any of the above methods.

[0032] In a third aspect, one or more non-transitory computer storage media are provided. The one or more non-transitory computer storage media store instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of any of the above methods.

[0033] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0034] The image retrieval system described in this specification can retrieve images in response to a received query, for example, to perform combined image retrieval (CIR).

[0035] Existing solutions to CIR generally have two drawbacks. First, existing solutions generally involve large amounts of labeled data to train CIR models via supervised learning. The labeled data is typically in the form of triplets, including: (i) a reference image, (ii) a text description, and (iii) a target image. Dataset collection efforts generally involve two processes: (i) collecting pairs of reference and target images to build query-output pairs, and (ii) providing a text description that modifies the reference image into the target image. Both steps can incur significant labeling costs. Second, CIR models trained on labeled data can be specialized for a specific use case and may not generalize to different tasks.

[0036] In contrast, the image retrieval system described in this specification can perform a variety of CIR tasks, such as object and scene composition, attribute editing, sentence assignment, and domain conversion, without the need for expensive labeled triplet datasets. In contrast to supervised learning, image retrieval systems can be trained via weakly supervised learning or unsupervised learning, for example, using large-scale weakly labeled image-caption pairs or unlabeled images. Compared to large-scale supervised datasets, these training methods can be much cheaper to implement in terms of computing resources (e.g., computing time, memory usage, etc.) and human effort, while also enabling zero-shot learning of CIR models, which is referred to herein as "zero-shot combinatorial image retrieval" (or ZS-CIR). These two features significantly improve the feasibility and versatility of CIR for real-world applications (e.g., e-commerce and internet search).

[0037] In addition, the image retrieval system described in this specification can generate visual embeddings of candidate images and / or stored images for performing high-fidelity CIR. The visual embeddings of the candidate images provide an efficient, compact, and effective indexing structure for searching a set of candidate images in response to a received query. For example, the candidate images can be stored in a database (e.g., a vector database) next to their corresponding visual embeddings, which act as an index for searching the database using the language embeddings of the query. The image retrieval system described herein can then perform a vector search algorithm on the database with respect to the language embeddings to efficiently retrieve candidate images in response to the query. For example, the image retrieval system can perform vector searches such as k-nearest neighbor searches (e.g., hierarchical navigable small world (HNSW) searches) and inverted file index (IVF) searches.

[0038] The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

[0039] Incorporated by Reference

[0040] All publications, patents, and patent applications mentioned in this specification are hereby incorporated by reference to the same extent as if each individual publication, patent, or patent application was individually and individually indicated to be incorporated by reference. In the event that the publications and patents or patent applications incorporated by reference conflict with the disclosure contained in this specification, this specification is intended to supersede and / or take precedence over any such conflicting material. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a schematic diagram of an example image retrieval system configured to perform combined image retrieval (CIR).

[0042] Figure 2 is a schematic diagram of an example training engine configured to train an image retrieval system.

[0043] Figure 3A is a flowchart of an example process for training an image retrieval system.

[0044] Figure 3B is a flow chart of an example process for updating a set of neural network parameters of a mapping neural network.

[0045] Figure 4 is a schematic diagram of an example retrieval engine configured to retrieve images in response to a query using an image retrieval system.

[0046] Figure 5A is a flow diagram of an example process for retrieving images in response to a query using an image retrieval system.

[0047] Figure 5B is a flow chart of an example process for selecting one or more candidate images from a set of candidate images.

[0048] Figures 6A-6C Examples of domain conversion, object and scene manipulation, and sentence specification tasks performed by image retrieval systems are shown.

[0049] Like reference numbers and designations throughout the various drawings refer to like elements. DETAILED DESCRIPTION

[0050] In combinatorial image retrieval (CIR), a user (or other entity) combines a query image with a textual prompt to describe their intended target image. Existing methods rely on supervised learning of CIR models via labeled triplets consisting of: (i) a reference image, (ii) a textual description, and (iii) a target image. Labeling such triplets is expensive and hinders the widespread applicability of CIR. Dataset collection efforts typically involve two processes: (i) collecting pairs of reference and target images to build query-output pairs, and (ii) providing a textual description that modifies the reference image into the target image. Both steps incur significant labeling costs. Furthermore, neural network models trained on labeled training data are often specialized for specific use cases (e.g., overtrained) and may not generalize to different CIR tasks, further limiting the widespread application of such models.

[0051] Among other things, this specification introduces novel systems and methods for zero-shot CIR (ZS-CIR), allowing the construction of CIR models that do not require labeled triplets for training. The image retrieval systems described herein can be trained on large-scale weakly labeled image-caption pairs and / or unlabeled images, which are much cheaper to collect than large-scale supervised training datasets. Unlike existing supervised models, the image retrieval systems described herein, trained via weakly supervised and / or unsupervised learning, exhibit strong generalization across a diverse range of CIR tasks, such as attribute compilation, object and scene composition, and domain conversion.

[0052] To take advantage of weakly labeled and unlabeled datasets, this specification introduces a two-stage training engine for zero-shot learning of CIR models. The image retrieval system described herein includes a visual encoder, a mapping neural network, and a language encoder. The training engine first jointly pre-trains the visual encoder and the language encoder on an image-caption dataset to maximize the similarity between the image embedding and the textual description of the image. Instead of relying on a triple-labeled training dataset, the training engine leverages the linguistic capabilities of the language encoder, which excels at generating language embeddings that are highly aligned with the corresponding visual embeddings generated by the visual encoder. The training engine then trains the mapping neural network to convert the visual embeddings generated by the visual encoder into one or more language tokens that can be efficiently processed by the language encoder. Of particular note, the mapping neural network can be trained on unlabeled images to reconstruct the visual embeddings. Therefore, the language encoder can flexibly construct query image features and textual descriptions in the language embeddings, thereby facilitating high-fidelity ZS-CIR.

[0053] These and other features related to the systems and methods disclosed in this specification are described in more detail below. Further details related to the systems and methods disclosed in this specification, such as additional experiments and discussions, are provided by Orvieto, Antonio et al., "Resurrecting recurrent neural networks for long sequences," arXiv preprint arXiv:2303.06349 (2023).

[0054] Figure 1 is a schematic diagram of an example image retrieval system 100 configured to perform CIR. Image retrieval system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0055] At a high level, image retrieval system 100 is configured to receive and process a query 10 comprising: (i) a query image 11 depicting a scene, and (ii) a query textual hint 12 describing the context of the scene. Image retrieval system 100 can then retrieve one or more images 20 depicting the scene in the context. By incorporating textual hint 12, such as a user's intent, such as a desired modification to query image 11, image retrieval system 100 allows for high-accuracy retrieval of images 20. More generally, query 10 can be provided by a user, an automated system (e.g., a computer program), or both. As an example, a user may desire a target image depicting a specific scene (e.g., including a dog) in a specific context (e.g., in a pool). The user may provide query 10 comprising: (i) a query image 11 depicting the specific scene, and (ii) a query textual hint 12 describing the specific context of the scene. Image retrieval system 100 can process query 10 and retrieve one or more images 20 including the target image (e.g., a dog in a pool) and / or variations of the target image.

[0056] The image retrieval system 100 can be used for a variety of real-world applications, such as e-commerce and internet search. The image retrieval system 100 can also perform various CIR tasks, such as attribute manipulation for fashion image search, composing objects, converting image styles for content creation, etc. In addition, the image retrieval system 100 can process and retrieve a variety of different types of images, such as three-dimensional (3D) images, photorealistic images, cartoon images, abstract visualizations, point cloud images, medical images of different modalities, etc. For example, medical images may include, but are not limited to, magnetic resonance imaging (MRI) images, computed tomography (CT) images, ultrasound images, x-ray images, etc. The types of scenes that can be depicted in the query image 11 and modified by the text prompt 12 are diverse, ranging from real-world settings to abstract ones. Figure 1 Several selection examples are depicted in .

[0057] like Figure 1As shown, for a first query 10-1, query image 11-1 is an image of a goldfish, and query text hint 12-1 includes "Origami of ___," where "___" represents query image 11-1 modified by text hint 12-1. Image retrieval system 100 processes query 10-1 and retrieves one or more images 20-1 of origami goldfish. For a second query 10-2, query image 11-2 is an image of a dog (of a specific breed), and query text hint 12-2 includes "___in the pool." Image retrieval system 100 processes query 10-2 and retrieves one or more images 20-2 of a dog of a specific breed in a pool. For a third query 10-3, query image 11-3 is an image of a white dress, and query text hint 12-3 includes "___with blue floral print." Image retrieval system 100 processes query 10-3 and retrieves one or more images 20-3 of a dress with a blue floral print.

[0058] As illustrated above, the textual prompt 12 may describe the type of CIR task that the image retrieval system 100 intends to perform on the query image 11. Such CIR tasks may include, but are not limited to, domain conversion, object and scene composition, sentence specification, and others. Domain conversion generally refers to textual prompts 12 that intend to modify the domain of a scene (e.g., converting a real-world scene to a sketch-style scene). Object and scene composition generally refers to textual prompts 12 that intend to add one or more objects and / or modify the scene of the query image 11. Sentence specification generally refers to textual prompts 12 that intend to modify the scene based on the sentence included in the textual prompt 12.

[0059] As an example of domain conversion, if the query image 11 is a real image of a hammerhead shark and the text hint 12 includes "Asculpture of___," the image retrieval system 100 may process the corresponding query 10 and retrieve one or more images 20 of sculptures of hammerhead sharks. Figure 6A Several examples of domain conversion that may be performed by the image retrieval system 100 are shown.

[0060] As an example of object and scene composition, if the query image 11 is an image of a computer mouse and the text hint 12 includes "computer mouse, laptop", the image retrieval system 100 can process the corresponding query 10 and retrieve one or more images 20 of computer mice and laptop computers. Figure 6B Several examples of object and scene composition that may be performed by the image retrieval system 100 are shown.

[0061] As an example of sentence modification, if the query image 11 is an image of a black shirt and the text prompt 12 includes "___has a red logo," the image retrieval system 100 may process the corresponding query 10 and retrieve one or more images 20 of a black shirt with a red logo. Figure 6C Several examples of sentence specification that may be performed by the image retrieval system 100 are shown.

[0062] Due to the flexibility of the image retrieval system 100, a query 10 may include multiple query images 11, rather than just a single query image. The textual hint 12 may then describe the common or related context of each of the query images 11. For example, if each query image 11 depicts a corresponding object, the textual hint 12 may describe the common context for the objects, e.g., "___, ___, and ___ in the forest," where each "___" represents a different query image 11 modified by the textual hint 12. The image retrieval system 100 may process the corresponding query 10 and retrieve one or more images 20 of the object in the forest. As another example, the textual hint 12 may describe the modification of the first query image using a second, different query image, e.g., "___ with the style of ___." The image retrieval system 100 may process the corresponding query 10 and retrieve one or more images 20 of the object depicted in the first query image rendered in the artistic style of the second query image.

[0063] To perform this type of CIR task, image retrieval system 100 includes a visual encoder 110, a mapping neural network 120, and a language encoder 130. Furthermore, visual encoder 110 may also be referred to as an image encoder, and language encoder 130 may also be referred to as a text encoder. These terms are often used interchangeably. However, for consistency, this document uses the terms "visual encoder" and "language encoder."

[0064] The visual encoder 110 is configured to: receive an image (x); and process the image to generate a visual embedding of the image For example, the visual embedding of an image can represent the semantic content of the scene depicted in the image. Here, V φ is a function representing the neural network model of the visual encoder 110, which is parameterized by a corresponding set of network parameters (φ).

[0065] The visual encoder 110 can have any suitable neural network architecture that enables it to perform its described function (i.e., process an image to generate a visual embedding of the image). Specifically, the visual encoder 110 can include any suitable type of neural network layers (e.g., fully connected layers, convolutional layers, self-attention layers, etc.) connected in any suitable number (e.g., 5 layers, 25 layers, or 100 layers) and in any suitable configuration (e.g., as a linear sequence of layers).

[0066] The mapping neural network 120 is configured to: receive the visual embedding of the image and one or more language tokens that process the visual embedding to generate an image Here, f θ is a function representing a neural network model of the mapping network 120, the function being parameterized by a corresponding set of network parameters (θ). In some implementations, the mapping network 120 converts the visual embedding of the image into a single language token, e.g., a "pseudo" language token. In some cases, a single language token may be advantageous for combining with other lemmatized text. For example, a single language token representation may correspond to a word or subword token. On the other hand, in some cases, multiple language tokens may more accurately capture the semantic content of the visual embedding, e.g., depending on the size, detail, and / or information content of the visual embedding. For example, multiple language token representations may correspond to multiple word or subword tokens or multiple character tokens. In any case, the mapping neural network 120 may be optimized such that the language tokens of the image faithfully represent the visual embedding of the image in a lemmatized form.

[0067] The mapping neural network 120 can have any suitable neural network architecture that enables it to perform its described function (i.e., processing a visual embedding of an image to generate one or more language tokens for the image). Specifically, the mapping neural network 120 can include any suitable type of neural network layers (e.g., fully connected layers, convolutional layers, self-attention layers, etc.) connected in any suitable number (e.g., 5 layers, 25 layers, or 100 layers) and in any suitable configuration (e.g., as a linear sequence of layers).

[0068] The language encoder 130 is configured to: receive a language word and processing the sequence of language tokens to generate a language embedding of the sequence The sequence of language tokens may include: (i) one or more language tokens of an image depicting a scene, and (ii) multiple language tokens of a textual prompt describing the context of the scene. For example, the language tokens of the image may be appended to the language tokens of the textual prompt to obtain a sequence of language tokens. In this case, the language embedding of the sequence may represent the semantic content of the scene depicted in the image in the context described by the textual prompt. Here, is a function representing the neural network model of the language encoder 130, which is composed of network parameters The corresponding set parameterization of .

[0069] Language encoder 130 can have any suitable neural network architecture that enables it to perform its described function (i.e., processing a sequence of language tokens to generate a language embedding for the sequence). Specifically, language encoder 130 can include any suitable type of neural network layers (e.g., fully connected layers, convolutional layers, self-attention layers, etc.) connected in any suitable number (e.g., 5 layers, 25 layers, or 100 layers) and in any suitable configuration (e.g., as a linear sequence of layers).

[0070] Typically, the visual encoder 110 and the language encoder 130 encode entities into the same embedding space In other words, visual embeddings and language embeddings are embedding vectors with the same dimension (d). Higher-dimensional embedding spaces can provide more granularity in embeddings, for example, encoding more contextual features of image and language data. The image retrieval system 100 can be encoded into an embedding space of any size, for example, with dimensions of 10, 25, 50, 100, 200, 500, 1000, 2000, 5000, 10000 or more. Embeddings can also be referred to as encoded representations that provide computationally amenable representations for processing. In general, embeddings can be sets, vectors, or arrays of values ​​(for example, encoded in UNICODE or Base64), alphanumeric values, symbols, or any convenient encoding. However, for ease of description, this article will describe visual embeddings and language embeddings based on vector representations in (vector) embedding space.

[0071] References below Figure 2-3B A detailed method for training the image retrieval system 100 is described. Figure 4-5BDescription Inference. In the described example of the image retrieval system 100, a pre-trained image-to-text model (e.g., a contrastive language-image pre-training (CLIP) model) is used for the visual encoder 110 and the language encoder 130. A more detailed review of the CLIP encoder is provided by Radford, Alec, et al., "Learning Transferable Visual Models from Natural Language Supervision," International Conference on Machine Learning, PMLR (2021). Here, the output embeddings of the visual encoder 110 and the language encoder 130 are aligned with respect to each other's modality. Given the frozen and pre-trained encoders 110 and 130, a mapping neural network 120 is trained to efficiently convert the visual embedding of the image into one or more language tokens of the image. The mapping network 120 can be optimized so that the language tokens faithfully represent the visual embeddings. At inference time, the language tokens of the query image 11 can be appended to the language tokens of the query text prompt 12. The resulting language embedding of the query 10 can then be compared with the visual embeddings of the candidate images for retrieval.

[0072] The image retrieval system 100 can be implemented in any suitable location, such as on a user device (e.g., a mobile device), or on one or more computers in a data center, etc. The modularity of the image retrieval system 100 allows multiple devices to implement individual components of the system 100 separately from each other. For example, the visual encoder 110, the mapping neural network 120, and the language encoder 130 can be executed on different devices and can send their output and / or input to each other (e.g., via telecommunications). A user can interact with the image retrieval system 100, for example, by providing a query to the image retrieval system 100 via an interface (e.g., a graphical user interface or an application programming interface (API)). Specifically, the user can provide input that includes: (i) a request to retrieve a target image, and (ii) a query 10 specifying the target image, for example, including one or more query images 11, and a text prompt 12. In response to receiving the input, the image retrieval system 100 may retrieve one or more images 20 in response to the request, for example, including a target image and / or a variant of the target image, and provide the images 20 to the user, for example, for display on the user's user device or for storage in a data storage device. In some cases, the image retrieval system 100 may, for example, send the images 20 to the user's user device via a data communication network (e.g., the Internet).

[0073] Figure 2is a schematic diagram of an example training engine 200 configured to train image retrieval system 100. Training engine 200 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0074] As described above, the training engine 200 typically performs a two-stage training scheme. First, the training engine 200 pre-trains the visual encoder 110 and the language encoder 130 on an image-caption pre-training dataset to maximize the similarity between the image embedding and the image's textual description. Thereafter, the training engine 200 trains the mapping network 120 on a weakly labeled and / or unlabeled training dataset to learn a mapping from visual embeddings to language tokens, while keeping the visual encoder 110 and the language encoder 130 frozen.

[0075] First, the training engine 200 obtains N0 training examples (x n , t n ) pre-training dataset For example, the training engine 200 may sample a pre-training dataset from an annotated image repository such as ImageNet and / or ImageNet-R. The pre-training dataset may include millions, tens of millions, hundreds of millions, or more training examples. For example, in CLIP, the training engine 200 may use over 400 million training examples. Each training example is an image-caption pair, comprising: (i) a training image depicting a scene (x n ), and (ii) a (lemmatized) training text prompt describing the scene (t n In general, the training text prompts may be in the form of characters, words, subwords, sentences, or other lemmatized forms of natural language. However, for integration with the mapping neural network 120, word, subword, or character lemmatization is generally preferred.

[0076] The training engine 200 uses the visual encoder 110 to process the training image of each training example to generate a visual embedding of the training image. In a similar manner, the training engine 200 uses the language encoder 130 to process the training text prompt of each training example to generate a language embedding for the training text prompt.

[0077] The training engine 200 then performs a pre-trained loss function by optimizing the visual embedding and language embedding of each training example in the batch. In one or more batches of training examples The visual encoder 110 and the language encoder 130 are trained jointly. The training engine 200 can use various optimization techniques to optimize the pre-trained loss function for batches of training examples, such as stochastic gradient descent methods such as implicit updates, momentum, AdaGrad, RMSProp, Adam, etc.

[0078] In some implementations, the training engine 200 uses a contrastive pre-training loss function, such as CLIP, which is designed to find representations that match training images with their paired textual prompts while separating unpaired ones. Specifically, for the i-th training image (x i ) and the jth lemmatized training text prompt (tj), Their embedding vectors are normalized to and Here, the training engine 200 trains the network parameters φ and φ of the visual encoder 110 and the language encoder 130. Optimize the symmetric contrast loss function:

[0079]

[0080] In other words, the training engine 200 determines the values ​​of the network parameters for the visual encoder 110 and the language encoder 130 that minimize the associated loss in equation (1). For example, when implementing a stochastic gradient descent method, the training engine 200 may use backpropagation to determine the gradients of the pre-trained loss function with respect to the network parameters of the visual encoder 110 and the language encoder 130. The training engine 200 may then apply an appropriate update rule (e.g., with a specific learning rate and / or weight decay) to use the gradients to update the network parameters. In this case, the pre-trained loss function includes two contrastive loss terms: and The temperature hyperparameter (τ) controls the penalty strength on hard negative examples, as follows:

[0081]

[0082] as well as,

[0083]

[0084] in represents the batch size. As shown in equations (2) and (3), the contrastive loss term depends on the orientation of the embedding vectors (but not on their magnitude). For example, The term characterizes the similarity measure between the visual embedding and the language embedding as a function of the normalized scalar product (or cosine similarity) between the two. If the scalar product If the scalar product is close to 1, corresponding to parallel embedding vectors, the similarity measure is enhanced due to the exponential. If the scalar product is close to -1, corresponding to antiparallel embedding vectors, the similarity measure is suppressed (approaches zero).

[0085] Now refer to Figure 2 , which schematically illustrates how the training engine 200 trains the mapping neural network 120. As described above, the visual encoder 110 and the language encoder 130 are optimized by the training engine 200 to align the visual embedding of the image with the language embedding of the textual prompt describing the image. Therefore, one or more language tokens of an image can faithfully represent the image semantics if the language embedding of the universal text prompt to which the language token is attached is aligned with the visual embedding of the image. To this end, the training engine 200 trains the mapping network 120 to map the visual embedding to a language token that is compatible with the language embedding generated by the language encoder 130.

[0086] To this end, the training engine 200 obtains N training images (x n ) training dataset For example, the training engine 200 can sample the training dataset from a public repository of stock images or other image sources. The training dataset 102 can include the same and / or different training images as the pre-training dataset and can be of similar or dissimilar size. For example, the training dataset 102 can include one million or more, two million or more, three million or more, four million or more, five million or more, ten million or more, twenty million or more, fifty million or more, or one hundred million or more training images 104. It is also noted that the training dataset 102 is unlabeled, i.e., does not include a corresponding target image for each training image 104.

[0087] The training engine 200 processes each training image 104 using the visual encoder 110 to generate a visual embedding of the training image. The training engine 200 then processes the visual embedding 112 of each training image 104 using the mapping neural network 120 to generate one or more language tokens for the training image. As described above, the language tokens 122 may correspond to characters, words, or subword tokens, depending on the lemmatized representation implemented by the image retrieval system 100 .

[0088] The training engine 200 then provides a training text prompt 106 for each training image 104. Here, the training text prompt 106 is generic and identical for each training image 104. For example, the training text prompt 106 may include a text description such as "A photo of ___," "An image of ___," "A depiction of ___," and so on, where "___" represents a language token 122 attached to the training text prompt 106. Typically, such training text prompts 106 do not provide labeling for the training images 104, i.e., they do not describe any information about the scene depicted in the training images 104. In these cases, the training text prompt 106 may specify to the language encoder 130 that the language token 122 corresponds to a language token of the actual image, rather than, for example, a sentence. In some implementations, the training engine 200 may not provide any training text prompts 106 for the training images 104, such that the language token 122 is processed directly by the language encoder 130. In some implementations, the training engine 200 can provide training text prompts 106 that weakly label the training images 104, for example, training text prompts 106 that describe the artistic style of the training images 104, such as "A real-word photo of___," "An abstract photo of___," "A sketch-style photo of___," etc.

[0089] The training engine 200 generates a plurality of language tokens (t) for the training text prompt 106. For example, the training engine 200 can tokenize the training text prompt 106 using character, word, subword, sentence, or other tokenization schemes to generate the language tokens 126 for the training text prompt 106. As described above, character, word, or subword tokenization may be preferred for combination with the language tokens 122 for the image, e.g., depending on the number of language tokens that the mapping network 120 is configured to output for the image.

[0090] The training engine 200 appends the language word gram 122 of each training image 104 to the language word gram 124 of the training text prompt 106 to obtain the language word gram for the training image 104. The training engine 200 then processes the sequence of language tokens 122 for each training image 104 using the language encoder 130 to generate a language embedding for the training image.

[0091] The training engine 200 then performs a training operation by optimizing a loss function that depends on the visual embedding 112 and the language embedding 132 of each training image 104 in the batch. In one or more batches of training images The training mapping neural network 120 is trained on the training image. As described above, the training engine 200 can use various optimization techniques to optimize the loss function 140 for the batch of training images, such as stochastic gradient descent methods such as implicit update, momentum, AdaGrad, RMSProp, Adam, etc.

[0092] In some implementations, the training engine 200 uses a contrastive loss function 140, such as CLIP, which forces the mapping network 120 to form a loop starting from the visual embedding 112 to the corresponding language embedding 132. The i-th training image (x i ), the visual embedding vector 112 and the language embedding vector 132 of the training image are normalized to and Here, the training engine 200 optimizes the symmetric contrastive loss function 140 with respect to the network parameters θ of the mapping network 120:

[0093]

[0094] In other words, the training engine 200 determines values ​​for the network parameters of the mapping neural network 120 that minimize the associated loss in equation (4). For example, when implementing a stochastic gradient descent method, the training engine 200 may use backpropagation to determine the gradients of the loss function 140 with respect to the network parameters of the mapping neural network 120. The training engine 200 may then apply an appropriate update rule (e.g., with a specific learning rate and / or weight decay) to use the gradients to update the network parameters. In this case, the loss function 140 includes two contrastive loss terms: and as follows:

[0095]

[0096] as well as,

[0097]

[0098] In this way, the training engine 200 trains the mapping network 120 to align the language embedding 132 of each training image 104 with its corresponding visual embedding 114 while keeping the visual encoder 110 and the language encoder 130 frozen.

[0099] Figure 3Ais a flow chart of an example process 400 for training the image retrieval system 100. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training engine appropriately programmed according to the present specification, such as Figure 2 The training engine 200 may execute process 400.

[0100] In some implementations of process 400, visual encoder 110 and language encoder 130 are pre-trained neural networks and are kept frozen during the training of mapping neural network 120. For example, visual encoder 110 and language encoder 130 may be ViT-L / 14CLIP-based neural network models pre-trained on approximately 400 million image-caption pairs.

[0101] The training engine obtains a training data set (410) comprising a plurality of training images each depicting a respective scene. In some implementations of process 400, the training data set comprises three million or more training images.

[0102] For each training image:

[0103] The training engine processes the training images using the visual encoder 110 to generate visual embeddings of the training images (420).

[0104] The training engine processes the visual embeddings of the training images using the mapping neural network 120 to generate one or more language tokens for the training images (430).

[0105] The training engine provides training text prompts for the training images (440). In some implementations of process 400, the training text prompt is the same for each training image, for example, including a general description of each training image. For example, the training text prompt may include: "A photo of".

[0106] The training engine generates a plurality of language tokens for the training text prompt (450).For example, the training engine may tokenize the training text prompt, such as using character, word, or subword tokenization, to obtain language tokens for the training text prompt.

[0107] The training engine appends the one or more language tokens of the training image to the language tokens of the training text prompt to obtain a sequence of language tokens for the training image (460).

[0108] The training engine processes the sequence of language tokens for the training image using the language encoder 130 to generate a language embedding for the training image (470).

[0109] The training engine determines a loss that is a loss function that depends on the visual embedding and the language embedding of each training image (480). In some implementations of process 400, the loss function depends on normalized versions of the visual embedding and the language embedding of the training image. In some implementations of process 400, the loss function includes one or more contrastive loss terms that, for each training image, represent a measure of similarity between: (i) the visual embedding of the training image, and (ii) the language embedding of the training image. For example, the loss function can include a symmetric contrastive loss term.

[0110] The training engine updates the set of network parameters of the mapping neural network 120 according to the loss (490).

[0111] Figure 3B is a flow chart of an example process 490 for updating network parameters of the mapping neural network 120. For convenience, the process 490 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training engine appropriately programmed according to the present specification, such as Figure 2 The training engine 200 may execute process 490.

[0112] The training engine determines values ​​of the network parameters that minimize the loss (492).

[0113] The training engine updates the network parameters of the mapping neural network 120 using the value that minimizes the loss (494). For example, the training engine may use a stochastic gradient descent method with a specific learning rate and / or weight decay, such as implicit update, momentum, AdaGrad, RMSProp, or Adam, to update the network parameters of the mapping neural network 120.

[0114] Figure 4 is a schematic diagram of an example retrieval engine 300 configured to retrieve images 20 in response to a query 10 using the image retrieval system 100. The retrieval engine 300 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0115] At inference after training, the retrieval engine 300 may receive a query 10 comprising: (i) a query image 11 depicting a scene, and (ii) a query text prompt 12 describing the context of the scene. The retrieval engine 300 may process the query 10 using the image retrieval system 100 to obtain a language embedding 132 for the query 10, which is concisely represented as a vector The retrieval engine 300 then uses the language embedding 132 of the query 10 to retrieve one or more images 20 in response, for example, depicting a scene in the context.

[0116] In more detail, the search engine 300 processes the query image 11 using the visual encoder 110 to generate a visual embedding 112 for the query image 11. The search engine 300 then processes the visual embedding 112 for the query image 11 using the mapping neural network 120 to generate one or more language tokens 122 for the query image 11. The search engine 300 generates a plurality of language tokens 124 for the text prompt 12, for example, by tokenizing the text prompt 12, as described elsewhere herein. The search engine 300 then appends the language tokens 122 for the query image 11 to the language tokens 124 for the text prompt 12 to obtain a sequence of language tokens 126 for the query 10. The search engine 300 processes the sequence of language tokens 126 for the query 10 using the language encoder 130 to generate a language embedding 132 for the query 10.

[0117] Before, after, and / or in parallel, the search engine 300 obtains a set of candidate images. Including N c candidate images (x k ). The retrieval engine 300 selects one or more images 20 from a set 302 of candidate images. The set 302 of candidate images may include hundreds, thousands, tens of thousands, hundreds of thousands, millions, tens of millions, or more candidate images 30, depending on the implementation. For example, the retrieval engine 300 may sample the set 302 of candidate images from a public repository of stock images (or another image source). The set 302 of candidate images may correspond to one or more categories of images in the repository. The image retrieval system 300 may isolate the one or more categories of images, e.g., cartoons, origami, toys, sculptures, tables, stoves, sombreros, etc., and then perform a vector search 306 on the set 302 of candidate images that belong to such categories. For example, a user may provide input that includes: (i) a request to retrieve target images of a particular category, and (ii) a query 10 that specifies the target images. In response to receiving the input, the search engine 300 may sample a set 302 of candidate images corresponding to the particular category in response to the request and thereafter retrieve one or more images 20 from the set 302, as described below. Alternatively or additionally, the search engine 300 may use the query 10 to determine one or more categories for the image. For example, the search engine 300 may perform a reverse image search on the query image 10 and / or a keyword search on the textual hint 12 to determine one or more categories. After obtaining the set 302 of candidate images, the search engine 300 then uses the language embedding 132 of the query 10 to retrieve one or more images 20 from the set 302.

[0118] In more detail, the retrieval engine 300 processes each candidate image 30 in the set 302 using the visual encoder 110 to generate a visual embedding of the candidate image 30. The retrieval engine 300 then performs a vector search 306 on the set 302 of candidate images with respect to the visual embedding 113 of the candidate image 30 and the language embedding 132 of the query 10. Specifically, the retrieval engine 300 computes the corresponding similarity measures (α k ): (i) the language embedding 132 of the query 10, and (ii) the visual embedding 113 of each candidate image 30 in the set 302. For example, the search engine 300 may calculate the cosine similarity for each candidate image 30 Or other appropriate similarity measures. The search engine 300 may then determine a corresponding score (σ ) for each candidate image 30 based on the similarity measure between the candidate image 30 and the query 10. k ). For example, the score for a candidate image 30 may be a linear or nonlinear function of the candidate image's corresponding similarity measure, such as an exponential function σ k =exp(τα k ) or the logic function σ k =[1+exp(-τα k )] -1 The retrieval engine 300 then ranks the candidate images 30 according to their scores, for example, from smallest relative value to largest relative value, thereby selecting one or more candidate images with the highest scores among the candidate images 30 as output 20 in response to the query 10 .

[0119] In some implementations, the retrieval engine 300 maintains a database, such as a vector database, of candidate images 30. In these implementations, the image retrieval engine 300 may process each candidate image 30 in the database using the visual encoder 110 to generate a visual embedding for the candidate image 30 before receiving the query 10. The resulting visual embeddings 113 can be stored in a database alongside their corresponding candidate images 30. Specifically, candidate images 30 having similar visual embeddings 113 can be arranged to be closest to each other, for example, based on a cosine similarity or inverse distance metric between the visual embeddings 113. The search engine 300 can then use the visual embeddings 113 as an index for efficiently searching the database to find candidate images 30 relevant to the query 10. For example, when the search engine 300 generates a language embedding 132 for the query 10, the search engine 300 can perform a vector search algorithm 306 on the database of candidate images 30 with respect to the language embedding 132. Examples of vector search algorithms 306 that can be performed by the search engine 300 include, but are not limited to, k-nearest neighbor searches (e.g., hierarchical navigable small world (HNSW) searches) and inverted file index (IVF) searches. This can significantly increase the speed of the vector search 306 performed by the search engine 300, for example, by eliminating the need to evaluate each visual embedding 113 stored in the database against the language embedding 132, as in a brute-force search.

[0120] Note that the operation of the retrieval engine 300 follows similar steps in cases when the query 10 includes multiple query images 11. In these cases, the retrieval engine 300 processes each query image 11 in the query 10 using the visual encoder 110 and the mapping neural network 120 to generate one or more language tokens 122 for the query image 11. The retrieval engine 300 then appends the language tokens 122 of each query image 11 to the language tokens 122 of the textual prompt 12 to obtain a sequence of language tokens 126 for the query 10. This is then processed by the language encoder 130 to generate a language embedding 132 for the query 10, which represents, for example, a combination of the query images 11 modified by the textual prompt 12. (See Figure 1 Describes an example of a multi-image text prompt 12. ) The search engine 300 may then use the language embedding 132 of the multi-image query 10 to retrieve one or more images 20, as described above (e.g., Figure 4 ) as described.

[0121] Figure 5A is a flow chart of an example process 500 for retrieving images in response to a query using the image retrieval system 100. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a retrieval engine appropriately programmed according to the present specification, such as Figure 4 The search engine 300 can execute process 500.

[0122] The search engine receives a query that includes: (i) an image depicting a scene, and (ii) a textual prompt describing the context of the scene (510). For example, the context described by the textual prompt may include one or more of: (i) domain conversion, (ii) object and scene composition, or (iii) sentence specification.

[0123] The retrieval engine processes the image using the visual encoder 110 to generate a visual embedding of the image (520).

[0124] The search engine processes the visual embedding of the image using the mapping neural network 120 to generate one or more language tokens for the image (530).

[0125] The search engine generates a plurality of language tokens for the text prompt (540).For example, the search engine may tokenize the text prompt, such as using character, word, or subword tokenization, to obtain language tokens for the text prompt.

[0126] The search engine appends the one or more language tokens of the image to the language tokens of the text prompt to obtain a sequence of language tokens for the query (550).

[0127] The search engine processes the sequence of language tokens for the query using the language encoder 130 to generate a language embedding for the query (560).

[0128] The retrieval engine uses the language embedding of the query to select one or more of the candidate images from the set of candidate images (570).

[0129] Figure 5B is a flow chart of an example process 570 for selecting one or more candidate images from a set of candidate images using a language embedding of a query. For convenience, process 570 will be described as being performed by a system of one or more computers located in one or more locations. For example, a retrieval engine appropriately programmed according to the present specification, such as Figure 4 The search engine 300 may execute process 570 .

[0130] For each candidate image:

[0131] The retrieval engine processes the candidate image using the visual encoder 110 to generate a visual embedding of the candidate image (571).

[0132] The retrieval engine computes a similarity measure between: (i) the visual embeddings of the candidate images, and (ii) the linguistic embedding of the query (572).

[0133] The retrieval engine determines scores for the candidate images based on the similarity measure (573).

[0134] The search engine ranks each of the candidate images in the set according to the score of the candidate images (574).

[0135] The retrieval engine selects one or more candidate images with the highest scores from the set of candidate images (575).

[0136] This specification uses the term "configured" in conjunction with system and computer program components. With respect to a system of one or more computers to be configured to perform specific operations or actions, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform those operations or actions. With respect to one or more computer programs to be configured to perform specific operations or actions, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operations or actions.

[0137] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuit systems, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing device or for controlling the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or in addition, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by the data processing device.

[0138] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of equipment, devices, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0139] A computer program, which may also be referred to or described as a program, software, software application, application, module, software module, script, or code, may be written in any form of programming language (including compiled or interpreted languages ​​or declarative or procedural languages); and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or code portions). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0140] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed and run on the same computer or computers.

[0141] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry (e.g., an FPGA or an ASIC), or by a combination of special purpose logic circuitry and one or more programmed computers.

[0142] A computer suitable for executing a computer program may be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or random access memory, or both. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory may be supplemented by or incorporated into a dedicated logic circuit system. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or be operatively coupled to receive data from or transfer data to or both. However, a computer need not have such devices. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0143] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks.

[0144] To provide for user interaction, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to provide for user interaction; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. Furthermore, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. Furthermore, a computer may interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving a response message from the user in return.

[0145] A data processing device used to implement a machine learning model may also include, for example, dedicated hardware accelerator units for processing general-purpose and computationally intensive parts of machine learning training or production (i.e., inference, workloads).

[0146] Machine learning models can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework).

[0147] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0148] A computing system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers, placing them in a client-server relationship. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, for the purpose of displaying data to a user interacting with the device acting as a client and receiving user input from the user. Data generated at the user device, such as the results of the user interaction, may be received at the server from the device.

[0149] Although this specification contains many specific implementation details, these details should not be interpreted as limiting the scope of any invention or the scope of what may be claimed, but rather as descriptions of features that may be unique to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable subcombination. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may involve a subcombination or a variant of a subcombination.

[0150] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be construed as requiring that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0151] Specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown or sequential order to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more computers, the method comprising: receiving a query comprising: (i) an image depicting a scene, and (ii) a textual prompt describing the context of the scene; processing the image using a visual encoder to generate a visual embedding of the image; processing the visual embedding of the image using a mapping neural network to generate one or more language tokens for the image; generating a plurality of language tokens of the text prompt; processing the image and the language tokens of the text prompt using a language encoder to generate a language embedding for the query; and One or more of the candidate images are selected from a plurality of candidate images using the language embedding of the query.

2. The method of claim 1 , wherein processing the language tokens of the image and the text prompt using the language encoder to generate the language embedding of the query comprises: appending the one or more language tokens of the image to the language tokens of the text prompt to obtain a sequence of language tokens for the query; as well as The sequence of language tokens for the query is processed using the language encoder to generate the language embedding for the query.

3. The method of any one of claims 1-2, wherein using the language embedding of the query to select one or more of the candidate images from the plurality of candidate images comprises: For each candidate image in the plurality of candidate images: processing the candidate image using the visual encoder to generate a visual embedding of the candidate image; as well as One or more of the candidate images are selected from the plurality of candidate images based on (i) the visual embeddings of the candidate images and (ii) the language embedding of the query.

4. The method of claim 3 , wherein selecting one or more of the candidate images from the plurality of candidate images based on (i) the visual embeddings of the candidate images and (ii) the language embedding of the query comprises: For each candidate image in the plurality of candidate images: computing a similarity measure between: (i) the visual embedding of the candidate image, and (ii) the language embedding of the query; as well as determining a score for the candidate image based on the similarity measure; as well as One or more candidate images of the candidate images are selected from the plurality of candidate images based on the scores of the plurality of candidate images.

5. The method of claim 4 , wherein selecting one or more of the candidate images from the plurality of candidate images based on the scores of the plurality of candidate images comprises: ranking each of the plurality of candidate images according to the scores of the plurality of candidate images; as well as One or more candidate images having the highest scores among the candidate images are selected from the plurality of candidate images.

6. The method of any preceding claim, wherein the context described by the text prompt includes one or more of: (i) domain conversion, (ii) object and scene composition, or (iii) sentence specification.

7. A method according to any preceding claim, wherein the mapping neural network has been trained on training data comprising a plurality of training images each depicting a respective scene. The method of claim 7 , wherein the plurality of training images comprises three million or more training images.

9. The method of any one of claims 7-8, wherein the visual encoder and the language encoder are pre-trained neural networks and are kept frozen during the training of the mapping neural network.

10. The method of claim 9, wherein the visual encoder and the language encoder are pre-trained on 400 million or more image-caption pairs.

11. The method according to any one of claims 7 to 10, wherein training the mapping neural network on the training data comprises: For each training image in the plurality of training images: processing the training image using the visual encoder to generate a visual embedding of the training image; processing the visual embedding of the training image using the mapping neural network to generate one or more language tokens for the training image; providing a training text prompt for the training image; generating a plurality of language tokens of the training text prompt; as well as processing the language tokens of the training image and the training text prompt using the language encoder to generate a language embedding for the training image; determining a loss function that depends on the visual embedding and the language embedding of each training image; as well as A set of network parameters of the mapping neural network is updated according to the loss.

12. The method of claim 11 , wherein, for each training image in the plurality of training images, processing the training image and the language tokens of the training text prompt using the language encoder to generate the language embedding for the training image comprises: appending the one or more language tokens of the training image to the language tokens of the training text prompt to obtain a sequence of language tokens for the training image; as well as The sequence of language tokens for the training image is processed using the language encoder to generate the language embedding for the training image.

13. The method of any one of claims 11-12, wherein the training text prompt is the same for each training image in the plurality of training images.

14. The method of claim 13, wherein the training text prompt comprises: "A photo of".

15. The method of any one of claims 11-14, wherein the loss function depends on normalized versions of the visual embedding and the language embedding of the training images.

16. The method of any one of claims 11-15, wherein the loss function comprises one or more contrastive loss terms that, for each training image, characterize a similarity measure between: (i) the visual embedding of the training image, and (ii) the language embedding of the training image. The method of claim 16 , wherein the loss function comprises a symmetric contrastive loss term.

18. The method according to any one of claims 11 to 17, wherein updating the network parameters of the mapping neural network according to the loss comprises: determining values ​​of the network parameters that minimize the loss; as well as The network parameters of the mapping neural network are updated using the values ​​that minimize the loss.

19. A system comprising: one or more computers; as well as One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the corresponding method according to any one of claims 1-18.

20. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform the operations of the corresponding method of any one of claims 1-18.

Images