Multimodal image classifier using textual and visual embeddings

Abstract

Methods, systems, and apparatus for implementing a multimodal image classifier, including a computer program encoded on a computer storage medium. In one aspect, a method includes, for each image in a plurality of images: processing the image through a text generator model to obtain a set of phrases describing the content of the image, wherein each phrase is one or more words, processing the set of phrases through a text embedding model to obtain an embedding of a predicted text for the image, and processing the image using an image embedding model to obtain an embedding of image pixels of the image. Then, a multimodal image classifier is trained on the embedding of the predicted text for the image and the embedding of the image pixels of the image to produce a label of an output taxonomy as an output for classifying the image based on the image as input.

Description

[0001] Priority Claim

[0002] This application claims priority to U.S. patent application serial number 62 / 768,701, filed on November 16, 2018, the entire contents of which are incorporated herein by reference. Background Art

[0003] This specification relates to image processing, and more particularly to a multimodal image classifier.

[0004] Machine learning image classification models take image pixels as input and produce labels using a predefined classification system. Image classification is typically a supervised learning problem, which defines a set of target classes (objects to be identified in an image) and trains a model to recognize them using labeled training images. Such systems can achieve high prediction accuracy and generalizability on unseen data. These models may rely on pixel data and features derived from the pixel data (such as color histograms, texture, and shape) as training features. Summary of the Invention

[0005] This specification describes a system, implemented as a computer program on one or more computers in one or more locations, that predicts image classification using a model that derives features from embeddings of predicted text and embeddings of image pixels. The specification also describes a method for predicting image classification, performed by one or more data processing devices, and one or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for predicting image classification.

[0006] According to one aspect, a method is provided, comprising: processing each image of a plurality of images through a text generator model to obtain a set of phrases describing the content of the image, wherein each phrase is one or more words; processing the set of phrases through a text embedding model to obtain an embedding of predicted text for the image; and processing the image using an image embedding model to obtain an embedding of image pixels of the image; and training a multimodal image classifier on the embedding of the predicted text for the image and the embedding of the image pixels of the image to produce a label of an output classification method as an output to classify the image based on the image as input. Additional aspects may be provided in apparatuses, systems, and computer program products.

[0007] Certain embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. While textual information (e.g., text surrounding an image) can provide valuable orthogonal signals, machine learning models typically rely on pixel data as input. Furthermore, for a set of images, textual data may not be available. The novel features of the present disclosure include text embeddings for images generated from phrases generated by a text generator, without requiring surrounding text as input. A multimodal image classifier is then trained on the text embeddings and pixel embeddings of the images. Training using the text embeddings results in overall performance that is superior to the overall performance of a trained model without the text embeddings, resulting in significant improvements in the field of image classification.

[0008] Furthermore, the systems and methods described herein can provide language-agnostic and cross-lingual text embeddings, which can be achieved, for example, by training on query-image pairs.

[0009] Additionally or alternatively, the systems and methods described herein can allow for characterization of both individual local features and overall visual structure. For example, this can be achieved by training network weights on an input dataset, identifying a bottleneck layer, and extracting the output of that layer for any image.

[0010] The present disclosure may also provide a system framework (and associated methods) that requires only an input image to generate predictions based partially on text features. For example, by concatenating an N-dimensional text feature vector of an image with an M-dimensional visual feature vector, the systems and methods described herein allow for singular feature vectors representing two concepts.

[0011] The systems and methods described herein may also allow for the generation of a probability distribution across each possible predicted phrase. For example, a softmax layer may be used to generate a probability distribution across a large number of queries.

[0012] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a diagram of a multimodal image classification system.

[0014] Figure 2 is a flowchart of an example process for training a multimodal image classifier.

[0015] Like reference numbers and designations throughout the various drawings refer to like elements. DETAILED DESCRIPTION

[0016] The patent document describes a framework that includes: a text generator that generates text labels, called phrases, from image data; and a multimodal image classifier that takes text features from the text generator and visual features from image pixels and produces labels based on an output taxonomy. In some implementations, the text generator is trained using web-based query / image pairs to incorporate contextual information associated with each image. In some implementations, the output of the text generator may exceed the output taxonomy of the image classifier, for example, the text generator may generate phrases that are not included in the output taxonomy of the multimodal image classifier.

[0017] These and other features are described in more detail below.

[0018] Figure 1 is an illustration of a multimodal image classification system 100. The multimodal image classification system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0019] The multimodal image classification system 100 is configured to process images to generate text embeddings and image pixel embeddings for training the multimodal image classification model 110. The system 100 includes a text generator 120 and a text embedding neural network 122. The text generator 120 generates a phrase of one or more words describing the content of each image in the images 102. The text embedding neural network 122 processes the phrase generated by the text generator for the image and generates a text embedding.

[0020] In some implementations, the text generator 100 is a neural network trained on web-based query-image pairs. A query-image pair is an image paired with a corresponding query for which a user selects an image.

[0021] In some implementations, long-tail queries are included in the query-image pairs. This results in fine-grained and descriptive queries, e.g., [2016 lamborghini aventador white], [equestrian statue of bartolomeo colleoni], etc. The query-image pair space also covers multiple visual concepts, including landmarks, animals, man-made products, people, artwork, etc. This results in a technical improvement in accuracy relative to systems trained on image datasets with much smaller vocabulary sizes and / or less descriptive phrases (e.g., [golden retriever], such as "dog." in a limited image domain).

[0022] In some implementations, text generator 120 is a convolutional neural network with a classification loss. One example architecture is based on ResNet-101, followed by an embedding layer that projects the ResNet response into a low-dimensional vector, topped by a multi-way softmax. In one example implementation, the vector has M dimensions and is followed by a K million-way softmax. In some implementations, M is 64 and K is 40 million. However, other values ​​may also be used.

[0023] In this particular example, the text generator network 120 is characterized by its ability to predict a large number of queries (e.g., 40 million) from an input image. To accommodate the large output space, the Resnet output is reduced to a 64-dimensional bottleneck layer. The network 120 can thus be conceptualized as comprising three stages: a main training mechanism for learning weights and biases, a 64-dimensional feature layer representing the image embedding, and a softmax layer that produces a probability distribution across the 40 million queries.

[0024] In this example architecture, at inference time, a 64-dimensional embedding is generated for each input image, and the top N most likely queries and their associated similarity scores are extracted from the query embedding index. In some implementations, N = 3, but other values ​​of N can also be used. Because the query represents contextual information, this process identifies relevant text for any image without web dependencies. Furthermore, the embedding additionally generates a way to measure image-to-image similarity in the query space.

[0025] For large-scale training, specialized circuits on AI accelerators such as Tensor Processing Units (TPUs) and a sampled softmax loss algorithm are used. Other appropriate training pipelines can also be used, but the specific pipeline used may depend on the size of the training corpus.

[0026] In some implementations, after training, the query q calculated by the softmax function c The predicted probability is:

[0027]

[0028] where x is the image embedding (e.g., from the bottleneck layer), w c is the query q from the fully connected layer after the bottleneck layer, among the K million queries c The corresponding weights are , and d(·,·) is a distance function, such as a cosine distance function. The use of this function is optional, but other ways of predicting probabilities for queries can also be used.

[0029] To further increase the vocabulary of predicted phrases from the phrase generation model, in some implementations, a prototypical network approach can be employed to construct prototypes for a significantly larger number of queries. Increasing the vocabulary through this process is optional. The prototypical network is constructed by embedding a function f with a learnable parameter φ. φ :R D →R M Calculate the M-dimensional representation c of each class k ∈R M Or prototype. In some implementations, the prototype φ of each query c c is defined as the mean vector of embeddings of the images associated with the query:

[0030]

[0031] Among them, S c is the set of images associated with the query c. Prototypes can also be obtained using other methods. c Substituting the weight w in equation (1) c , to increase the output size from K to the larger output space of the phrase generation model. In some implementations, K is 40 million. Other ways of increasing the output space can also be used.

[0032] In some implementations, the prediction of the most relevant queries (phrases) for a given image can be done more efficiently by using a nearest neighbor process. Since many queries in the output space may not be relevant to a given image, there is no need to calculate their softmax probabilities to retrieve the top queries. Therefore, in some implementations, the system uses a top-k nearest neighbor search technique for phrase generation:

[0033]

[0034] The text generator 120 achieves several advantages, including the ability to distinguish very fine-grained concepts of an image, such as [2016 Lamborghini Aventador white] and [2010 Lamborghini Aventador white] compared to [white Lamborghini]. Such fine-grained concepts of an image are derived from a long tail of queries from query-image pairs. Additionally, because it is trained on query-image pairs, the resulting text embeddings are language-agnostic and cross-lingual.

[0035] The text embedding neural network 122 is an artificial neural network that is trained to map discrete inputs (e.g., feature vectors of phrases generated by the text generator 120) to continuous-valued outputs (e.g., vectors or matrices). The output of the text embedding neural network 122 has the following property: similar inputs are mapped to outputs that are close to each other in a multidimensional space. Therefore, the output of the text embedding neural network 122 can be described as a potential representation of the data input to the embedding neural network. In this specification, the potential representation of a set of data refers to a digital representation of the data generated internally by the network (e.g., as a vector or matrix).

[0036] In an example implementation, the text embedding network 122 is a Siamese network pre-trained on a dataset of query-query associations. A Siamese network is an artificial neural network that uses the same weights when working together on two different input vectors to compute comparable output vectors. To improve the quality of the embedding, an attention mechanism can optionally be used. The resulting embedding has the property that related queries are closely placed together in the output embedding space.

[0037] The embedding model is then used to embed the top N text results obtained from the text generator 120. In some implementations, the text embeddings are obtained via a bag-of-words model of unigrams and bigrams. The embeddings (or text predictions) of the top N queries are then averaged to obtain an M-dimensional text feature vector. In some implementations, N is 3 and M is 200. However, other values ​​may also be used.

[0038] During this processing phase, the system generates embeddings from a network trained on textual signals (query-query associations) that represent images for which the system was initially given only pixels as input. Thus, the generated independent text embeddings for text queries bridge the gap between visual and textual modalities.

[0039] The pixel embedding neural network 124 generates an embedding having properties similar to those of the text embedding neural network 122, i.e., the output of the pixel embedding neural network 124 has the property that similar inputs are mapped to outputs that are close to each other in a multidimensional space. In some implementations, the embedding of image features is obtained from the bottleneck layer of a pre-trained convolutional network. This involves training the network weights on the input dataset, identifying the bottleneck layer (usually the layer before the output), and extracting the output of the bottleneck layer for any image. In this way, features that characterize individual local characteristics and overall visual structure are obtained.

[0040] The multimodal image classifier 110 receives text embeddings from a text embedding neural network 122 and pixel embeddings from a pixel embedding neural network 124. A classifier is trained on these embeddings to produce as output a label of an output taxonomy to classify an image based on the image as input. Pixel and text embeddings are obtained for any input image and then combined to generate a representation of the image that incorporates both types of features. In some implementations, the process for combining these features involves concatenating a 200-dimensional text feature vector with a 1024-dimensional visual feature vector into a singular feature vector representing the two concepts. The resulting vector is then input into a two-stage fully connected neural network where the output softmax layer predicts the probability of being associated with a class according to a specified taxonomy. However, other suitable training procedures may also be used.

[0041] Advantageously, while extracting orthogonal text features to supplement pixel information and generate context-aware predictions, system 100 only requires images to generate label-wise results. Since text generation using text generator 120 enables identification of relevant text for any input image, the system framework only requires the input image to generate predictions based partially on text features. This is another technical advancement in the field of image classification.

[0042] Figure 2 is a flow chart of an example process 200 for training a multimodal image classifier. Process 200 can be implemented in one or more computers in one or more locations.

[0043] For each image in the collection of images (202), process 200 obtains a text embedding for the image (204) and obtains a pixel embedding for the image (206). For example, as described above, the text embedding for each image can be obtained by processing the image by text generator 120 and processing the output phrase (text or query) by text embedding neural network 122. The pixel embedding for each image can be obtained by processing the image by pixel embedding neural network 124.

[0044] The process 200 then trains a multimodal image classifier (208) using the text embeddings and the pixel embeddings. As described above, the classifier is trained on these embeddings to produce as output a label for an output taxonomy to classify the image based on the image as input. Once both the visual and text embeddings are generated, the multimodal image classifier 110 fuses the two signals to produce a final image classification. As described above, in some implementations, the input is a concatenation of features from a 1024-dimensional visual embedding and a 200-dimensional text embedding. The resulting vector is input to a multi-stage fully connected neural network, where the output softmax layer predicts the probability of being associated with a class based on the target taxonomy.

[0045] This specification uses the term "configured" in conjunction with system and computer program components. When referring to a system of one or more computers configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof that, when operated, causes the system to perform the operation or action. When referring to one or more computer programs configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0046] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of digital electronic circuitry, tangibly embodied computer software or firmware, or computer hardware. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium, to be executed by a data processing apparatus or to control the operation of the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus.

[0047] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be or further include special-purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program, for example, code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0048] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script, or code, may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages; and the computer program may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store portions of one or more modules, subroutines, or code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0049] Throughout this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed and run on the same computer or on multiple computers.

[0050] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry (e.g., an FPGA or ASIC) or by a combination of special purpose logic circuitry and one or more programmed computers.

[0051] The computer suitable for executing a computer program can be based on a general or special microprocessor or both, or based on a central processing unit of any other type. Typically, the central processing unit will receive instructions and data from a read-only memory or random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by or incorporated into a dedicated logic circuit. Typically, the computer will also include or be operably coupled to receive data from one or more large-capacity storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or to transfer data to one or more large-capacity storage devices or to receive and transfer data. However, the computer does not necessarily have such devices. In addition, the computer can be embedded in another device, for example, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0052] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0053] To provide for user interaction, embodiments of the subject matter described herein can be implemented on a computer having a display device, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user, and a keyboard and pointing device; the user can provide input to the computer via the keyboard and pointing device, such as a mouse or trackball. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, voice, or tactile input. Furthermore, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. Furthermore, a computer can interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and in turn receiving a response message from the user.

[0054] The data processing apparatus for implementing a machine learning model may also include, for example, dedicated hardware accelerator units for processing both the general purpose and computationally intensive parts, i.e., inference, workloads, of machine learning training or production.

[0055] Machine learning models can be implemented and deployed using a machine learning framework such as the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework.

[0056] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, web browser, or app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0057] A computing system may include a client and a server. The client and server are typically remote from each other and typically interact via a communication network. The relationship between client and server arises through computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, the server sends data, such as an HTML page, to a user device, for example, for the purpose of displaying data to a user interacting with the device as a client and receiving user input from the user. Data generated at the user device, such as the results of the user interaction, may be received from the device on the server.

[0058] Although this specification contains many specific implementation details, these details should not be interpreted as limiting the scope of any invention or the scope of what is claimed, but rather as descriptions of features that may be specific to a particular embodiment of a particular invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as functioning in certain combinations, or even initially claimed as such, in some cases one or more features from the combination may be removed from the claimed combination, and the claimed combination may involve a sub-combination or a variant of a sub-combination.

[0059] Similarly, although operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated into a single software product or packaged into multiple software products.

[0060] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

Claims

1. A method performed by one or more data processing devices, the method comprising: For each image in multiple images: Processing pixels of the image through a text generator model to obtain a set of phrases describing the content of the image, wherein each phrase is one or more words, wherein processing pixels of the image through the text generator model comprises: receiving a plurality of outputs from a convolutional neural network of a text generator model, each output of the plurality of outputs representing a possible query associated with the image; projecting each of the plurality of outputs into a low-dimensional vector, the low-dimensional vector representing an image embedding associated with the corresponding output; generating a probability distribution over the plurality of outputs based on the low dimensional vector associated with each of the plurality of outputs, the probability distribution indicating an image-to-image similarity score for each of the plurality of outputs; and Select the top N most likely queries and associated similarity scores as the set of phrases to be obtained; Processing the collection of phrases through a text embedding model to obtain a predicted text embedding for the image; and Processing the image using an image embedding model to obtain embeddings of image pixels of the image; A multimodal image classifier is trained on an embedding of predicted text for an image and an embedding of image pixels of the image to produce as an output a label of an output taxonomy to classify the image based on the image as input.

2. The method according to claim 1, wherein: The text generator model is a text query based model trained on text query-image pairs.

3. The method according to claim 1, wherein: Processing the image using the image embedding model to obtain embeddings of image pixels of the image includes obtaining image features from the final fully connected layer of the pre-trained convolutional network.

4. The method according to claim 1, wherein: Training a multimodal image classifier involves: concatenate the N-dimensional text feature vector and the M-dimensional visual feature vector into a singular feature vector; and Provide the singular feature vector as input to the multimodal classifier.

5. The method according to claim 1, wherein: Selecting the top N most likely queries and associated similarity scores as the set of phrases to be obtained includes selecting the top N most likely queries using a nearest neighbor process.

6. A system for predicting image classification, comprising: Data processing equipment; A memory that communicates data with the data processing device and stores instructions for causing the data processing device to perform operations, the operations comprising: For each image in multiple images: Processing pixels of the image through a text generator model to obtain a set of phrases describing the content of the image, wherein each phrase is one or more words, wherein processing pixels of the image through the text generator model comprises: receiving a plurality of outputs from a convolutional neural network of a text generator model, each output of the plurality of outputs representing a possible query associated with the image; projecting each of the plurality of outputs into a low-dimensional vector, the low-dimensional vector representing an image embedding associated with the corresponding output; generating a probability distribution over the plurality of outputs based on the low dimensional vector associated with each of the plurality of outputs, the probability distribution indicating an image-to-image similarity score for each of the plurality of outputs; and Select the top N most likely queries and associated similarity scores as the set of phrases to be obtained; Processing the collection of phrases through a text embedding model to obtain a predicted text embedding for the image; and Processing the image using an image embedding model to obtain embeddings of image pixels of the image; A multimodal image classifier is trained on an embedding of predicted text for an image and an embedding of image pixels of the image to produce as an output a label of an output taxonomy to classify the image based on the image as input.

7. The system according to claim 6, wherein: The text generator model is a text query based model trained on text query-image pairs.

8. The system according to claim 6, wherein: Processing the image using the image embedding model to obtain embeddings of image pixels of the image includes obtaining image features from the final fully connected layer of the pre-trained convolutional network.

9. The system according to claim 6, wherein: Training a multimodal image classifier involves: concatenate the N-dimensional text feature vector and the M-dimensional visual feature vector into a singular feature vector; and Provide the singular feature vector as input to the multimodal classifier.

10. The system according to claim 6, wherein: Selecting the top N most likely queries and associated similarity scores as the set of phrases to be obtained includes selecting the top N most likely queries using a nearest neighbor process.

11. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: For each image in multiple images: Processing pixels of the image through a text generator model to obtain a set of phrases describing the content of the image, wherein each phrase is one or more words wherein processing pixels of the image through a text generator model comprises: receiving a plurality of outputs from a convolutional neural network of a text generator model, each output of the plurality of outputs representing a possible query associated with the image; projecting each of the plurality of outputs into a low-dimensional vector, the low-dimensional vector representing an image embedding associated with the corresponding output; generating a probability distribution over the plurality of outputs based on the low dimensional vector associated with each of the plurality of outputs, the probability distribution indicating an image-to-image similarity score for each of the plurality of outputs; and Select the top N most likely queries and associated similarity scores as the set of phrases to be obtained; Processing the collection of phrases through a text embedding model to obtain a predicted text embedding for the image; and Processing the image using an image embedding model to obtain embeddings of image pixels of the image; A multimodal image classifier is trained on an embedding of predicted text for an image and an embedding of image pixels of the image to produce as an output a label of an output taxonomy to classify the image based on the image as input.

12. The one or more non-transitory computer storage media of claim 11, wherein: The text generator model is a text query based model trained on text query-image pairs.

13. The one or more non-transitory computer storage media of claim 11, wherein: Processing the image using the image embedding model to obtain embeddings of image pixels of the image includes: obtaining image features from a final fully connected layer of a pre-trained convolutional network.

14. The one or more non-transitory computer storage media of claim 11, wherein: Training a multimodal image classifier involves: concatenate the N-dimensional text feature vector and the M-dimensional visual feature vector into a singular feature vector; and Provide the singular feature vector as input to the multimodal classifier.

Images