Training image embedding model and text embedding model

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an image embedding model. In one aspect, a method includes obtaining training data including a plurality of training examples, where each training example includes: an image pair including a first image and a second image; and selection data indicating one or more of (i) a common click rate of the image pair, and (ii) a similar image click rate of the image pair; and training an image embedding model having a plurality of image embedding model parameters using the training data.

Description

[0001] This application is a divisional application of the patent application for invention with the application date of October 24, 2019, application number 201980029223.1, and invention title "Training Image Embedding Model and Text Embedding Model". Technical Field

[0002] This specification relates to processing data using machine learning models. Background Art

[0003] Machine learning models receive inputs and produce outputs based on the received inputs, such as prediction outputs. Some machine learning models are parametric models and produce outputs based on the received inputs and the parameter values of the models.

[0004] Some machine learning models are deep models that use multiple layers of models to produce outputs for the received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, and each hidden layer applies a non-linear transformation to the received input to produce an output. Summary of the Invention

[0005] This specification describes a training system implemented as a computer program on one or more computers in one or more locations, and the training system uses training data derived from the historical query logs of a search system to train an image embedding model and a text embedding model.

[0006] According to a first aspect, there is provided a method performed by one or more data processing devices, the method comprising: obtaining training data comprising a plurality of training examples, wherein each training example comprises: an image pair comprising a first image and a second image; and selection data indicating one or more of: (i) a co-click-through rate of the image pair, and (ii) a similar-image click-through rate of the image pair, wherein: the co-click-through rate of the image pair characterizes the frequency with which a user selects both the first image and the second image in response to both the first image and the second image being simultaneously identified by search results of a search query; and the similar-image click-through rate of the image pair characterizes the frequency with which a user selects the first image or the second image in response to the first image or the second image being identified by search results of a search query that respectively comprise the second image or the first image; and using the training data to train an image embedding model having a plurality of image embedding model parameters, wherein, for each of the plurality of training examples, the training comprises: processing the first image of the training example using the image embedding model to generate an embedding of the first image; processing the second image of the training example using the image embedding model to generate an embedding of the second image; determining a similarity metric between the embedding of the first image and the embedding of the second image; and adjusting the image embedding model parameters at least in part based on: (i) the similarity metric between the embedding of the first image and the embedding of the second image, and (ii) the selection data of the training example.

[0007] In some embodiments, the training data is generated using a historical query log of a web search system.

[0008] In some embodiments, the co-click-through rate of each training example indicates a fraction of the number of times a user selects both the first image and the second image of the training example in response to both the first image and the second image of the training example being simultaneously identified by search results of a search query.

[0009] In some embodiments, the similar-image click-through rate of each training example indicates a fraction of the number of times a user selects the first image or the second image in response to the first image or the second image being identified by search results of a search query that respectively comprise the second image or the first image.

[0010] In some embodiments, the image embedding model comprises a convolutional neural network.

[0011] In some embodiments, adjusting the image embedding model parameters comprises: determining a gradient of a loss function that depends on: (i) the similarity metric between the embedding of the first image and the embedding of the second image, and (ii) the selection data of the training example; and using the gradient to adjust the image embedding model parameters.

[0012] In some embodiments, the loss function comprises a multiplicative scaling factor determined based on the selection data of the training example.

[0013] In some embodiments, the multiplicative scaling factor is determined as a linear combination of the co-click-through rate of the training examples and the click-through rate of the similar images.

[0014] In some embodiments, determining a similarity metric between an embedding of a first image and an embedding of a second image includes: determining an Euclidean distance between the embedding of the first image and the embedding of the second image.

[0015] In some embodiments, the method may further include receiving a query image and processing the query image using a trained image embedding model obtained by a method according to any aspect or embodiment described herein. As an example, processing the query image may include determining corresponding relevance scores between the query image and a plurality of images in a search index.

[0016] According to another aspect, there is provided a method including: receiving a query image and processing the query image using a trained image embedding model obtained by a method according to any aspect or embodiment described herein. As an example, processing the query image may include determining corresponding relevance scores between the query image and a plurality of images in a search index.

[0017] According to a second aspect, there is provided a system including: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations including the operations of the previously described method.

[0018] According to a third aspect, there is provided one or more computer storage media storing instructions (which may or may not be non-transitory storage media) that, when executed by one or more computers, cause the one or more computers to perform operations including the operations of the previously described method.

[0019] Certain embodiments of the subject matter described in this specification may be implemented to realize one or more of the following advantages.

[0020] The training system described in this specification can generate a large amount of training data (e.g., tens or hundreds of millions of training examples) for training an image embedding model and a text embedding model by processing data from the historical query logs of a search system. The large amount of training data that can be effectively derived from the historical query logs (e.g., of a web search system) enables the training system to train highly effective image embedding models and text embedding models. Such scalability (e.g., for training examples that may potentially include hundreds of millions of search queries) is a technological improvement in the field of model training. For example, such scalability enables the training of an image embedding model that generates image embeddings that implicitly represent concepts over a large range (e.g., food, scenes, landmarks, man-made products, etc.). In contrast, some conventional image embedding models generate image embeddings that only implicitly represent concepts within a small range (e.g., only food or only landmarks).

[0021] The training system can process the historical query logs to generate "query-image" training examples that associate text search queries with relevant images (e.g., images that users frequently select when identifying the search results of their text search queries). Specifically, the query-image training examples can associate highly specific text search queries (e.g., "red 2014 Ford Mustang") with relevant images (e.g., images depicting the object specified by the text search query). By jointly training the image embedding model and the text embedding model using the query-image training examples derived from the historical query logs, the training system can enable the image embedding model to generate image embeddings that implicitly represent highly specific concepts. For example, the trained image embedding model can process an image to generate an image embedding that implicitly represents the color, style, and model of the car depicted in the image. This is a technological improvement in the field of model training. In contrast, for example, using training examples that associate images with general labels (e.g., "car") as in some conventional training datasets to train the image embedding model and the text embedding model may result in the image embedding model generating relatively uninformative embeddings.

[0022] In some embodiments, a training system can generate query-image training examples that include search queries expressed in a large number of different natural languages (e.g., English, French, German, etc.). By jointly training an image embedding model and a text embedding model using the multilingual query-image training examples, the training system can train the text embedding model to generate informative text embeddings independent of the text language. For example, the training system can train the text embedding model to generate similar embeddings of text based on the similarity of images associated with search queries including the text "young Queen Elizabeth" (English) and "jeune reine Elizabeth" (French). This is another technical improvement in the field of model training, where the training system can train the image embedding model and the text embedding model based on selected data that characterizes, for example, the frequency with which two images are "co-clicked" or the frequency with which a given image is selected when it is identified by the search results of a search query (i.e., via "image-image" training examples). The selected data can be determined by aggregating user-derived signals (e.g., clicks) from millions of users and enables the training system to train the image embedding model and the text embedding model more effectively.

[0023] Generating training data using conventional methods lacks many of the advantages of generating training data by processing the historical query logs of a search system. For example, manually generating training data (e.g., manually specifying text labels for images by a person) is time-consuming and difficult, and generally only a relatively small amount of training data can be generated in this way. As another example, generating training data by associating images and captions drawn from a social network (or other sources) may produce less and lower-quality training data compared to generating training data from historical query logs, e.g., because the captions may not accurately represent the content of the images. Generating more and / or higher-quality training data and using that training data to train a machine learning model will improve the accuracy of the trainable machine learning model (and thus provide a machine learning model that makes fewer errors than a machine learning model trained on training data generated using conventional methods). Additionally or alternatively, by reducing the time required to obtain labeled training data, a larger training dataset can be established within a given time period, and this will also improve the accuracy of the trainable machine learning model. Furthermore, improving the accuracy of the trainable machine learning model can make it possible to use a simpler machine learning model (e.g., with fewer hidden layers), and this can result in a reduction in the computational resources required to implement the machine learning model. In a complementary way, it may be necessary to store the data for a shorter period of time, and thus generally less memory storage may be required.

[0024] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Shows an example image embedding model.

[0026] Figure 2 Shows an example text embedding model.

[0027] Figure 3 Shows an example training system for training an image embedding model and a text embedding model using training data derived from the historical query log of a search system.

[0028] Figure 4 Shows an example search result page provided by a search system, which includes image search results for a search query that includes a sequence of one or more terms.

[0029] Figure 5 Shows an example search result page provided by a search system, which includes image search results for a search query that includes an image.

[0030] Figure 6 Shows an example process for jointly training an image embedding model and a text embedding model using query-image training examples.

[0031] Figure 7A Shows an example process for training an image embedding model using image-image training examples.

[0032] Figure 7B Shows an example process for jointly training an image embedding model and a text embedding model using query-image training examples and image-image training examples.

[0033] Figure 8 Is a flowchart of an example process for jointly training an image embedding model and a text embedding model using query-image training examples derived from the historical query log of a search system.

[0034] Figure 9 Is a flowchart of an example process for training an image embedding model using image-image training examples derived from the historical query log of a search system.

[0035] Figure 10 Shows an example of a portion of a graphical representation of query-image training examples and image-image training examples.

[0036] Figure 11 Shows an example search system.

[0037] Figure 12 Shows an example computer system.

[0038] In the various figures, the same reference numerals and symbols indicate the same elements. DETAILED DESCRIPTION

[0039] This specification describes a training system for training an image embedding model and a text embedding model using training data derived from a historical query log of a search system. The training data derived from the historical query log can include query-image training examples, image-image training examples, or both.

[0040] Query-image training examples include: (i) a text search query, (ii) an image, and (iii) selection data that characterizes the frequency with which a user selects the image in response to the image being identified by the search results of the text search query. If the selection data indicates that the user frequently selects the image when the image is identified by the search results of the search query, the training system can jointly train the image embedding model and the text embedding model to generate similar embeddings for the text search query and the image.

[0041] Image-image training examples include an image pair (including a first image and a second image) and selection data indicating: (i) the co-clickthrough rate of the image pair, (ii) the similar-image clickthrough rate of the image pair, or (iii) both. The co-clickthrough rate of the image pair characterizes the frequency with which a user selects both the first image and the second image in response to both the first image and the second image being simultaneously identified by the search results of a search query. The similar-image clickthrough rate of the image pair characterizes the frequency with which a user selects the first image in response to the first image being identified by the search results of a search query that includes the second image, and vice versa. The training system can train the image embedding model to generate similar embeddings for the first image and the second image when the co-clickthrough rate, the similar-image clickthrough rate, or both of the first image and the second image indicate that they are related.

[0042] In some embodiments, the training system can use query-image training examples and image-image training examples derived from the historical query log to jointly train the image embedding model and the query embedding model with a graph-regularized loss function. Specifically, the query-image training examples and the image-image training examples can be represented as a graph structure, and the training system can jointly train the image embedding model and the query embedding model based on the graph structure using the loss function.

[0043] These features and other features are described in more detail below.

[0044] Figure 1An example image embedding model 100 is shown. The image embedding model 100 is configured to process an image 102 according to current values of a set of image embedding model parameters to generate an embedding 104 of the image 102. The embedding 104 is a representation of the image 102 as an ordered set of numerical values (e.g., as a vector or a matrix). As will be described in more detail below, machine learning techniques can be used to train the image embedding model 100 to generate the embedding 104 of the image 102, which implicitly represents the semantic content of the image 102 (e.g., the object depicted by the image 102).

[0045] The image embedding model 100 can be configured to process the image 102 represented in any suitable format. For example, the image embedding model 100 can be configured to process an image 102 represented in a red-green-blue (RGB) color format (i.e., a format that represents an image by associating respective red values, green values, and blue values with each pixel of the image). As another example, the image embedding model 100 can be configured to process a feature representation of the image 102, such as a histogram of oriented gradients (HOG) representation, a scale-invariant feature transform (SIFT) representation, or a speeded-up robust features (SURF) representation of the image 102. Other feature representations can also be used for training.

[0046] The image embedding model 100 can be a neural network model implemented by a computer program on one or more computers in one or more locations. For example, the image embedding model 100 can be a convolutional neural network having an architecture derived from an Inception neural network or a ResNet neural network.

[0047] Figure 2 An example text embedding model 200 is shown. The text embedding model 200 is configured to process a representation of a sequence of one or more words in natural language (i.e., text 202) according to current values of a set of text embedding model parameters to generate an embedding 204 of the text 202. The embedding 204 is a representation of the text 202 as an ordered set of numerical values (e.g., as a vector or a matrix). As will be described in more detail below, machine learning techniques can be used to train the text embedding model 200 to generate the embedding 204 of the text 202, which implicitly represents the semantic content of the text 202 (e.g., the object described by the text 202).

[0048] The text embedding model 200 can be configured to process the text 202 represented in any suitable format. For example, the text embedding model 200 can be configured to process text 202 represented as a series of "one-hot" vectors, where each one-hot vector represents a respective character (or word) of the text 202. As another example, the text embedding model 200 can be configured to process text 202 represented by the output of a Word2vec model.

[0049] The text embedding model 200 can be a neural network model implemented by a computer program on one or more computers in one or more locations. For example, the text embedding model 200 can be a convolutional neural network having an architecture that includes multiple one-dimensional (1D) convolutional layers. As another example, the text embedding model 200 can be a lookup-based mapping from text 202 to embedding 204. As another example, the text embedding model 200 can be a series of fully connected layers configured to process n-gram text tokens. As another example, the text embedding model can be a recurrent neural network model (e.g., LSTM) configured to sequentially process representations of characters of text 202.

[0050] Figure 3 An example training system 300 is shown for training the image embedding model 100 and the text embedding model 200 using training data 302 derived from the historical query log 304 of the search system 306. The training system 300 trains the image embedding model 100 and the text embedding model 200 by determining values for the image embedding model parameters 308 and the text embedding model parameters 310. For example, when the image embedding model 100 and the text embedding model 200 are implemented as respective neural networks, the training system 300 can iteratively adjust the parameters of the neural networks using the gradients of a loss function, as will be described in more detail below. The training system 300 can be implemented by a computer program on one or more computers in one or more locations. In some embodiments, the training system 300 uses one or more tensor processing units (TPUs - application-specific integrated circuits (ASICs) designed for machine learning) during the training of the image embedding model 100 and the text embedding model 200.

[0051] The search system 306 can be any system configured to perform an image search by processing a search query that includes text, images, or both to generate search results that identify images in response to the search query. Example search systems will be described in more detail with reference to Figure 11 and will be described in more detail.

[0052] The historical query log 304 of the search system 306 indexes a large number (e.g., millions) of search queries previously processed by the search system 306. Specifically, the historical query log 304 can index search queries by maintaining data including: (i) the search query, and (ii) data specifying one or more search results selected by the user of the device that sent the search query. A user can "select" a search result by expressing an interest in the search result via any kind of interaction with the search result. For example, a user can select a search result by clicking on a hypertext link included in the search result or by hovering the cursor over the search result for a predetermined period of time to generate a request for an electronic document (e.g., an image) identified by the search result.

[0053] The training system 300 can process data from the historical query log 304 to generate query-image training examples and image-image training examples for training the image embedding model 100 and the text embedding model 200.

[0054] Query-image training examples include: (i) a text search query, (ii) an image, and (iii) selection data characterizing the frequency with which the user selects the image in response to the image being identified by the search results of the text search query. If the selection data indicates that the user frequently selects the image when the image is identified by the search results of the search query, the training system 300 can jointly train the image embedding model 100 and the text embedding model 200 to generate similar embeddings of the search query and the image. Any suitable numerical similarity metric (e.g., Euclidean distance if the image embedding model 100 and the text embedding model 200 are configured to generate embeddings of the same dimension) can be used to determine the similarity between the embeddings.

[0055] Image-image training examples include an image pair (which includes a first image and a second image) and selection data indicating: (i) the co-clickthrough rate of the image pair, (ii) the similar-image clickthrough rate of the image pair, or (iii) both. The co-clickthrough rate of the image pair characterizes the frequency with which the user selects both the first image and the second image in response to both the first image and the second image being simultaneously identified by the search results of a search query. The similar-image clickthrough rate of the image pair characterizes the frequency with which the user selects the first image in response to the first image being identified by the search results of a search query that includes the second image, and vice versa. The training system 300 can train the image embedding model 100 to generate similar embeddings of the first image and the second image when the co-clickthrough rate, the similar-image clickthrough rate, or both of the first image and the second image indicates that they are related.

[0056] Figure 4Illustrates an example search result page 400 provided by a search system 306. The example search result page 400 includes image search results for a search query that includes a sequence of one or more words. Specifically, the search result page 400 displays search results 402, 404, and 406 for the search query 408: "red Ford Mustang".

[0057] Figure 5 Illustrates an example search result page 500 provided by a search system 306. The example search result page 500 includes image search results for a search query that includes an image. Specifically, the search result page 500 displays search results 502, 504, and 506 for the search query 508, which includes an image depicting a truck. In response to receiving a search query that includes a query image, the search system 306 can be configured to provide search results that identify images similar to the query image. In this example, each of the search results 502, 504, and 506 identifies an image similar to the query image.

[0058] If a user selects search results that respectively identify a first image and a second image from the same set of search results, the user is said to "co-click" the first image and the second image. For example, the user can co-click the image identified by search result 402 and the image identified by search result 404 by selecting two search results (e.g., one after the other) on the search result page 400. As another example, the user can co-click the image identified by search result 504 and the image identified by search result 506 by selecting two search results (e.g., one after the other) on the search result page 500. If a user selects three or more search results from the same set of search results, the images identified by each pair of the selected search results can be considered co-clicked. For example, if a user selects search results A, B, and C from the same set of search results, the pairs of images identified by search results {A,B}, {A,C}, and {B,C} can all be considered co-clicked.

[0059] Figure 6 Illustrates an example process for jointly training an image embedding model 100 and a text embedding model 200 using a query-image training example 600. The query-image training example 600 includes a search query 602 and an image 604, and the search query 602 includes a sequence of one or more words. A training system 300 uses the image embedding model 100 and processes the image 604 according to the current values of the image embedding model parameters 308 to generate an embedding 606 of the image 604. The training system 300 uses the text embedding model 200 and processes the search query 602 according to the current values of the text embedding model parameters 310 to generate an embedding 608 of the search query 602.

[0060] The training system 300 determines a similarity metric 610 between the embedding 606 of the image 604 and the embedding 608 of the search query 602, and determines a model parameter adjustment 612 based on the similarity metric 610. Thereafter, the training system 300 uses the model parameter adjustment 612 to adjust the values of the image embedding model parameters 308 and the text embedding model parameters 310. In some embodiments, when determining the model parameter adjustment 612, the training system 300 uses selection data that characterizes the frequency with which a user selects the image 604 in response to the image 604 being identified by the search results of the search query 602. An example process for jointly training an image embedding model and a text embedding model using query-image training examples is described in more detail with reference to Figure 8 is described in more detail.

[0061] Figure 7A An example process for training the image embedding model 100 using an image-image training example 700 including a first image 702 and a second image 704 is shown. The training system 300 uses the image embedding model 100 and processes the first image 702 according to the current value of the image embedding model parameters 308 to generate an embedding 706 of the first image 702. Similarly, the training system 300 uses the image embedding model 100 and processes the second image 704 according to the current value of the image embedding model parameters 308 to generate an embedding 708 of the second image 704.

[0062] The training system 300 determines a similarity metric 710 between the embedding 706 of the first image 702 and the embedding 708 of the second image 704, and determines a model parameter adjustment 712 based on the similarity metric 710. In some embodiments, when determining the model parameter adjustment 712, the training system 300 uses selection data that characterizes the co-click-through rate, similar image click-through rate, or both of the first image 702 and the second image 704. An example process for training an image embedding model using an image-image training example is described in more detail with reference to Figure 9 is described in more detail.

[0063] Figure 7B An example process for jointly training an image embedding model and a text embedding model using query-image training examples and image-image training examples is shown. Specifically, in each of multiple training iterations, one or more query-image training examples 600 and one or more image-image training examples 700 can be processed by the image embedding model to generate corresponding embeddings (as described with reference to Figure 6 and Figure 7A ). The training system 300 can be based on the query-image training examples 600 (as described with reference to Figure 6 ), and the image-image training examples (as described with reference to Figure 7AAccordingly, the corresponding model parameter adjustment 714 is determined. Thereafter, the training system 300 may use the model parameter adjustment 714 to adjust the current values of the image embedding model parameters 308 and the text embedding model parameters 310. The model parameter adjustment determined based on the query-image training examples may be weighted more heavily or less heavily (e.g., using a gradient scaling factor) compared to the model parameter adjustment determined based on the image-image training examples. The weight applied to the model parameter adjustment may be an adjustable system hyperparameter.

[0064] Figure 8 FIG. 8 is a flow diagram of an example process 800 for jointly training an image embedding model and a text embedding model using query-image training examples derived from a historical query log of a search system. For convenience, process 800 will be described as being performed by a system of one or more computers located at one or more locations. For example, a training system (e.g., Figure 3 training system 300) programmed appropriately according to this specification may perform process 800.

[0065] The system processes data from the historical query log of the search system to generate a candidate set of query-image training examples (802). Each query-image training example includes: (i) a search query including a sequence of one or more words, (ii) an image, and (iii) selection data characterizing the frequency with which a user selects the image in response to the image being identified by the search results of the search query. The selection data may indicate a score for the number of times the image is selected in response to the image being identified by the search results of the search query.

[0066] The system selects query-image training examples (804) for jointly training an image embedding model and a text embedding model from a candidate set of training examples, at least in part based on selected data of the training examples. For example, if the image of a particular training example is most frequently selected by a user in response to the image being identified in search results of a search query of the particular query-image training example, the system may select the particular query-image training example. As another example, if the image of a particular query-image training example is among the top N images most frequently selected by the user after being identified in search results of a search query of the particular query-image training example, the system may select the particular query-image training example. The system may use any one of a variety of other suitable criteria when selecting query-image training examples for jointly training an image embedding model and a text embedding model. For example, the system may limit the number of selected query-image training examples that include a search query specifying the name of a particular person and a corresponding image depicting the particular person. In this example, since the appearance of the same person may vary significantly between images (e.g., due to the person wearing different clothes, shoes, glasses, etc.), including a large number of query-image training examples corresponding to a particular person may reduce the effectiveness of the training process.

[0067] Steps 806-812 describe an example process that may be performed for each selected query-image training example to jointly train an image embedding model and a text embedding model. For convenience, steps 806-812 describe the steps that may be performed for a given query-image training example. More generally, any suitable method may be used to jointly train an image embedding model and a text embedding model. For example, a stochastic gradient descent method may be used to jointly train an image embedding model and a text embedding model, where steps 806-812 are iteratively repeated for a "batch" (i.e., set) of query-image training examples. In this example, when a training termination criterion is met, the system may determine that training is complete. For example, the training termination criterion may be that a predetermined number of iterations of steps 806-812 have been performed. As another example, the training termination criterion may be that the values of the parameters of the image embedding model and the text embedding model change by less than a predetermined threshold between iterations of steps 806-812.

[0068] The system uses the image embedding model and processes the image of a given query-image training example according to the current values of the parameters of the image embedding model to generate an embedding of the image (806). For example, if the image embedding model is a neural network model, the system uses a series of neural network layers defined by the architecture of the neural network model to process the image.

[0069] The system uses a text embedding model and processes a representation of a search query of a given query-image training example according to current values of text embedding model parameters to generate an embedding of the search query (808). For example, if the text embedding model is a neural network model, the system uses a series of neural network layers defined by the architecture of the neural network model to process the representation of the search query.

[0070] The system determines a similarity metric between an embedding of the image of the given query-image training example and the embedding of the search query (810). For example, the embedding of the image and the embedding of the search query can have the same dimension, and the system can determine the similarity metric by determining the Euclidean distance or cosine similarity metric between the corresponding embeddings.

[0071] The system adjusts the image embedding model parameters and the text embedding model parameters at least in part based on the similarity metric between the embedding of the image of the given query-image training example and the embedding of the search query (812). For example, when the image embedding model and the text embedding model are respective neural network models, the system can determine the gradient of a loss function and use the gradient of the loss function to adjust the image embedding model parameters and the text embedding model parameters. The system can use any suitable method (e.g., backpropagation) to determine the gradient of the loss function. The loss function can be any suitable loss function that depends on the similarity metric between the embedding of the image of the given query-image training example and the embedding of the search query. Here are some examples.

[0072] In some embodiments, the loss function can be a classification loss function. In these embodiments, the search query of the given query-image training example is regarded as identifying a "positive" label for the image of the given query-image training example. The search queries of other query-image training examples are regarded as identifying the corresponding "negative" labels for the image of the given query-image training example. More specifically, the system can determine the similarity metric between the embedding of the image of the given query-image training example and the embedding of the search query of the given query-image training example as a "positive" score. The system can determine the corresponding "negative" score for each other training example as the similarity metric between the embedding of the image of the given query-image training example and the embedding of the search query of that other training example. The system can process the positive score and the negative scores using a soft-max (or sampled soft-max) function and provide the output of the soft-max (or sampled soft-max) function to a cross-entropy loss function (or any other suitable classification loss function).

[0073] In some embodiments, the loss function can be a triplet loss function. In these embodiments, the system can determine the embedding of the image of a given query-image training example as the "anchor", the embedding of the search query of the given query-image training example as the "positive", and the embedding of the search query of another query-image training example as the "negative".

[0074] Optionally, the loss function can depend on the selected data of the given query-image training example, where the selected data characterizes the frequency with which a user selects the image in response to the search results of the search query of the given query-image training example identifying the image. For example, the loss function can include a multiplicative scaling factor based on the fraction of the number of times the user selects the image in response to the search results of the search query of the given query-image training example identifying the image.

[0075] Figure 9 is a flowchart of an example process 900 for training an image embedding model using image-image training examples derived from the historical query logs of a search system. For convenience, process 900 will be described as being executed by a system of one or more computers located in one or more locations. For example, a training system (e.g., Figure 3 training system 300) programmed appropriately according to this specification can execute process 900.

[0076] The system processes data from the historical query logs of the search system to generate image-image training examples (902). Each image-image training example includes: (i) an image pair including a first image and a second image, (ii) selection data indicating the co-clickthrough rate of the image pair, the similar-image clickthrough rate of the image pair, or both.

[0077] The co-clickthrough rate of the image pair characterizes the frequency with which a user selects both the first image and the second image in response to both the first image and the second image being simultaneously identified by the search results of a search query. For example, the co-clickthrough rate of the image pair can indicate the fraction of the number of times the user selects both the first image and the second image in response to both the first image and the second image being simultaneously identified by the search results of a search query.

[0078] The similar-image clickthrough rate of the image pair characterizes the frequency with which a user selects the first image in response to the first image being identified by the search results of a search query including the second image, and vice versa. For example, the similar-image clickthrough rate of the image pair can indicate the fraction of the number of times the user selects the first image in response to the first image being identified by the search results of a search query including the second image, and vice versa.

[0079] Steps 904 - 908 describe an example process that can be performed for each image - image training example to train an image embedding model. For convenience, steps 904 - 908 describe the steps that can be performed for each image - image training example. More generally, any suitable method can be used to train the image embedding model. For example, a stochastic gradient descent method can be used to train the image embedding model, where steps 904 - 908 are iteratively repeated for a "batch" (i.e., set) of selected training examples. In this example, when a training termination criterion is met, the system can determine that training is complete. For example, the training termination criterion can be that a predetermined number of iterations of steps 904 - 908 have been performed. As another example, the training termination criterion can be that the change in the values of the parameters of the image embedding model between iterations of steps 904 - 908 is below a predetermined threshold. As will be described in more detail with reference to Figure 10 the image - image training examples can also be used in conjunction with query - image training examples to jointly train an image embedding model and a text embedding model using a graph - regularized loss function.

[0080] The system uses the image embedding model and processes the first image and the second image of the training example according to the current values of the image embedding model parameters to generate respective embeddings of the first image and the second image of the training example (904). For example, if the image embedding model is a neural network model, the system processes the first image and the second image (e.g., one after another) using a series of neural network layers defined by the architecture of the neural network model.

[0081] The system determines a similarity measure between the embedding of the first image and the embedding of the second image (906). For example, the system can determine the similarity measure by determining the Euclidean distance or cosine similarity measure between the respective embeddings of the first image and the second image.

[0082] The system adjusts the image embedding model parameters based at least in part on: (i) the similarity measure between the respective embeddings of the first image and the second image, and (ii) selection data (i.e., co - click - through rate, similar - image click - through rate, or both) (910). For example, when the image embedding model is a neural network, the system can determine the gradient of a loss function and use the gradient of the loss function to adjust the image embedding model parameters. The system can use any suitable method (e.g., backpropagation) to determine the gradient of the loss function. The loss function can be any suitable loss function that depends on the similarity measure between the respective embeddings and the selection data. For example, the loss function can be given by:

[0083] w·D(h θ (I1),h θ (I2))(1)

[0084] where hθ (I1) represents the embedding of the first image of an image-image training example, h θ (I2) represents the embedding of the second image of an image-image training example, D(·,·) is a similarity metric (e.g., Euclidean similarity metric), and the scaling factor w can be determined in any suitable manner using the co-click rate, similar image click rate, or both of the image-image training example. For example, the scaling intensity w can be determined as a linear combination of the co-click rate and similar image click rate of the image-image training example (e.g., using a predetermined weighting factor).

[0085] In some embodiments, the training system 300 can use query-image training examples and image-image training examples to jointly train the image embedding model 100 and the text embedding model 200 with a graph-regularized loss function. The query-image training examples and image-image training examples can be understood as representing a graph structure, where each node of the graph corresponds to a query-image training example, and each edge corresponds to an image-image training example. More specifically, an edge connecting a first node and a second node corresponding to a first query-image training example and a second query-image training example, respectively, in the graph can be defined by an image-image training example that includes the image pair specified by the first query-image training example and the second query-image training example. Specifically, the "intensity" of the edge connecting the first node and the second node can be defined based on the co-click rate, similar image click rate, or both specified by the image-image training example corresponding to the edge. Figure 10 An example of a portion of the graph representation is shown, where nodes 1002 and 1004 represent query-image training examples including respective images 1006 and 1008 of a car. In this example, the edge 1010 connecting nodes 1002 and 1004 is associated with the co-click rate X (where X can be a real number) and the similar image click rate Y (where Y can be a real number) of the image pair 1006 and 1008. More generally, some or all of the nodes of the graph may correspond to images included in the image-image training example, where the images are not included in any query-image training example (i.e., not associated with a corresponding text search query).

[0086] In one example, the graph-regularized loss function can have the following form:

[0087]

[0088] where i indexes the nodes in the graph representation, N is the total number of nodes, I i represents the image associated with the i-th node (e.g., the image of the image-query training example corresponding to the i-th node, if any), Q iThe search query representing the image-query training example (if any) corresponding to the i-th node, L1(I i ,Q i ) represents the loss function associated with the image-query training example corresponding to the i-th node (e.g., the classification loss or triplet loss described with reference to 812), N(i) represents the set of "neighbors" of node i in the graphical representation, w ij represents the strength of the edge connecting node i and node j in the graphical representation, h θ (I i ) represents the embedding of the image I i associated with the i-th node, h θ (I j ) represents the embedding of the image I j associated with the j-th node, D(·,·) is a similarity metric (e.g., Euclidean similarity metric). If two nodes in the graphical representation are connected by an edge, they are called neighbors. The strength w ij of the edge connecting nodes i and j can be determined in any appropriate way using the co-click rate, similar-image click rate, or both of the image-image training examples that define the edge. For example, the strength w ij of the edge connecting nodes i and j can be determined as a linear combination of the co-click rate and the similar-image click rate (e.g., using a predetermined weighting factor). For nodes associated with pictures rather than text search queries, the L1(I i ,Q i ) component of the loss defined by Equation 2 can be removed.

[0089] The training system 300 can jointly train the image embedding model 100 and the text embedding model 200 using any appropriate machine learning training technique, with a loss function using graph regularization (e.g., as described in Equation 2). For example, the training system 300 can jointly train the image embedding model 100 and the text embedding model 200 by stochastic gradient descent using an alternative representation of the loss function in Equation 2:

[0090]

[0091] Where i and j index the nodes represented by the graph. If nodes i and j are connected by an edge in the graph representation, then (i,j)∈ε. |i| represents the number of edges incident to node i, |j| represents the number of edges incident to node j, and the remaining variables are defined in the same way as in Equation 2. In this example, the training system 300 can perform stochastic gradient descent by sampling the edges from the graph representation in each training iteration and using backpropagation (or any other suitable technique) to perform the gradient of the loss function given by Equations 3 and 4. The training system 300 can determine that training is complete when any suitable training termination criterion is met, e.g., when a predetermined number of iterations of stochastic gradient descent have been performed. An example method for training the image embedding model 100 and the text embedding model 200 using a loss function with graph regularization is described with reference to T.D. Bui, S. Ravi, V. Ramavajjala, “Neural Graph Machines: Learning Neural Networks Using Graphs”, 2017, arXiv:1703.04818vl.

[0092] After the training system 300 determines the values of the image embedding model parameters 308 and the text embedding model parameters 310, the trained image embedding model 100 and the text embedding model 200 can be used for any of a variety of purposes. Some examples are given below.

[0093] In one example, the trained image embedding model 100 can be used by the search system 306 to rank image search results in response to a search query that includes a query image. More specifically, the search system 306 can use the image embedding model 100 to generate a corresponding embedding for each image in the search index maintained by the search system (as described with reference to Figure 11After receiving a search query that includes a query image, the search system 306 can use an image embedding model to generate an embedding of the query image, and thereafter use the generated embedding to determine a corresponding relevance score for each of the multiple images in the search index. The search system 306 can determine the relevance score of a given image in the search index based on a similarity metric (e.g., Euclidean distance) between the embedding of the given image and the embedding of the query image. The search system 306 can determine the ranking of the image search results of the search query at least in part based on the relevance scores determined using the embeddings generated by the image embedding model 100. As an example, consider a scenario where images of a piece of machinery are acquired regularly and used to determine when it may be necessary to replace or maintain components. The search index can contain images of components with various degrees of wear, and the search system can process the received component image to obtain an indication of the current state (e.g., current degree of wear) of the component embodied in the image. The system can further provide an output of the determined state and / or warn the operator if the component needs to be replaced or maintained. As another example, the query image can be a medical image, and the images can be of patients with various medical conditions or in various health states. The search system can process the received component image to obtain an indication of the current medical condition of the person embodied in the image, e.g., as a first-stage check to determine whether the medical image should be reviewed by a specialist.

[0094] In another example, both the trained text embedding model and the trained image embedding model can be used by the search system 306 to rank image search results in response to a search query that includes a sequence of one or more words. More specifically, the search system 306 can use the image embedding model 100 to generate a corresponding embedding for each image in the search index maintained by the search system. After receiving a search query that includes a sequence of one or more words, the search system can use the text embedding model to generate an embedding of the word sequence, and thereafter use the generated embedding to determine a corresponding relevance score for each of the multiple images in the search index. The search system can determine the relevance score of a given image in the search index based on a similarity metric (e.g., Euclidean distance) between the embedding of the given image and the embedding of the word sequence of the search query. The search system 306 can determine the ranking of the image search results of the search query at least in part based on the relevance scores determined using the embeddings generated by the image embedding model and the text embedding model.

[0095] In another example, a trained text embedding model can be used to determine "clusters" of similar keywords (or sequences of keywords), i.e., sets of keywords that express similar semantic content. In a specific example, clusters of similar keywords can be: "shoes", "shoe", "footwear", "boots", "cleats", "high heels", "slippers", "athletic shoes", etc. The text embedding model can be used to generate keyword clusters by determining the respective embeddings of each keyword in a keyword corpus and then using a clustering algorithm to cluster the keywords based on their respective embeddings. The clustering algorithm can be, for example, the k-means clustering algorithm or the expectation maximization clustering algorithm. The keyword clusters generated using the trained text embedding model can be used as distribution parameters that regulate the transmission of digital components (e.g., advertisements) to be presented with an electronic document (e.g., a web page).

[0096] In another example, both a trained text embedding model and a trained image embedding model can be used in an image classification system configured to process an image to generate an output that associates the image with a label from a predetermined set of labels. For example, the labels can specify object categories (e.g., person, cat, vehicle, etc.), and the image classification system can be trained to associate an image with the label of the object depicted in the image. In this example, the image classification system can use the image embedding model 100 to generate an embedding of the input image and can use the text embedding model to generate the respective embeddings of each search query in a search query corpus. The image classification system can determine a respective similarity metric between the embedding of the input image and the respective embeddings of each search query and can then associate the input image with a particular search query having the highest similarity metric. The image classification system can determine the label associated with the input image based on: (i) visual features derived from the input image, and (ii) semantic features derived from the particular search query. An example of an image classification system that can use the text embedding model 200 and the image embedding model 100 is described with reference to U.S. Patent Application No. 62 / 768,701.

[0097] Figure 11 An example search system 100 is shown. The search system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations, in which the systems, components, and techniques described below are implemented.

[0098] The search system 1100 is configured to receive a search query 1102 from a user device 1104, process the search query 1102 to determine one or more search results 1106 in response to the search query 1102, and provide the search results 1106 to the search device 1104. The search query 1102 may include search terms expressed in natural language (e.g., English), images, audio data, or any other suitable form of data. The search results 1106 identify electronic documents 1108 from websites 1110 in response to the search query 1102 and include links to the electronic documents 1108. The electronic documents 1108 may include, for example, images, HTML web pages, word processing documents, Portable Document Format (PDF) documents, and videos. The electronic documents 1108 may include content such as words, phrases, images, and audio data and may include embedded information (e.g., meta information and hyperlinks) and embedded instructions (e.g., scripts). A website 1110 is a collection of one or more electronic documents 1108 associated with a domain name and hosted by one or more servers. For example, a website 1110 may be a collection of web pages formatted in Hypertext Markup Language (HTML) that may contain text, images, multimedia content, and programming elements (e.g., scripts).

[0099] In a specific example, the search query 1102 may include the search term "Apollo moon landing", and the search system 1100 may be configured to perform an image search, i.e., provide search results 1106 that identify corresponding images in response to the search query 1102. Specifically, the search system 1100 may provide search results 1106, each search result including: (i) a title of a web page, (ii) a representation of an image extracted from the web page, and (iii) a hypertext link (e.g., specifying a Uniform Resource Locator (URL)) to the web page or the image itself. In this example, the search system 1100 may provide search results 1106 that include: (i) a title of a web page "Apollo moon landing", (ii) a reduced-size representation (i.e., thumbnail) of an image of an Apollo spacecraft included in the web page, and (iii) a hypertext link to the image.

[0100] A computer network 1112 such as a local area network (LAN), wide area network (WAN), the Internet, a mobile phone network, or a combination thereof connects the websites 1110, the user device 1104, and the search system 1100 (i.e., enables them to send and receive data over the network 1112). Generally, the network 1112 may connect the search system 1100 to thousands of websites 1110 and user devices 1104.

[0101] The user device 1104 is an electronic device under the control of a user and capable of sending and receiving data (including electronic document 1108) via a network 1112. Example user devices 1104 include personal computers, mobile communication devices, and other devices capable of sending and receiving data via the network 1112. The user device 1104 generally includes user applications (such as a web browser) that facilitate sending and receiving data via the network 1112. Specifically, the user applications included in the user device 1104 enable the user device 1104 to send a search query 1102 to the search system 1100 via the network 1112 and receive search results 1106 provided by the search system 1100 in response to the search query 1102.

[0102] The user applications included in the user device 1104 may present the search results 1106 received from the search system 1100 to the user of the user device (e.g., by rendering a search results page that displays an ordered list of the search results 1106). The user may select one of the search results 1106 presented by the user device 1104 (e.g., by clicking on a hypertext link included in the search results 1106), which may cause the user device 1104 to generate a request for the electronic document 1108 identified by the search results 1106. The request for the electronic document 1108 identified by the search results 1106 is sent via the network 1112 to the website 1110 that hosts the electronic document 1108. In response to receiving the request for the electronic document 1108, the website 1110 that hosts the electronic document 1108 may send the electronic document 1108 to the user device 1104.

[0103] The search system 1100 uses a ranking engine 1114 to process the search query 1102 to determine the search results 1106 in response to the search query 1102.

[0104] The search system 1100 uses an indexing engine 1120 to generate and maintain a search index 1116 by "crawling" (i.e., systematically browsing) the electronic documents 1108 of the website 1110. For each of a large number (e.g., millions) of electronic documents 1108, the search index 1116 suggests an index for the electronic document by maintaining (i) an identification of the electronic document 1108 (e.g., by a link pointing to the electronic document 1108) and (ii) data characterizing the electronic document 1108. The data maintained by the search index 1116 that characterizes the electronic document may include, for example, data specifying the type of the electronic document (e.g., image, video, PDF document, etc.), the quality of the electronic document (e.g., the resolution of the electronic document when the electronic document is an image or a video), keywords associated with the electronic document, a cached copy of the electronic document, or a combination thereof.

[0105] Search system 1100 may store a search index 1116 in a data store that may include thousands of data storage devices. An index engine 1120 may maintain the search index 1116 by continuously updating the search index 1116 (e.g., by indexing new electronic documents 1108 and removing electronic documents 1108 that are no longer available from the search index 1116).

[0106] Search system 1100 uses a query logging engine 1122 to generate and maintain a historical query log 1118 (as described above). Search system 1100 may store the historical query log 1118 in a data store that may include thousands of data storage devices. The query logging engine 1122 may maintain the historical query log 1118 by continuously updating the historical query log 1118 (e.g., by indexing new search queries as they are processed by the search system 1100).

[0107] A ranking engine 1114 determines search results 1106 in response to a search query 1102 by scoring electronic documents 1108 indexed by the search index 1116. The ranking engine 1114 may score the electronic documents 1108 based in part on data accessed from the historical query log 1118. The score determined by the ranking engine 1114 for an electronic document 1108 characterizes the responsiveness (e.g., relevance) of the electronic document to the search query 1102. The ranking engine 1114 determines a ranking based on the respective scores of the electronic documents 1108 indexed by the search index 1116, and determines search results based on the ranking. For example, the ranking engine 1114 may generate search results 1106 that identify the highest-ranked electronic documents 1108 indexed by the search index 1116.

[0108] Figure 12 is a block diagram of an example computer system 1200 that may be used to perform the operations described above. System 1200 includes a processor 1210, a memory 1220, a storage device 1230, and an input / output device 1240. Each of the components 1210, 1220, 1230, and 1240 may be interconnected, for example, using a system bus 1250. The processor 1210 is capable of processing instructions running within the system 1200. In one embodiment, the processor 1210 is a single-threaded processor. In another embodiment, the processor 1210 is a multi-threaded processor. The processor 1210 is capable of processing instructions stored in the memory 1220 or the storage device 1230.

[0109] The memory 1220 stores information within the system 1200. In one embodiment, the memory 1220 is a computer-readable medium. In one embodiment, the memory 1220 is a volatile memory unit. In another embodiment, the memory 1220 is a non-volatile memory unit.

[0110] The storage device 1230 can provide large-capacity storage for the system 1200. In one embodiment, the storage device 1230 is a computer-readable medium. In various different embodiments, the storage device 1230 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices (e.g., cloud storage device) over a network, or some other large-capacity storage device.

[0111] The input / output device 1240 provides input / output operations for the system 1200. In one embodiment, the input / output device 1240 may include one or more network interface devices, such as an Ethernet card, a serial communication device (e.g., RS-232 port), and / or a wireless interface device (e.g., 802.11 card). In another embodiment, the input / output device may include drive devices configured to receive input data and send output data to other input / output devices, such as a keyboard, a printer, and the display device 1260. However, other embodiments may also be used, such as mobile computing devices, mobile communication devices, set-top box TV client devices, etc.

[0112] Although example processing systems have been described in Figure 12 implementations and functional operations of the subject matter described in this specification can be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more of them.

[0113] This specification uses the term "configured" in connection with systems and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it means that the system has installed thereon software, firmware, hardware, or a combination thereof, which in operation causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it means that the one or more programs include instructions that, when run by a data processing apparatus, cause the apparatus to perform the operation or action.

[0114] Embodiments and functional operations of the subject matter described in this specification can be implemented in digital electronic circuitry, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and structural equivalents thereof), or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, a data processing apparatus. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiver apparatus for execution by the data processing apparatus.

[0115] The term “data processing apparatus” refers to data processing hardware and encompasses a variety of devices, apparatus, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. The apparatus may also be, or further include, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). In addition to hardware, the apparatus may optionally include code that creates an execution environment for the computer program, such as code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0116] A computer program (which may also be referred to as or described as a program, software, a software application, an application, a module, a software module, a script, or code) can be written in any form of programming language (including a compiled or interpreted language, or a declarative or procedural language); it can be deployed in any form (including as a stand-alone program or as a module, a component, a subroutine, or other unit suitable for use in a computing environment). A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program being discussed, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program can be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communication network.

[0117] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. In general, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines may be installed and run on the same computer or the same set of computers.

[0118] The processes and logical flows described in this specification can be performed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows can also be performed by, or by a combination of, special purpose logic circuitry, such as an FPGA or ASIC, or special purpose logic circuitry and one or more programmed computers.

[0119] Computers suitable for running a computer program can be based on a general purpose microprocessor or a special purpose microprocessor or both, or on any other kind of central processing unit. In general, the central processing unit will receive instructions and data from, or both, a read only memory or a random access memory. The basic elements of a computer are a central processing unit for executing or running instructions and one or more memory devices for storing the instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. In general, a computer will also include one or more mass storage devices for storing data (such as, for example, magnetic disks, magneto-optical disks, or optical disks), or operatively coupled to receive data from, or to send data to, or both to receive and to send data to, the one or more mass storage devices. However, a computer need not have such devices. In addition, a computer may be embedded in another device, for example, a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (such as, for example, a universal serial bus (USB) flash drive).

[0120] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0121] To support interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices may also be used to support interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic input, speech input, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from the devices used by the user; for example, interacting with the user by sending a web page to a web browser on the user device in response to a request received from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device (e.g., a smartphone running a messaging application) and receiving a response message as a reply from the user.

[0122] The data processing apparatus for implementing a machine learning model may also include, for example, a dedicated hardware accelerator unit for processing general and computationally intensive portions of machine learning training or production (i.e., inference) workloads.

[0123] A machine learning model may be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework).

[0124] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a backend component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a frontend component (e.g., a client computer having a graphical user interface, a web browser, or an application through which the user can interact with an implementation of the subject matter described in this specification), or includes any combination of one or more such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN) and a wide area network (WAN), such as the Internet.

[0125] A computing system may include a client and a server. The client and the server are generally remote from each other and typically interact via a communication network. The relationship between the client and the server is created by computer programs running on the respective computers and having a client-server relationship with each other. In some embodiments, for example, for the purpose of displaying data to a user interacting with a device acting as a client and receiving user input from the user, the server sends data such as an HTML page to the user device. Data generated at the user device, such as the result of a user interaction, can be received at the server from the device.

[0126] Although this specification contains many specific implementation details, these details should not be construed as limitations on the scope of any invention or on the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of separate embodiments in this specification may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Additionally, although features may be described above as acting in certain combinations and even initially claimed as such, in some cases, one or more features from the claimed combination may be excluded from the combination, and the claimed combination may refer to a sub-combination or a variant of a sub-combination.

[0127] Similarly, although operations are depicted in the drawings in a particular order and recited in the claims, this should not be understood to require that the operations be performed in the particular order shown or in a sequential order, or that all of the operations shown be performed to achieve a desired result. In some cases, multitasking and parallel processing may be advantageous. Additionally, the separation of the various system modules and components in the above embodiments should not be understood to require such separation in all embodiments; it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0128] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the acts recited in the claims may be performed in a different order and still achieve a desired result. As one example, the processes depicted in the drawings do not necessarily require the particular order shown or a sequential order to achieve a desired result. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A computing system, the system comprising: one or more processors; and one or more non - transitory computer - readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations including: obtaining historical query log data, wherein the historical query log data describes one or more search queries, one or more image search results responsive to the one or more search queries, and selections associated with each of the one or more image search results; generating image - image training examples based on the historical query log data, wherein the image - image training examples describe image pairs comprising a first image and a second image and selection data associated with the image pairs; processing the first image using an image embedding model to generate an embedding of the first image; processing the second image using the image embedding model to generate an embedding of the second image; estimating a loss function based on the embedding of the first image, the embedding of the second image, and the selection data associated with the image pair; and adjusting one or more parameters of the image embedding model based on the loss function estimated based on the embedding of the first image, the embedding of the second image, and the selection data associated with the image pair.

2. The system according to claim 1, wherein The selection data includes data describing a co - click - through rate of an image pair associated with the frequency that multiple users select both the first image and the second image simultaneously in response to search results of a search query for both the first image and the second image.

3. The system according to claim 1, wherein The selection data includes data describing a similar - image click - through rate of an image pair associated with the frequency that multiple users select the first image in response to search results of a search query including the second image that identify the first image.

4. The system according to claim 1, wherein, Estimating the loss function includes: determining a gradient of the loss function depending on: (i) a similarity metric between the embedding of the first image and the embedding of the second image, and (ii) the selection data of the image - image training example.

5. The system according to claim 4, wherein, Adjusting one or more parameters of the image embedding model includes using the gradient to adjust one or more parameters of the image embedding model.

6. The system according to claim 1, wherein The loss function includes a multiplicative scaling factor determined based on the selection data of the training example.

7. The system according to claim 1, wherein The historical query log data is associated with one or more web search systems.

8. The system according to claim 1, wherein If the co - click - through rate of the selection data describes that the first image and the second image are related, adjust one or more parameters to generate similar embeddings of the first image and the second image.

9. The system according to claim 1, wherein If the similar - image click - through rate of the selection data describes that the first image and the second image are related, adjust one or more parameters to generate similar embeddings of the first image and the second image.

10. The system according to claim 1, wherein, The image embedding model includes a neural network model configured to process a feature representation of an image.

11. The system according to claim 1, wherein, The image embedding model is jointly trained with a text embedding model, wherein the text embedding model is trained to embed one or more words of a text search query.

12. A computer - implemented method, the method comprising: obtaining, by a computing system including one or more processors, a search query from a user computing system, wherein the search query includes a query image; The computing system processes a query image using an image embedding model to generate a query image embedding, where the image embedding model is trained on a training data set that includes a plurality of image-image training examples, and each image training example describes an image pair and selection data associated with the image pair; The computing system determines one or more image search results based on the query image embedding, where the one or more image search results are based on respective relevance scores for each of the plurality of images; and The computing system provides the one or more image search results to a user computing system.

13. The computer-implemented method according to claim 12, wherein, The respective relevance scores are based on a similarity metric between the query image embedding and the respective image embedding of the respective image.

14. The computer-implemented method according to claim 12, wherein, The image embedding model is trained based on a similarity metric between the embedding of the first image of the image pair and the embedding of the second image of the image pair.

15. The computer-implemented method according to claim 12, wherein, The image embedding model is trained based on selection data associated with the image pair, where the selection data describes the co-click-through rate of the image pair associated with the frequency with which a plurality of users select both the first image and the second image in response to both the first image and the second image being simultaneously identified by search results.

16. The computer-implemented method according to claim 12, wherein, The image embedding model is trained by: Processing the first image of the image pair using the image embedding model to generate an embedding of the first image; Processing the second image in the image pair using the image embedding model to generate an embedding of the second image; Estimating a loss function based on the embedding of the first image, the embedding of the second image, and the selection data associated with the image pair; and And Adjusting one or more parameters of the image embedding model based on the loss function estimated based on the embedding of the first image, the embedding of the second image, and the selection data associated with the image pair.

17. The computer-implemented method according to claim 12, wherein, The search query further includes one or more words, and where the one or more image search results are determined based on the embeddings of the one or more words, and the embeddings of the one or more words are generated by processing the one or more words using a text embedding model.

18. One or more non-transitory computer-readable media, including instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations including: Obtaining historical query log data, where the historical query log data describes one or more search queries, one or more image search results in response to the one or more search queries, and selections associated with each of the one or more image search results; Generating image-image training examples based on the historical query log data, where the image-image training examples describe image pairs including a first image and a second image and selection data associated with the image pairs; Processing the first image using the image embedding model to generate an embedding of the first image; Processing the second image using the image embedding model to generate an embedding of the second image; Determining a similarity metric between the embedding of the first image and the embedding of the second image; and Adjusting one or more parameters of the image embedding model at least in part based on: (i) the similarity metric between the embedding of the first image and the embedding of the second image, and (ii) the selection data of the image-image training example.

19. The one or more non-transitory computer-readable media according to claim 18, wherein, Determining a similarity metric between an embedding of a first image and an embedding of a second image includes: Determining an Euclidean distance between an embedding of the first image and an embedding of the second image.

20. The one or more non-transitory computer-readable media according to claim 18, wherein, Selecting a co-click-through rate of the data description image pair and a similar image click-through rate of the image pair.

Images