Train image embedding models and text embedding models

Abstract

Methods, systems, and apparatus for training image embedding models, including computer programs encoded on computer storage media. In one aspect, a method includes: obtaining training data including a plurality of training examples, wherein each training example includes: an image pair including a first image and a second image; and selection data indicating one or more of: (i) a common click-through rate of the image pair, and (ii) a similar image click-through rate of the image pair; and using the training data to train an image embedding model having a plurality of image embedding model parameters.

Description

Technical Field

[0001] This specification relates to using machine learning models to process data. Background Art

[0002] A machine learning model receives input and produces an output, such as a predicted output, based on the received input. Some machine learning models are parametric models and produce an output based on the received input and the model's parameter values.

[0003] Some machine learning models are deep models that use multiple layers of models to produce outputs for received inputs. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers, each of which applies a nonlinear transformation to the received input to produce an output. Summary of the invention

[0004] This specification describes a training system implemented as a computer program on one or more computers in one or more locations that trains an image embedding model and a text embedding model using training data derived from a historical query log of a search system.

[0005] According to a first aspect, a method performed by one or more data processing devices is provided, the method comprising: obtaining training data comprising a plurality of training examples, wherein each training example comprises: an image pair comprising a first image and a second image; and selection data indicating one or more of the following: (i) a common click-through rate of the image pair, and (ii) a similar image click-through rate of the image pair, wherein: the common click-through rate of the image pair represents the frequency with which a user selects both the first image and the second image in response to the first image and the second image being simultaneously identified by search results of a search query; and the similar image click-through rate of the image pair represents the frequency with which a user selects both the first image and the second image in response to the first image or the second image being identified by a search result comprising the second image or the first image, respectively. The invention relates to a method for detecting a frequency of selecting a first image or a second image based on search result identification of a search query; and using training data to train an image embedding model having multiple image embedding model parameters, wherein, for each of the multiple training examples, the training includes: processing a first image of the training example using the image embedding model to generate an embedding of the first image; processing a second image of the training example using the image embedding model to generate an embedding of the second image; determining a similarity measure between the embedding of the first image and the embedding of the second image; and adjusting the image embedding model parameters based at least in part on: (i) a similarity measure between the embedding of the first image and the embedding of the second image, and (ii) the selection data of the training example.

[0006] In some implementations, the training data is generated using historical query logs of a web search system.

[0007] In some implementations, the co-click rate for each training example indicates a fraction of the number of times a user selected both the first image and the second image of the training example in response to both being simultaneously identified by search results of a search query.

[0008] In some implementations, the similar image click-through rate for each training example indicates a fraction of the number of times a user selected the first image or the second image in response to the first image or the second image being identified by a search result of a search query that included the second image or the first image, respectively.

[0009] In some implementations, the image embedding model includes a convolutional neural network.

[0010] In some embodiments, adjusting the image embedding model parameters includes: determining a gradient of a loss function that depends on: (i) a similarity measure between an embedding of a first image and an embedding of a second image, and (ii) selected data of training examples; and using the gradient to adjust the image embedding model parameters.

[0011] In some implementations, the loss function includes a multiplicative scaling factor determined based on selected data of the training examples.

[0012] In some implementations, the multiplier scaling factor is determined as a linear combination of the common click-through rates of the training examples and the click-through rates of similar images.

[0013] In some implementations, determining a similarity measure between the embedding of the first image and the embedding of the second image includes determining a Euclidean distance between the embedding of the first image and the embedding of the second image.

[0014] In some embodiments, the method may further include receiving a query image and processing the query image using a trained image embedding model obtained by a method according to any aspect or embodiment described herein. As an example, processing the query image may include determining a corresponding relevance score between the query image and a plurality of images in the search index.

[0015] According to another aspect, a method is provided, comprising: receiving a query image and processing the query image using a trained image embedding model obtained by a method according to any aspect or implementation described herein. As an example, processing the query image may include determining corresponding relevance scores between the query image and a plurality of images in a search index.

[0016] According to a second aspect, a system is provided, comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations including the operations of the previously described method.

[0017] According to a third aspect, there is provided one or more computer storage media (which may or may not be non-transitory storage media) storing instructions which, when executed by one or more computers, cause the one or more computers to perform operations including those of the previously described methods.

[0018] Particular embodiments of the subject matter described in this specification can be implemented to realize one or more of the following advantages.

[0019] The training system described in this specification can generate a large amount of training data (e.g., tens or hundreds of millions of training examples) for training image embedding models and text embedding models by processing data from historical query logs of a search system. The large amount of training data that can be effectively derived from historical query logs (e.g., of a web search system) enables the training system to train highly effective image embedding models and text embedding models. Such scalability (e.g., for training examples that may potentially include hundreds of millions of search queries) is a technical improvement in the field of model training. For example, this scalability enables the training of an image embedding model that generates image embeddings that implicitly represent concepts within a large range (e.g., food, scenes, landmarks, man-made products, etc.). In contrast, some conventional image embedding models generate image embeddings that only implicitly represent concepts within a small range (e.g., only food or only landmarks).

[0020] The training system can process the historical query logs to generate "query-image" training examples that associate text search queries with related images (e.g., images that users frequently select when they are identified by the search results of the text search query). Specifically, the query-image training examples can associate highly specific text search queries (e.g., "red 2014 Ford Mustang") with related images (e.g., depicting the object specified by the text search query). By jointly training the image embedding model and the text embedding model using the query-image training examples derived from the historical query logs, the training system can cause the image embedding model to generate image embeddings that implicitly represent highly specific concepts. For example, the trained image embedding model can process images to generate image embeddings that implicitly represent the color, style, and model of the car depicted in the image. This is a technical improvement in the field of model training, in contrast, for example, using training examples that associate images with general labels (e.g., "car") as in some conventional training datasets to train image embedding models and text embedding models may cause the image embedding model to generate relatively uninformative embeddings.

[0021] In some embodiments, the training system can generate query-image training examples that include search queries expressed in a large number of different natural languages ​​(e.g., English, French, German, etc.). By jointly training the image embedding model and the text embedding model using multilingual query-image training examples, the training system can train the text embedding model to generate informative text embeddings that are independent of the language of the text. For example, the training system can train the text embedding model to generate similar embeddings for the text based on the similarity of images associated with search queries including the text "young Queen Elizabeth" (English) and "young Queen Elizabeth" (French). This is another technical improvement in the field of model training,

[0022] The training system can train the image embedding model and the text embedding model based on selection data that characterizes, for example, how often two images are "co-clicked" or how often a given image is selected when it is identified by a search result for a search query (i.e., by "image-image" training examples). The selection data can be determined by aggregating user-derived signals (e.g., clicks) from millions of users and enables the training system to more efficiently train the image embedding model and the text embedding model.

[0023] Generating training data using conventional methods lacks many advantages of generating training data by processing historical query logs of search systems. For example, manually generating training data (e.g., manually specifying text labels for images by a person) is time-consuming and difficult, and generally only a relatively small amount of training data can be generated in this way. As another example, generating training data by associating images and captions drawn from social networks (or other sources) may generate less and lower quality training data than generating training data from historical query logs, for example, because the captions may not accurately characterize the content of the image. Generating more and / or higher quality training data and using the training data to train a machine learning model will improve the accuracy of the trainable machine learning model (therefore, providing a machine learning model that makes fewer mistakes than a machine learning model trained on training data generated using conventional methods). Additionally or alternatively, by reducing the time required to obtain labeled training data, a larger training data set can be established within a given time period, and this will also improve the accuracy of the trainable machine learning model. In addition, improving the accuracy of the trainable machine learning model can make it possible to use a simpler machine learning model (e.g., with fewer hidden layers), and this can lead to a reduction in the computing resources required to implement the machine learning model. In a complementary manner, data may need to be stored for shorter periods of time, so less memory storage overall may be required.

[0024] Details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 An example image embedding model is shown.

[0026] Figure 2 An example text embedding model is shown.

[0027] Figure 3 An example training system for training image embedding models and text embedding models using training data derived from historical query logs of a search system is shown.

[0028] Figure 4 An example search results page provided by a search system is shown that includes image search results for a search query that includes a sequence of one or more terms.

[0029] Figure 5 An example search results page provided by a search system is shown that includes image search results for a search query that includes images.

[0030] Figure 6An example process for jointly training an image embedding model and a text embedding model using query-image training examples is shown.

[0031] Fig. 7A An example process for training an image embedding model using image-image training examples is shown.

[0032] Figure 7B An example process for jointly training an image embedding model and a text embedding model using query-image training examples and image-image training examples is shown.

[0033] Figure 8 is a flow chart of an example process for jointly training an image embedding model and a text embedding model using query-image training examples derived from a historical query log of a search system.

[0034] Fig. 9 is a flow chart of an example process for training an image embedding model using image-image training examples derived from historical query logs of a search system.

[0035] Fig.10 An example of a portion of a graphical representation of query-image training examples and image-image training examples is shown.

[0036] Fig.11 An example search system is shown.

[0037] Fig.12 An example computer system is shown.

[0038] Like reference numerals and symbols in the various drawings indicate like elements. DETAILED DESCRIPTION

[0039] This specification describes a training system for training image embedding models and text embedding models using training data derived from historical query logs of a search system. The training data derived from the historical query logs may include query-image training examples, image-image training examples, or both.

[0040] A query-image training example includes: (i) a text search query, (ii) an image, and (iii) selection data characterizing how often a user selects the image in response to the image being identified by a search result of the text search query. If the selection data indicates that the user frequently selects the image when the image is identified by a search result of the search query, the training system can jointly train the image embedding model and the text embedding model to generate similar embeddings for the text search query and the image.

[0041] The image-image training example includes an image pair (including a first image and a second image) and selection data indicating the following: (i) a common click-through rate of the image pair, (ii) a similar image click-through rate of the image pair, or (iii) both. The common click-through rate of the image pair characterizes the frequency with which a user selects both the first image and the second image in response to both the first image and the second image being simultaneously identified by search results of a search query. The similar image click-through rate of the image pair characterizes the frequency with which a user selects the first image in response to the first image being identified by search results of a search query that includes the second image, and vice versa. The training system can train the image embedding model to generate similar embeddings for the first image and the second image when the common click-through rate, the similar image click-through rate, or both of the first image and the second image indicate that they are related.

[0042] In some embodiments, the training system may use query-image training examples and image-image training examples derived from historical query logs to jointly train the image embedding model and the query embedding model using a graph regularized loss function. Specifically, the query-image training examples and the image-image training examples may be represented as a graph structure, and the training system may jointly train the image embedding model and the query embedding model using a loss function based on the graph structure.

[0043] These and other features are described in more detail below.

[0044] Figure 1 An example image embedding model 100 is shown. The image embedding model 100 is configured to process an image 102 according to current values ​​of a set of image embedding model parameters to generate an embedding 104 for the image 102. The embedding 104 is a representation of the image 102 as an ordered set of numerical values ​​(e.g., as a vector or matrix). As will be described in more detail below, the image embedding model 100 may be trained using machine learning techniques to generate an embedding 104 for the image 102 that implicitly represents the semantic content of the image 102 (e.g., the object depicted by the image 102).

[0045] The image embedding model 100 may be configured to process an image 102 represented in any suitable format. For example, the image embedding model 100 may be configured to process an image 102 represented in a red-green-blue (RGB) color format (i.e., a format in which an image is represented by associating respective red values, green values, and blue values ​​with each pixel of the image). As another example, the image embedding model 100 may be configured to process a feature representation of the image 102, such as a Histogram of Oriented Gradients (HOG) representation, a Scale-Invariant Feature Transform (SIFT) representation, or a Speeded Up Robust Features (SURF) representation of the image 102. Other feature representations may also be used for training.

[0046] The image embedding model 100 may be a neural network model implemented by a computer program on one or more computers in one or more locations. For example, the image embedding model 100 may be a convolutional neural network with an architecture derived from an Inception neural network or a ResNet neural network.

[0047] Figure 2 An example text embedding model 200 is shown. The text embedding model 200 is configured to process a representation of a sequence of one or more words in a natural language (i.e., text 202) according to current values ​​of a set of text embedding model parameters to generate an embedding 204 for the text 202. The embedding 204 is a representation of the text 202 as an ordered set of numerical values ​​(e.g., as a vector or matrix). As will be described in more detail below, the text embedding model 200 can be trained using machine learning techniques to generate an embedding 204 for the text 202 that implicitly represents the semantic content of the text 202 (e.g., the object described by the text 202).

[0048] The text embedding model 200 may be configured to process text 202 represented in any suitable format. For example, the text embedding model 200 may be configured to process text 202 represented as a series of "one-hot" vectors, where each one-hot vector represents a corresponding character (or word) of the text 202. As another example, the text embedding model 200 may be configured to process text 202 represented by the output of a Word2vec model.

[0049] The text embedding model 200 can be a neural network model implemented by a computer program on one or more computers in one or more locations. For example, the text embedding model 200 can be a convolutional neural network with an architecture including multiple one-dimensional (1D) convolutional layers. As another example, the text embedding model 200 can be a lookup-based mapping from text 202 to embedding 204. As another example, the text embedding model 200 can be a series of fully connected layers configured to process n-gram text tokens. As another example, the text embedding model can be a recursive neural network model (e.g., LSTM) that is configured to sequentially process representations of characters of the text 202.

[0050] Figure 3An example training system 300 is shown for training the image embedding model 100 and the text embedding model 200 using training data 302 derived from a historical query log 304 of a search system 306. The training system 300 trains the image embedding model 100 and the text embedding model 200 by determining the values ​​of the image embedding model parameters 308 and the text embedding model parameters 310. For example, when the image embedding model 100 and the text embedding model 200 are implemented as corresponding neural networks, the training system 300 can iteratively adjust the parameters of the neural network using the gradient of the loss function, as described in more detail below. The training system 300 can be implemented by a computer program on one or more computers in one or more locations. In some embodiments, the training system 300 uses one or more tensor processing units (TPUs—application-specific integrated circuits (ASICs) designed for machine learning) during the training of the image embedding model 100 and the text embedding model 200.

[0051] Search system 306 can be any system configured to perform an image search by processing a search query that includes text, images, or both to generate search results that identify images in response to the search query. Fig.11 is described in more detail.

[0052] The historical query log 304 of the search system 306 indexes a large number (e.g., millions) of search queries previously processed by the search system 306. Specifically, the historical query log 304 can index the search queries by maintaining data including: (i) the search query, and (ii) data specifying one or more search results selected by a user of the device that sent the search query. The user can "select" the search result by expressing interest in the search result through any kind of interaction with the search result. For example, the user can select the search result by clicking on a hypertext link included in the search result or by hovering a cursor over the search result for a predetermined period of time to generate a request for an electronic document (e.g., an image) identified by the search result.

[0053] The training system 300 may process data from the historical query log 304 to generate query-image training examples and image-image training examples for training the image embedding model 100 and the text embedding model 200 .

[0054] The query-image training example includes: (i) a text search query, (ii) an image, and (iii) selection data characterizing how often a user selects the image in response to the image being identified by a search result of the text search query. If the selection data indicates that the user frequently selects the image when the image is identified by a search result of the search query, the training system 300 can jointly train the image embedding model 100 and the text embedding model 200 to generate similar embeddings for the search query and the image. Any suitable numerical similarity measure can be used to determine the similarity between embeddings (e.g., if the image embedding model 100 and the text embedding model 200 are configured to generate embeddings of the same dimensionality, then the Euclidean distance is used).

[0055] The image-image training example includes an image pair (which includes a first image and a second image) and selection data indicating: (i) a co-click rate of the image pair, (ii) a similar image click rate of the image pair, or (iii) both. The co-click rate of the image pair characterizes the frequency with which a user selects both the first image and the second image in response to both being simultaneously identified by search results of a search query. The similar image click rate of the image pair characterizes the frequency with which a user selects the first image in response to the first image being identified by search results of a search query that includes the second image, and vice versa. The training system 300 can train the image embedding model 100 to generate similar embeddings for the first image and the second image when the co-click rate, the similar image click rate, or both of the first image and the second image indicate that they are related.

[0056] Figure 4 An example search results page 400 provided by the search system 306 is shown, which includes image search results for a search query that includes a sequence of one or more words. Specifically, the search results page 400 displays search results 402, 404, and 406 for the search query 408: "red Ford Mustang".

[0057] Figure 5 An example search results page 500 provided by the search system 306 is shown, which includes image search results for a search query that includes images. Specifically, the search results page 500 displays search results 502, 504, and 506 for a search query 508 that includes images depicting trucks. In response to receiving a search query that includes a query image, the search system 306 can be configured to provide search results that identify images that are similar to the query image. In this example, each of the search results 502, 504, and 506 identifies an image that is similar to the query image.

[0058] If a user selects search results that respectively identify the first image and the second image from the same set of search results, the user is said to "co-click" the first image and the second image. For example, the user can co-click the image identified by search result 402 and the image identified by search result 404 by selecting two search results on search result page 400 (e.g., one after another). As another example, the user can co-click the image identified by search result 504 and the image identified by search result 506 by selecting two search results on search result page 500 (e.g., one after another). If the user selects three or more search results from the same set of search results, the images identified by each pair of selected search results can be considered to be co-clicked. For example, if the user selects search results A, B, and C from the same set of search results, the image pairs identified by search results {A, B}, {A, C}, and {B, C} can all be considered to be co-clicked.

[0059] Figure 6 An example process for jointly training the image embedding model 100 and the text embedding model 200 using a query-image training example 600 is shown. The query-image training example 600 includes a search query 602 and an image 604, the search query 602 including a sequence of one or more words. The training system 300 processes the image 604 using the image embedding model 100 and according to the current values ​​of the image embedding model parameters 308 to generate an embedding 606 for the image 604. The training system 300 processes the search query 602 using the text embedding model 200 and according to the current values ​​of the text embedding model parameters 310 to generate an embedding 608 for the search query 602.

[0060] The training system 300 determines a similarity measure 610 between the embedding 606 of the image 604 and the embedding 608 of the search query 602, and determines a model parameter adjustment 612 based on the similarity measure 610. Thereafter, the training system 300 uses the model parameter adjustment 612 to adjust the values ​​of the image embedding model parameters 308 and the text embedding model parameters 310. In some embodiments, in determining the model parameter adjustment 612, the training system 300 uses selection data that characterizes the frequency with which the user selected the image 604 in response to the image 604 being identified by the search results of the search query 602. An example process for jointly training an image embedding model and a text embedding model using query-image training examples is provided with reference to Figure 8 is described in more detail.

[0061] Fig. 7AAn example process is shown for training the image embedding model 100 using an image-image training example 700 including a first image 702 and a second image 704. The training system 300 processes the first image 702 using the image embedding model 100 and according to current values ​​of the image embedding model parameters 308 to generate an embedding 706 for the first image 702. Similarly, the training system 300 processes the second image 704 using the image embedding model 100 and according to current values ​​of the image embedding model parameters 308 to generate an embedding 708 for the second image 704.

[0062] The training system 300 determines a similarity metric 710 between the embedding 706 of the first image 702 and the embedding 708 of the second image 704, and determines a model parameter adjustment 712 based on the similarity metric 710. In some embodiments, the training system 300 uses selection data in determining the model parameter adjustment 712, the selection data characterizing the common click-through rate, the similar image click-through rate, or both of the first image 702 and the second image 704. An example process for training an image embedding model using image-image training examples is provided in Fig. 9 is described in more detail.

[0063] Figure 7B An example process for jointly training an image embedding model and a text embedding model using query-image training examples and image-image training examples is shown. Specifically, in each of a plurality of training iterations, one or more query-image training examples 600 and one or more image-image training examples 700 may be processed by the image embedding model to generate corresponding embeddings (e.g., referring to Figure 6 and Fig. 7A The training system 300 can be based on the query-image training example 600 (as shown in FIG. Figure 6 ) and image-to-image training examples (as described in Fig. 7A The training system 300 may then use the model parameter adjustments 714 to adjust the current values ​​of the image embedding model parameters 308 and the text embedding model parameters 310. The model parameter adjustments determined based on the query-image training examples may be weighted more heavily or less heavily (e.g., using a gradient scaling factor) than the model parameter adjustments determined based on the image-image training examples. The weights applied to the model parameter adjustments may be adjustable system hyperparameters.

[0064] Figure 8is a flow chart of an example process 800 for jointly training an image embedding model and a text embedding model using query-image training examples derived from a historical query log of a search system. For convenience, process 800 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system (e.g., Figure 3 The training system 300) can perform process 800.

[0065] The system processes data from a historical query log of a search system to generate a candidate set of query-image training examples (802). Each query-image training example includes: (i) a search query including a sequence of one or more words, (ii) an image, and (iii) selection data characterizing how often a user selected the image in response to the image being identified by a search result of the search query. The selection data may indicate a score for the number of times the image was selected in response to the image being identified by a search result of the search query.

[0066] The system selects query-image training examples for jointly training the image embedding model and the text embedding model from a candidate set of training examples based at least in part on the selection data for the training examples (804). For example, if an image of a particular training example is most frequently selected by a user in response to the image being identified by a search result of a search query for the particular query-image training example, the system may select the particular query-image training example. As another example, if an image of a particular query-image training example is among the top N images most frequently selected by a user after being identified by a search result of a search query for the particular query-image training example, the system may select the particular query-image training example. The system may use any of a variety of other appropriate criteria when selecting query-image training examples for jointly training the image embedding model and the text embedding model. For example, the system may limit the number of query-image training examples selected to include search queries that specify a name of a particular person and corresponding images depicting the particular person. In this example, because the appearance of the same person may vary significantly between images (e.g., because the person is wearing different clothes, shoes, glasses, etc.), including a large number of query-image training examples corresponding to a particular person may reduce the effectiveness of the training process.

[0067] Steps 806-812 describe an example process that can be performed for each selected query-image training example to jointly train the image embedding model and the text embedding model. For convenience, steps 806-812 describe steps that can be performed for a given query-image training example. More generally, any suitable method can be used to jointly train the image embedding model and the text embedding model. For example, a stochastic gradient descent method can be used to jointly train the image embedding model and the text embedding model, where steps 806-812 are iteratively repeated for a "batch" (i.e., a collection) of query-image training examples. In this example, the system can determine that the training is complete when the training termination criteria are met. For example, the training termination criteria can be that a predetermined number of iterations of steps 806-812 have been performed. As another example, the training termination criteria can be that the values ​​of the parameters of the image embedding model and the text embedding model change between iterations of steps 806-812 below a predetermined threshold.

[0068] The system processes the image of the given query-image training example using the image embedding model and according to the current values ​​of the image embedding model parameters to generate an embedding for the image (806). For example, if the image embedding model is a neural network model, the system processes the image using a series of neural network layers defined by the architecture of the neural network model.

[0069] The system processes the representation of the search query for the given query-image training example using the text embedding model and according to the current values ​​of the text embedding model parameters to generate an embedding of the search query (808). For example, if the text embedding model is a neural network model, the system processes the representation of the search query using a series of neural network layers defined by the architecture of the neural network model.

[0070] The system determines a similarity measure between the embedding of the image and the embedding of the search query for a given query-image training example (810). For example, the embedding of the image and the embedding of the search query may have the same dimensionality, and the system may determine the similarity measure by determining a Euclidean distance or a cosine similarity measure between the corresponding embeddings.

[0071] The system adjusts image embedding model parameters and text embedding model parameters based at least in part on a similarity measure between an embedding of an image of a given query-image training example and an embedding of a search query (812). For example, when the image embedding model and the text embedding model are respective neural network models, the system can determine a gradient of a loss function and use the gradient of the loss function to adjust the image embedding model parameters and the text embedding model parameters. The system can determine the gradient of the loss function using any suitable method (e.g., back propagation). The loss function can be any suitable loss function that depends on a similarity measure between an embedding of an image of a given query-image training example and an embedding of a search query. The following are some examples.

[0072] In some embodiments, the loss function can be a classification loss function. In these embodiments, the search query of a given query-image training example is considered to identify a "positive" label for the image of the given query-image training example. The search queries of other query-image training examples are considered to identify corresponding "negative" labels for the images of the given query-image training example. More specifically, the system can determine the similarity measure between the embedding of the image of the given query-image training example and the embedding of the search query of the given query-image training example as a "positive" score. The system can determine the corresponding "negative" score of each other training example as a similarity measure between the embedding of the image of the given query-image training example and the embedding of the search query of the other training example. The system can use a soft-max (or sampled soft-max) function to process the positive and negative scores, and provide the output of the soft-max (or sampled soft-max) function to a cross entropy loss function (or any other appropriate classification loss function).

[0073] In some embodiments, the loss function may be a triplet loss function. In these embodiments, the system may determine the embedding of the image of a given query-image training example as an "anchor," the embedding of the search query of the given query-image training example as a "positive," and the embedding of the search query of another query-image training example as a "negative."

[0074] Alternatively, the loss function may depend on selection data for a given query-image training example, the selection data characterizing how often a user selected an image in response to the image being identified by a search result of a search query for the given query-image training example. For example, the loss function may include a multiplicative scaling factor based on a fraction of the number of times a user selected an image in response to the image being identified by a search result of a search query for the given query-image training example.

[0075] Fig. 9is a flow chart of an example process 900 for training an image embedding model using image-image training examples derived from a historical query log of a search system. For convenience, process 900 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system (e.g., Figure 3 The training system 300) can perform process 900.

[0076] The system processes data from a historical query log of a search system to generate image-image training examples (902). Each image-image training example includes: (i) an image pair including a first image and a second image, and (ii) selection data indicating a common click-through rate of the image pair, a similar image click-through rate of the image pair, or both.

[0077] The co-click rate of the image pair characterizes the frequency with which a user selects both the first image and the second image in response to both the first image and the second image being simultaneously identified by the search results of the search query. For example, the co-click rate of the image pair may indicate a fraction of the number of times both the first image and the second image were selected in response to both the first image and the second image being simultaneously identified by the search results of the search query.

[0078] The similar image click-through rate of an image pair characterizes how often a user selects a first image in response to the first image being identified by a search result of a search query that includes the second image, and vice versa. For example, the similar image click-through rate of an image pair may indicate a fraction of the number of times a user selects a first image in response to the first image being identified by a search result of a search query that includes the second image, and vice versa.

[0079] Steps 904-908 describe an example process that can be performed for each image-image training example to train the image embedding model. For convenience, steps 904-908 describe steps that can be performed for each image-image training example. More generally, any suitable method can be used to train the image embedding model. For example, the image embedding model can be trained using a stochastic gradient descent method, where steps 904-908 are iteratively repeated for a "batch" (i.e., a collection) of selected training examples. In this example, the system can determine that training is complete when a training termination criterion is met. For example, the training termination criterion can be that a predetermined number of iterations of steps 904-908 have been performed. As another example, the training termination criterion can be that the change in the value of a parameter of the image embedding model between iterations of steps 904-908 is below a predetermined threshold. As will be referred to Fig.10 Described in more detail, image-image training examples can also be used in conjunction with query-image training examples to jointly train the image embedding model and the text embedding model using a graph regularized loss function.

[0080] The system processes the first image and the second image of the training example using the image embedding model and according to the current values ​​of the image embedding model parameters to generate respective embeddings of the first image and the second image of the training example (904). For example, if the image embedding model is a neural network model, the system processes the first image and the second image using a series of neural network layers defined by the architecture of the neural network model (e.g., one after the other).

[0081] The system determines a similarity measure between the embedding of the first image and the embedding of the second image (906). For example, the system can determine the similarity measure by determining a Euclidean distance or a cosine similarity measure between the respective embeddings of the first image and the second image.

[0082] The system adjusts image embedding model parameters based at least in part on: (i) a similarity measure between respective embeddings of a first image and a second image, and (ii) the selection data (i.e., co-click rate, similar image click rate, or both) (910). For example, when the image embedding model is a neural network, the system can determine a gradient of a loss function and use the gradient of the loss function to adjust the image embedding model parameters. The system can determine the gradient of the loss function using any suitable method (e.g., back propagation). The loss function can be any suitable loss function that depends on a similarity measure between the corresponding embedding and the selection data. For example, the loss function can be given by:

[0083] w·D(h θ (I 1 ),h θ (I 2 )) (1)

[0084] where h θ (I 1 ) represents the embedding of the first image of an image-image training example, h θ (I 2 ) denotes the embedding of the second image of the image-image training example, D(·,·) is a similarity measure (e.g., a Euclidean similarity measure), and the scaling factor w can be determined in any suitable manner using the co-click rate, the similar image click rate, or both of the image-image training example. For example, the scaling strength w can be determined as a linear combination of the co-click rate and the similar image click rate of the image-image training example (e.g., using a predetermined weighting factor).

[0085] In some embodiments, the training system 300 can use query-image training examples and image-image training examples to jointly train the image embedding model 100 and the text embedding model 200 with a graph regularized loss function. The query-image training examples and the image-image training examples can be understood as representing a graph structure, where each node of the graph corresponds to a query-image training example and each edge corresponds to an image-image training example. More specifically, the edge in the graph connecting the first node and the second node corresponding to the first query-image training example and the second query-image training example, respectively, can be defined by the image-image training example, which includes the image pair specified by the first query-image training example and the second query-image training example. Specifically, the "strength" of the edge connecting the first node and the second node can be defined based on the co-click rate, similar image click rate, or both specified by the image-image training example corresponding to the edge. Fig.10 An example of a portion of a graph representation is shown, where nodes 1002 and 1004 represent query-image training examples that include respective images 1006 and 1008 of cars. In this example, edge 1010 connecting nodes 1002 and 1004 is associated with a common click rate X (where X can be a real number) and a similar image click rate Y (where Y can be a real number) for the image pair 1006 and 1008. More generally, some or all nodes of the graph may correspond to images included in the image-image training examples, where the images are not included in any query-image training examples (i.e., are not associated with a corresponding text search query).

[0086] In one example, the loss function for graph regularization can have the following form:

[0087]

[0088] Where i is the index of the node in the graph representation, N is the total number of nodes, and I i represents the image associated with the ith node (e.g., the image corresponding to the ith node - the image of the query training example (if any)), Q i denotes the search query corresponding to the image-query training example (if any) of the i-th node, L 1 (I i ,Q i ) represents the loss function associated with the image-query training example corresponding to the i-th node (e.g., the classification loss or triplet loss described with reference to 812), N(i) represents the set of “neighbors” of node i in the graph representation, and w ij represents the strength of the edge connecting node i and node j in the graph representation, h θ (I i ) represents the image I associated with the i-th nodei The embedding of h θ (I j ) represents the image I associated with the jth node j is the embedding of , and D(·,·) is a similarity measure (e.g., a Euclidean similarity measure). If two nodes in the graph representation are connected by an edge, they are called neighbors. The strength w of the edge connecting nodes i and j can be determined in any appropriate manner using the co-click rate of image-image training examples defining the edge, the similar image click rate, or both ij For example, the strength w of the edge connecting nodes i and j can be ij is determined as a linear combination of the common click rate and the similar image click rate (eg, using a predetermined weighting factor). For nodes associated with images rather than textual search queries, the loss L defined by Equation 2 may be removed. 1 (I i ,Q i ) quantity.

[0089] The training system 300 may jointly train the image embedding model 100 and the text embedding model 200 using any suitable machine learning training technique, using a graph regularized loss function (e.g., as described in Equation 2). For example, the training system 300 may jointly train the image embedding model 100 and the text embedding model 200 by stochastic gradient descent using an alternative representation of the loss function in Equation 2:

[0090]

[0091]

[0092] Where i and j are indexes of nodes of the graph representation, if node i and node j are connected by an edge in the graph representation, then, (i, j)∈ε, |i| represents the number of edges incident to node i, |j| represents the number of edges incident to node j, and the remaining variables are defined in the same manner as in Equation 2. In this example, the training system 300 can perform stochastic gradient descent by sampling edges from the graph representation in each training iteration and using backpropagation (or any other suitable technique) to perform the gradient of the loss function given by equations 3 and 4. The training system 300 can determine that training is complete when any suitable training termination criteria are met, for example, when a predetermined number of iterations of stochastic gradient descent have been performed. An example method for training the image embedding model 100 and the text embedding model 200 using a graph regularized loss function is described with reference to T. D. Bai, S. Ravi, V. Ramavajjala, “Neural Graph Machines: Learning Neural Networks Using Graphs”, 2017, arXiv:1703.04818vl.

[0093] After the training system 300 determines the values ​​of the image embedding model parameters 308 and the text embedding model parameters 310, the trained image embedding model 100 and the text embedding model 200 can be used for any of a variety of purposes. The following are some examples.

[0094] In one example, the trained image embedding model 100 can be used by the search system 306 to rank image search results in response to a search query that includes a query image. More specifically, the search system 306 can use the image embedding model 100 to generate a corresponding embedding for each image in a search index maintained by the search system (e.g., referring to Fig.11As described). After receiving a search query including a query image, the search system 306 can use the image embedding model to generate an embedding of the query image, and thereafter use the generated embedding to determine a corresponding relevance score for each of the multiple images in the search index. The search system 306 can determine the relevance score of a given image in the search index based on a similarity measure (e.g., Euclidean distance) between the embedding of the given image and the embedding of the query image. The search system 306 can determine the ranking of image search results for the search query based at least in part on the relevance score determined using the embedding generated by the image embedding model 100. As an example, consider a situation in which images of a piece of mechanical equipment are periodically acquired and used to determine when a component may need to be replaced or maintained. The search index can contain images of components with various degrees of wear, and the search system can process the received component images to obtain an indication of the current state of the component embodied in the image (e.g., the current degree of wear). The system can further provide an output of the determined state and / or warn an operator if the component needs to be replaced or maintained. As another example, the query image can be a medical image, and the image can be an image of a patient with various medical conditions or in various health states. The search system may process the received component images to obtain an indication of the current medical condition of the person represented in the image, for example, as a first stage check to determine whether the medical image should be reviewed by a specialist.

[0095] In another example, both the trained text embedding model and the trained image embedding model can be used by the search system 306 to rank image search results in response to a search query that includes a sequence of one or more words. More specifically, the search system 306 can use the image embedding model 100 to generate a corresponding embedding for each image in a search index maintained by the search system. After receiving a search query that includes a sequence of one or more words, the search system can use the text embedding model to generate an embedding for the word sequence, and then use the generated embedding to determine a corresponding relevance score for each of the multiple images in the search index. The search system can determine the relevance score of a given image in the search index based on a similarity measure (e.g., Euclidean distance) between the embedding of the given image and the embedding of the word sequence of the search query. The search system 306 can determine the ranking of the image search results for the search query based at least in part on the relevance scores determined using the embeddings generated by the image embedding model and the text embedding model.

[0096] In another example, the trained text embedding model can be used to determine a "cluster" of similar keywords (or keyword sequences), that is, a set of keywords that express similar semantic content. In a specific example, the cluster of similar keywords can be: "shoes", "shoes", "footwear", "boots", "spikes", "high heels", "slippers", "sneakers", etc. The text embedding model can be used to generate keyword clusters by determining the corresponding embedding of each keyword in the keyword corpus and then using a clustering algorithm to cluster the keywords based on their respective embeddings. The clustering algorithm can be, for example, a k-means clustering algorithm or an expectation-maximization clustering algorithm. The keyword clusters generated using the trained text embedding model can be used as distribution parameters that regulate the transmission of digital components (e.g., advertisements) for presentation with electronic documents (e.g., web pages).

[0097] In another example, both the trained text embedding model and the trained image embedding model can be used in an image classification system that is configured to process an image to generate an output that associates the image with a label from a predetermined set of labels. For example, the label can specify an object category (e.g., a person, a cat, a vehicle, etc.), and the image classification system can be trained to associate the image with the label of the object depicted in the image. In this example, the image classification system can use the image embedding model 100 to generate an embedding for an input image, and can use the text embedding model to generate a corresponding embedding for each search query in a search query corpus. The image classification system can determine a corresponding similarity measure between the embedding of the input image and the corresponding embedding of each search query, and thereafter can associate the input image with a specific search query with the highest similarity measure. The image classification system can determine the label associated with the input image based on the following two items: (i) visual features derived from the input image, and (ii) semantic features derived from a specific search query. An example of an image classification system that can use the text embedding model 200 and the image embedding model 100 is described with reference to U.S. Patent Application No. 62 / 768,701.

[0098] Fig.11 An example search system 100 is shown. Search system 100 is an example of a system implemented as a computer program on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented.

[0099] The search system 1100 is configured to receive a search query 1102 from a user device 1104, to process the search query 1102 so as to determine one or more search results 1106 in response to the search query 1102, and to provide the search results 1106 to the search device 1104. The search query 1102 may include a search term expressed in a natural language (e.g., English), an image, audio data, or any other suitable form of data. The search results 1106 identify an electronic document 1108 in response to the search query 1102 from a website 1110, and include a link to the electronic document 1108. The electronic document 1108 may include, for example, an image, an HTML web page, a word processing document, a portable document format (PDF) document, and a video. The electronic document 1108 may include content such as words, phrases, images, and audio data, and may include embedded information (e.g., meta information and hyperlinks) and embedded instructions (e.g., scripts). The website 1110 is a collection of one or more electronic documents 1108 associated with a domain name and hosted by one or more servers. For example, website 1110 may be a collection of web pages formatted in Hypertext Markup Language (HTML) that may contain text, images, multimedia content, and programming elements (eg, scripts).

[0100] In a specific example, the search query 1102 may include the search term "Apollo Moon Landing", and the search system 1100 may be configured to perform an image search, i.e., provide search results 1106 that identify corresponding images responsive to the search query 1102. Specifically, the search system 1100 may provide search results 1106, each search result including: (i) the title of a web page, (ii) a representation of an image extracted from the web page, and (iii) a hypertext link (e.g., specifying a uniform resource locator (URL)) pointing to the web page or the image itself. In this example, the search system 1100 may provide search results 1106 that include: (i) the title of the web page "Apollo Moon Landing", (ii) a reduced size representation (i.e., a thumbnail) of an image of an Apollo spacecraft included in the web page, and (iii) a hypertext link pointing to the image.

[0101] A computer network 1112, such as a local area network (LAN), a wide area network (WAN), the Internet, a mobile phone network, or a combination thereof, connects the websites 1110, user devices 1104, and the search system 1100 (i.e., enables them to send and receive data through the network 1112). In general, the network 1112 can connect the search system 1100 to thousands of websites 1110 and user devices 1104.

[0102] User device 1104 is an electronic device under user control and capable of sending and receiving data (including electronic documents 1108) over network 1112. Example user devices 1104 include personal computers, mobile communication devices, and other devices that can send and receive data over network 1112. User device 1104 typically includes a user application (e.g., a web browser) that facilitates sending and receiving data over network 1112. In particular, the user application included in user device 1104 enables user device 1104 to send search queries 1102 to search system 1100 over network 1112 and receive search results 1106 provided by search system 1100 in response to search queries 1102.

[0103] The user application included in the user device 1104 may present the search results 1106 received from the search system 1100 to the user of the user device (e.g., by rendering a search results page that displays an ordered list of search results 1106). The user may select one of the search results 1106 presented by the user device 1104 (e.g., by clicking on a hypertext link included in the search results 1106), which may cause the user device 1104 to generate a request for the electronic document 1108 identified by the search results 1106. The request for the electronic document 1108 identified by the search results 1106 is sent via the network 1112 to the website 1110 that hosts the electronic document 1108. In response to receiving the request for the electronic document 1108, the website 1110 that hosts the electronic document 1108 may send the electronic document 1108 to the user device 1104.

[0104] The search system 1100 processes the search query 1102 using a ranking engine 1114 to determine search results 1106 that are responsive to the search query 1102 .

[0105] The search system 1100 uses an indexing engine 1120 to generate and maintain a search index 1116 by “crawling” (i.e., systematically browsing) the electronic documents 1108 of the website 1110. For each of a large number (e.g., millions) of electronic documents 1108, the search index 1116 suggests an index for the electronic document by maintaining (i) identifying the electronic document 1108 (e.g., by a link pointing to the electronic document 1108) and (ii) data characterizing the electronic document 1108. The data characterizing the electronic document maintained by the search index 1116 may include, for example, data specifying the type of electronic document (e.g., image, video, PDF document, etc.), the quality of the electronic document (e.g., the resolution of the electronic document when the electronic document is an image or video), keywords associated with the electronic document, a cached copy of the electronic document, or a combination thereof.

[0106] The search system 1100 can store the search index 1116 in a data store that can include thousands of data storage devices. The index engine 1120 can maintain the search index 1116 by continuously updating the search index 1116 (e.g., by indexing new electronic documents 1108 and deleting electronic documents 1108 that are no longer available from the search index 1116).

[0107] The search system 1100 generates and maintains a historical query log 1118 (as described above) using a query logging engine 1122. The search system 1100 can store the historical query log 1118 in a data store that can include thousands of data storage devices. The query logging engine 1122 can maintain the historical query log 1118 by continuously updating the historical query log 1118 (e.g., by indexing new search queries as they are processed by the search system 1100).

[0108] The ranking engine 1114 determines search results 1106 that are responsive to the search query 1102 by scoring the electronic documents 1108 indexed by the search index 1116. The ranking engine 1114 can score the electronic documents 1108 based in part on data accessed from the historical query log 1118. The scores determined by the ranking engine 1114 for the electronic documents 1108 characterize the responsiveness (e.g., relevance) of the electronic documents to the search query 1102. The ranking engine 1114 determines the rankings of the electronic documents 1108 indexed by the search index 1116 based on their respective scores, and determines search results based on the rankings. For example, the ranking engine 1114 can generate search results 1106 that identify the highest ranked electronic documents 1108 indexed by the search index 1116.

[0109] Fig.12 12 is a block diagram of an example computer system 1200 that can be used to perform the above operations. System 1200 includes a processor 1210, a memory 1220, a storage device 1230, and an input / output device 1240. Each of the components 1210, 1220, 1230, and 1240 can be interconnected, for example, using a system bus 1250. Processor 1210 is capable of processing instructions that are executed within system 1200. In one embodiment, processor 1210 is a single-threaded processor. In another embodiment, processor 1210 is a multi-threaded processor. Processor 1210 is capable of processing instructions stored in memory 1220 or storage device 1230.

[0110] The memory 1220 stores information within the system 1200. In one embodiment, the memory 1220 is a computer readable medium. In one embodiment, the memory 1220 is a volatile memory unit. In another embodiment, the memory 1220 is a non-volatile memory unit.

[0111] The storage device 1230 can provide mass storage for the system 1200. In one embodiment, the storage device 1230 is a computer-readable medium. In various embodiments, the storage device 1230 may include, for example, a hard disk device, an optical disk device, a storage device shared by multiple computing devices (e.g., a cloud storage device) on a network, or some other mass storage device.

[0112] The input / output devices 1240 provide input / output operations for the system 1200. In one embodiment, the input / output devices 1240 may include one or more network interface devices, such as an Ethernet card, a serial communication device (e.g., an RS-232 port), and / or a wireless interface device (e.g., an 802.11 card). In another embodiment, the input / output devices may include a driver device configured to receive input data and send output data to other input / output devices, such as a keyboard, a printer, and a display device 1260. However, other embodiments may also be used, such as a mobile computing device, a mobile communication device, a set-top television client device, etc.

[0113] Although there are Fig.12 An example processing system is described in the specification, but the implementation and functional operations of the subject matter described in this specification may be implemented in other types of digital electronic circuits, or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more thereof.

[0114] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers to be configured to perform a particular operation or action, it is meant that the system has installed thereon software, firmware, hardware, or a combination thereof, which, in operation, causes the system to perform the operation or action. For one or more computer programs to be configured to perform a particular operation or action, it is meant that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform the operation or action.

[0115] The embodiments and functional operations of the subject matter described in this specification may be implemented in digital electronic circuits, in tangibly embodied computer software or firmware, in computer hardware (including the structures disclosed in this specification and their structural equivalents), or in a combination of one or more thereof. The embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium to be run by a data processing device or to control the operation of a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical signal, optical signal, or electromagnetic signal) that is generated to encode information and transmitted to a suitable receiver device for operation by a data processing device.

[0116] The term "data processing apparatus" refers to data processing hardware and covers various apparatuses, devices and machines for processing data, including, for example, a programmable processor, a computer, or a plurality of processors or computers. The apparatus may also be or may further include a dedicated logic circuit, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to the hardware, the apparatus may optionally include code that creates an operating environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more thereof.

[0117] A computer program (which may also be referred to or described as a program, software, software application, application, module, software module, script, or code) may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to run on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0118] In this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally speaking, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines can be installed and run on the same computer or multiple computers.

[0119] The processes and logic flows described in this specification can be performed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a special purpose logic circuit (e.g., an FPGA or ASIC), or by a combination of a special purpose logic circuit and one or more programmed computers.

[0120] The computer suitable for running a computer program can be based on a general-purpose microprocessor or a special-purpose microprocessor or both, or based on a central processing unit of any other kind. Generally speaking, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The basic element of a computer is a central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by a special logic circuit or incorporated into the special logic circuit. Generally speaking, the computer will also include one or more large-capacity storage devices (for example, disks, magneto-optical disks or optical disks) for storing data, or operationally coupled to receive data from the one or more large-capacity storage devices, or to send data to the one or more large-capacity storage devices, or both receive and send. However, the computer does not have to have such devices. In addition, the computer can be embedded in another device, for example, the other device has a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver or a portable storage device (for example, a universal serial bus (USB) flash drive).

[0121] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example: semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0122] To support interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices may also be used to support interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form (including sound input, voice input, or tactile input). In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by interacting with the user by sending a web page to a web browser on the user's device in response to a request received from the web browser. In addition, the computer may interact with the user by sending a text message or other form of message to a personal device (e.g., a smartphone running a messaging application) and receiving a response message from the user as a reply.

[0123] The data processing apparatus for implementing a machine learning model may also include, for example, dedicated hardware accelerator units for processing general and computationally intensive portions of machine learning training or production (i.e., inference) workloads.

[0124] A machine learning model can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework).

[0125] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface, a web browser, or an application through which a user can interact with an implementation of the subject matter described in this specification), or includes any combination of one or more such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0126] A computing system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship between the client and the server is generated by computer programs that run on respective computers and have a client-server relationship with each other. In some embodiments, for example, for the purpose of displaying data to a user interacting with a device used as a client and receiving user input from the user, the server sends data such as an HTML page to the user device. Data generated at the user device, such as the result of a user interaction, may be received from the device on the server.

[0127] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or on the scope that may be claimed, but should be interpreted as descriptions of features that may be exclusive to a particular invention of a particular embodiment. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. On the contrary, the various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although the features may be described above as working in certain combinations, and even initially claimed as such, in some cases, one or more features in the claimed combination may be eliminated from the combination, and the claimed combination may point to a variant of a sub-combination or a sub-combination.

[0128] Similarly, although operations are depicted in the drawings and described in the claims in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0129] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some cases, multitasking and parallel processing can be advantageous.

Claims

1. A method performed by one or more data processing devices, the method comprising: Obtain training data comprising a plurality of training examples, wherein each training example comprises: an image pair, comprising a first image and a second image; and Select data indicating one or more of: (i) a common click-through rate for an image pair, and (ii) a similar image click-through rate for an image pair, wherein: the co-click rate of the image pair characterizing a frequency with which users select both the first image and the second image in response to both being simultaneously identified by search results of the search query; and The similar image click rate of the image pair characterizes the frequency with which a user selects the first image or the second image in response to the first image or the second image being identified by a search result of a search query that includes the second image or the first image, respectively; and Using the training data, an image embedding model having a plurality of image embedding model parameters is trained, wherein, for each of the plurality of training examples, the training comprises: Processing a first image of the training example using the image embedding model to generate an embedding of the first image; processing a second image of the training example using the image embedding model to generate an embedding for the second image; determining a similarity measure between the embedding of the first image and the embedding of the second image; and Adjusting image embedding model parameters based at least in part on: (i) a similarity measure between an embedding of a first image and an embedding of a second image, and (ii) selection data of training examples, wherein adjusting the image embedding model parameters comprises determining a gradient of a loss function, and wherein the loss function comprises a scaling factor determined as a multiple of a linear combination of a common click-through rate of the training examples and a similar image click-through rate.

2. The method according to claim 1, wherein the training data is generated using historical query logs of a web search system.

3. A method according to claim 1 or 2, wherein the co-click rate of each training example indicates the fraction of times that users select both the first image and the second image of the training example in response to both the first image and the second image of the training example being simultaneously identified by the search results of the search query.

4. A method according to claim 1 or 2, wherein the similar image click-through rate of each training example indicates the fraction of times a user selects the first image or the second image in response to the first image or the second image being identified by a search result of a search query that includes the second image or the first image, respectively.

5. The method of claim 1 or 2, wherein the image embedding model comprises a convolutional neural network.

6. The method according to claim 5, wherein: The loss function depends on: (i) a similarity measure between the embedding of the first image and the embedding of the second image, and (ii) the selection data of the training examples, and where, Adjusting image embedding model parameters also includes using the gradient to adjust image embedding model parameters.

7. The method of claim 1 or 2, wherein determining a similarity measure between an embedding of a first image and an embedding of a second image comprises: Determine a Euclidean distance between the embedding of the first image and the embedding of the second image.

8. A training system comprising: one or more computers; as well as One or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising: Obtain training data comprising a plurality of training examples, wherein each training example comprises: an image pair, comprising a first image and a second image; and Select data indicating one or more of: (i) a common click-through rate for an image pair, and (ii) a similar image click-through rate for an image pair, wherein: the co-click rate of the image pair characterizing a frequency with which users select both the first image and the second image in response to both being simultaneously identified by search results of the search query; and The similar image click rate of the image pair characterizes the frequency with which a user selects the first image or the second image in response to the first image or the second image being identified by a search result of a search query that includes the second image or the first image, respectively; and Using the training data, an image embedding model having a plurality of image embedding model parameters is trained, wherein, for each of the plurality of training examples, the training comprises: Processing a first image of the training example using the image embedding model to generate an embedding of the first image; processing a second image of the training example using the image embedding model to generate an embedding for the second image; determining a similarity measure between the embedding of the first image and the embedding of the second image; and Adjusting image embedding model parameters based at least in part on: (i) a similarity measure between an embedding of a first image and an embedding of a second image, and (ii) selection data of training examples, wherein adjusting the image embedding model parameters comprises determining a gradient of a loss function, and wherein the loss function comprises a scaling factor determined as a multiple of a linear combination of a common click-through rate of the training examples and a similar image click-through rate.

9. The system of claim 8, wherein the training data is generated using historical query logs of a web search system.

10. A system according to claim 8 or 9, wherein the co-click rate of each training example indicates the fraction of times that users selected both the first image and the second image of the training example in response to both the first image and the second image of the training example being simultaneously identified by the search results of the search query.

11. A system according to claim 8 or 9, wherein the similar image click-through rate of each training example indicates a fraction of the number of times a user selected the first image or the second image in response to the first image or the second image being identified by a search result of a search query that includes the second image or the first image, respectively.

12. The system of claim 8 or 9, wherein the image embedding model comprises a convolutional neural network.

13. The system according to claim 12, wherein: The loss function depends on: (i) a similarity measure between the embedding of the first image and the embedding of the second image; and (ii) a selection of training examples, and wherein, Adjusting image embedding model parameters also includes using the gradient to adjust image embedding model parameters.

14. One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: Obtain training data comprising a plurality of training examples, wherein each training example comprises: An image pair, comprising a first image and a second image; as well as Select data indicating one or more of: (i) a common click-through rate for an image pair, and (ii) a similar image click-through rate for an image pair, wherein: the co-click rate of the image pair characterizing a frequency with which users select both the first image and the second image in response to both being simultaneously identified by search results of the search query; and The similar image click rate of the image pair characterizes the frequency with which a user selects the first image or the second image in response to the first image or the second image being identified by a search result of a search query that includes the second image or the first image, respectively; and Using the training data, an image embedding model having a plurality of image embedding model parameters is trained, wherein, for each of the plurality of training examples, the training comprises: Processing a first image of the training example using the image embedding model to generate an embedding of the first image; processing a second image of the training example using the image embedding model to generate an embedding for the second image; determining a similarity measure between the embedding of the first image and the embedding of the second image; and Adjusting image embedding model parameters based at least in part on: (i) a similarity measure between an embedding of a first image and an embedding of a second image, and (ii) selection data of training examples, wherein adjusting the image embedding model parameters comprises determining a gradient of a loss function, and wherein the loss function comprises a scaling factor determined as a multiple of a linear combination of a common click-through rate of the training examples and a similar image click-through rate.

15. The non-transitory computer storage medium of claim 14, wherein the training data is generated using historical query logs of a web search system.

16. A non-transitory computer storage medium according to claim 14 or 15, wherein the co-click rate of each training example indicates the fraction of times that users selected both the first image and the second image of the training example in response to both the first image and the second image of the training example being simultaneously identified by the search results of the search query.

17. A non-transitory computer storage medium according to claim 14 or 15, wherein the similar image click-through rate of each training example indicates a fraction of the number of times a user selected the first image or the second image in response to the first image or the second image being identified by a search result of a search query that includes the second image or the first image, respectively.

18. The non-transitory computer storage medium of claim 14 or 15, wherein the image embedding model comprises a convolutional neural network.

Images