Embedding-based image search and retrieval

Abstract

Methods, systems, and apparatus, including a computer program encoded on a computer storage medium, are provided for retrieving image search results using an embedding neural network model. In one aspect, an image search query is received. A corresponding pairwise numerical embedding is determined for each of a plurality of image-landing page pairs. Each pairwise numerical embedding is a numerical representation in an embedding space. An image search query embedding neural network processes features of the image search query and generates a query numerical embedding. The query numerical embedding is a numerical representation of the image search query in the same embedding space. A subset of image-landing page pairs having pairwise numerical embeddings in the embedding space that are closest to the query numerical embedding of the image search query are identified as first candidate image search results.

Description

Technical Field

[0001] The inventive concepts of the present application relate to retrieving image search results, and more particularly, to techniques for retrieving image search results in response to image search queries using a trained embedding neural network model. Background Art

[0002] This description generally relates to retrieving image search results.

[0003] An online search engine typically retrieves candidate resources (e.g., images) in response to a received search query to present search results that identify resources that are responsive to the search query. The search engine typically retrieves search results using a term-based retrieval system that identifies search results based on keywords in the search query. The search engine may retrieve resources based on a variety of factors.

[0004] Some conventional image search engines, i.e., search engines configured to identify images on landing pages (e.g., on web pages on the Internet), generate separate signals from i) characteristics of the image and ii) characteristics of the landing page in response to a received search query, and then combine the separate signals according to a fixed weighting scheme that is the same for each received search query. Summary of the Invention

[0005] This specification describes techniques for retrieving image search results in response to an image search query using a trained embedding neural network model.

[0006] In one aspect, an image search query is received; a corresponding pairwise numerical embedding is determined for each of a plurality of image-landing page pairs, each image-landing page pair comprising a corresponding image and a corresponding landing page for the corresponding image, wherein each pairwise numerical embedding is a numerical representation in an embedding space; features of the image search query are processed using an image search query embedding neural network to generate a query numerical embedding for the image search query, wherein the query numerical embedding is a numerical representation in the same embedding space; and image search results identifying a subset of image-landing page pairs having a pairwise numerical embedding in the embedding space that is closest to the query numerical embedding for the image search query are identified as first candidate image search results for the image search query. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each computer storage device being configured to perform the actions of the method. For a system consisting of one or more computers, being configured to perform a particular operation or action means that the system has software, firmware, hardware, or a combination thereof installed thereon that, in operation, causes the system to perform the operation or action. By one or more computer programs being configured to perform certain operations or actions, it is meant that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform those operations or actions.

[0007] The subject matter described herein can be implemented in certain embodiments to achieve one or more of the following advantages. As described herein, image search query pairs are retrieved by evaluating proximity in an embedding space defined by embeddings generated by a trained embedding neural network model, allowing images to be provided in response to an image search query, which is an example of a search query. That is, the images provided in response to the image search query are responsive to the image search query. Unlike traditional methods for retrieving resources, the embedding neural network model receives a single input comprising an image search query, a landing page, and features of the image identified by a given image search result, and generates an embedding representation of the image search result in the same embedding space as the generated embedding representation of the received query. This embedding representation can model more general semantic relationships between features. Therefore, distances in the embedding space reflect the similarity of one point to another, and any query or search result can be represented as a point in this embedding space. This can enable efficient retrieval of relevant image search results. Retrieval in the embedding space is computationally efficient because fast algorithms can be developed to efficiently find nearest neighbors or approximate nearest neighbors in the embedding space. In some embodiments, distances in the embedding space can be used for ranking. For example, given a query and a set of image-landing page pairs, the image-landing page pairs can be sorted and ranked by their corresponding distances in the embedding space. Furthermore, by leveraging an embedding-based retrieval system alongside a term-based retrieval system, the system can retrieve relevant candidate search results that do not exactly match all the terms of the search query, which is beneficial for long or ambiguous search queries.

[0008] Having query and image-landing page pairs in the same embedding space enables features needed to identify relationships between different queries and different landing pages. For example, features can include one or more of the following: obtaining related queries based on the query, obtaining related documents based on the document, obtaining related queries based on the document, or obtaining related documents based on the query. These features can be powered by the same embedding neural network model, eliminating the need for separate indexing and retrieval systems used in traditional approaches.

[0009] In some implementations, embedding spaces for queries and image-landing page pairs in different languages ​​can be learned simultaneously. Distances in the embedding space can be used to connect landing pages with similar content in different languages. Distances in the embedding space can be used to understand that queries in different languages ​​have similar content. These connections provided by images can be captured by an embedding neural network model. The same or similar images may appear on landing pages in different languages. The embedding neural network model can leverage this language-independent similarity in the embedding space to help identify connections.

[0010] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1A is a block diagram of an example search system.

[0012] Figure 1B An example of identifying image-landing page pairs as candidate image search results for an image search query is shown.

[0013] Figure 2 An example architecture of an embedded neural network for generating candidate image search results from image-landing page pairs and image search queries is shown.

[0014] Figure 3 is a flow chart of an example process for generating image search results from an image search query.

[0015] Figure 4 is a flowchart of an example process for training an embedding neural network.

[0016] The same reference numbers and names in different drawings indicate the same elements. DETAILED DESCRIPTION

[0017] Figure 1A Shown is an example image search system 114. Image search system 114 is an example of an information retrieval system in which the systems, components, and techniques described below may be implemented.

[0018] User 102 can interact with image search system 114 through user device 104. For example, user device 104 can be a computer coupled to image search system 114 via a data communication network 112 (e.g., a local area network (LAN) or a wide area network (WAN), such as the Internet, or a combination of networks). In some cases, image search system 114 can be implemented on user device 104, for example, if the user installs an application that performs searches on user device 104. User device 104 will typically include memory (e.g., random access memory (RAM) 106) for storing instructions and data and a processor 108 for executing the stored instructions. The memory can include both read-only and writeable memory.

[0019] The image search system 114 is configured to search a collection of images. Typically, the images in the collection are images found on web pages on the Internet or a private network (e.g., an intranet). The web page on which the image is found (i.e., the web page containing the image) will be referred to as the image's landing page in this specification.

[0020] A user 102 may submit a search query 110 to an image search system 114 using a user device 104. When the user 102 submits the search query 110, the search query 110 is sent to the image search system 114 over a network 112.

[0021] When the image search system 114 receives the search query 110, the search engine 130 within the image search system 114 identifies image-landing page pairs that satisfy the search query 110 and responds to the query 110 by generating search results 128, each of which identifies a corresponding image-landing page pair that satisfies the search query 110. Each image-landing page pair includes an image and the landing page on which the image is found. For example, the image search results may include a lower-resolution version of the image or a crop of the image and data identifying the landing page, such as a resource locator for the landing page, a title for the landing page, or other identifying information. The image search system 114 transmits the search results 128 to the user device 104 via the network 112 for presentation to the user 102, i.e., in a form that can be presented to the user 102.

[0022] Search engine 130 may include indexing engine 132, ranking engine 134, and retrieval engine 135. Indexing engine 132 indexes image-landing page pairs and adds the indexed image-landing page pairs to index database 122. That is, index database 122 includes data identifying images and corresponding landing pages for each image.

[0023] Index database 122 also associates the image-landing page pair with (i) features of the image (ie, features that characterize the image) and (ii) features of the landing page (ie, features that characterize the landing page). Examples of image and landing page features are described in more detail below.

[0024] The retrieval engine 135 identifies candidate image-landing page pairs for the search query 110. The candidate image-landing page pairs comprise a subset of the available image-landing page pairs, ie, a subset of the pairs identified in the index database 122.

[0025] Specifically, as part of identifying candidate image search results, the retrieval engine 135 can map the search query 110 and each of the image-landing page pairs to the same embedding space using the trained embedding neural network model 136. The distance between the embedding of the image-landing page pair and the embedding of the search query 110 in the embedding space can reflect the relevance of the image-landing page pair to the search query 110. The retrieval engine 135 identifies the subset of available image-landing page pairs in the embedding space that are closest to the search query as candidate image search results. The candidate image search results can then be ranked by the ranking engine 134.

[0026] For each image-landing page pair, the search engine 135 determines a pairwise numerical embedding that is a numerical representation of the image-landing page pair in the embedding space. In some embodiments, the system may access an index database 122 that associates image-landing page pairs with corresponding previously generated pairwise numerical embeddings. In some other embodiments, the system may use a trained embedding neural network to process the features of each image-landing page pair to generate a corresponding pairwise numerical embedding for the image-landing page pair at query time.

[0027] In some embodiments, the search engine 135 may include two or more search systems that each generate a set of candidate image-landing page pairs. For example, in addition to the embedding-based search system discussed above, the search engine 135 may include a term-based search system that identifies image-landing page pairs based on keywords. The search engine 135 may combine the search results from the embedding-based search system with the search results from the term-based search system to generate a final set of candidate image-landing page pairs. By utilizing the embedding-based search system in addition to the term-based search system, the search engine 135 can retrieve relevant results that do not exactly match all terms of the query. This advantage is useful for long or ambiguous queries.

[0028] The ranking engine 134 generates corresponding ranking scores for the candidate image-landing page pairs. The ranking engine 134 may generate relevance scores based on scores stored in the index database 122 or relevance scores calculated at query time, and then ranks the candidate image-landing page pairs based on the corresponding ranking scores. The relevance score for a given image-landing page pair reflects the relevance of the image-landing page pair to the received search query 110, the quality of the given image-landing page pair, or both.

[0029] Embedded neural network model 136 can be any of a variety of embedded neural network models. For example, embedded neural network model 1360 can be a deep machine learning model, such as a neural network including multiple layers of nonlinear operations.

[0030] Will be referenced below Figure 2 and Figure 3 Describe in more detail the use of an embedding neural network model to retrieve candidate image-landing page pairs.

[0031] In order to train the embedding neural network model 136 so that the embedding neural network model 136 can be used to accurately generate embedding representations of image-landing page pairs and search queries in the embedding space, the image search system 114 includes a training engine 160. The training engine 160 trains the embedding neural network model 136 on training data that is generated using image-landing page pairs that have been associated with ground truth or known search queries. Figure 4 Describes training machine learning models in more detail.

[0032] Figure 1B An example of identifying an image-landing page pair as a candidate image search result for an image search query is shown. Figure 1B In the example of FIG, a user submits an image search query 170 ("coniferous trees"). The system generates image query features 172 based on the image search query 170 submitted by the user. Examples of query features 172 are referenced below. Figure 2 Provide a description.

[0033] The system also generates or obtains landing page features 174 for the landing page that is part of a particular image-landing page pair identified in the index database and image features 176 for the image in the particular image-landing page pair. Examples of landing page features 174 and image features 176 are provided below with reference to Figure 2 The system then provides the landing page features 174 and the image features 176 as input to the pairwise embedding neural network 178. The system also provides the query features 172 as input to the image search query embedding neural network 180.

[0034] The pairwise embedding neural network 178 receives input including features of the landing page and features of the image and generates a pairwise numerical embedding of the image-landing page pair. The pairwise numerical embedding is a numerical representation of the image-landing page pair in the embedding space.

[0035] The image search query embedding neural network 180 receives input including features of an image search query and generates a query numerical embedding for the image search query. The query numerical embedding is a numerical representation of the image search query in the same embedding space as the numerical embedding of the image-landing page pair.

[0036] The system then determines 186 whether the paired numerical embedding 182 is sufficiently close to the query numerical embedding 184 in the embedding space. For example, the system can identify K candidate image-landing page pairs from the image-landing page pairs in the index that have paired numerical embeddings that are closest to the query numerical embedding. If the system determines that the paired numerical embedding 182 is sufficiently close to the query numerical embedding 184, the system identifies 188 the image-landing page pair as a candidate image search result. The candidate image search result can be later processed by the ranking engine 134.

[0037] Figure 2 An example architecture of an embedding neural network 200 for generating candidate image search results from image-landing page pairs and image search queries is shown. For each image-landing page pair and image search query, the embedding neural network 200 takes query features 202, image features 206, and landing page features 208 as input and can generate output that helps the system identify whether the image-landing page pair is a candidate image search result. The embedding neural network 200 includes two sub-neural networks: an image search query embedding neural network 204 and a pairwise embedding neural network 210.

[0038] The image search query embedding neural network 204 takes the query features 202 as input and generates a query numerical embedding representation 184 of the search query. The query features 202 may include multiple features, such as location features, text features, etc. The location feature may represent the location where the image search query was submitted. The text feature may include a single word or two words of the image search query.

[0039] Typically, the image search query embedding neural network 204 can be a deep neural network that includes multiple embedding subnetworks for each of the multiple query features. Each embedding subnetwork can generate an embedding representation of an example of the corresponding feature. For example, the position embedding subnetwork can generate an embedding representation of the position feature, and the text embedding subnetwork can generate an embedding representation of the query single word or bigram. For example, a single word or bigram in the text feature can be represented as a separate token. The embedding of the single word or bigram can be calculated using a lookup table. The lookup table can be an embedding weight matrix and can be a shortcut for matrix multiplication to improve efficiency. The lookup table can be trained similarly to the parameters in the training weight matrix. The output of the lookup table can be a one-dimensional integer vector. For example, the word "cat" can be represented as token 543. Then the embedding of the word "cat" can be the value in row 543 of the lookup table, for example, a vector of embedding dimension or length 5 [1 4 6 7 9]. After computing the embedding of each token, the numerical embedding representation of the text feature can be the average of all token embeddings.

[0040] The output of each embedding sub-network can be a numerical vector. For example, the numerical vector can be a vector of length 128 with floating point numbers.

[0041] Each embedding subnetwork is pre-trained to generate embedding vectors for a specific type of query feature. The trained subnetwork can map different query features of a specific type into a common space. For example, the text embedding subnetwork can map different types of query text into a common space by generating corresponding embedding vectors. A query text [red hat] can be mapped to a numeric vector [0.1, -0.2, 0.0, …, -0.3, 0.2], which is a vector of length 128. These embeddings can model more general semantic relationships and can be effectively used in image search systems.

[0042] The output of each embedding sub-network is merged together through operations such as concatenation or addition to generate an embedding representation of the image search query. For example, assuming that the output of the position embedding sub-network is a vector of length 128, and the output of the text embedding sub-network is also a vector of length 128, these outputs can be concatenated together and a vector of length 256 can be generated that summarizes the embedding representation of the text features and position features of the image search query.

[0043] In some implementations, the merged features are processed through one or more fully connected layers that further extract features from the merged features to generate a final query value embedding 184 for the image search query.

[0044] The pairwise embedding neural network 210 takes as input the image features 206 and the landing page features 208 and generates a pairwise numerical embedding 182 for the image-landing page pair. The image features 206 and the landing page features 208 may come from the index database 122 or from other data maintained by the system that associates images and landing pages with corresponding features.

[0045] Image features 206 may include one or more of pixel data of an image or an embedding of an image representing the content of the image. For example, an image feature may include all or a portion of the pixels of an image that can represent the original content information of the image. As another example, image features 206 may include embedding vectors representing the content of the image. These embedding vectors representing the image may be obtained by processing the image through another embedding neural network. Alternatively, the embedding vectors may be generated by other image processing techniques for feature extraction. Example feature extraction techniques include edge, corner, ridge, and blob detection.

[0046] In some embodiments, the embedding vector of the image content can be pre-generated and stored in an index database. Therefore, the embedding representation of the image content can be directly obtained by accessing the index database without having to calculate it in the embedding neural network 200.

[0047] Image features 206 may also include data identifying the domain of the image, and / or text from the image's Uniform Resource Locator (URL), such as a single word or a double word. The text features of the image and the text features from the search query both include single words or double words. Therefore, they can both be later mapped to the same embedding space by the embedding neural network 200. The corresponding embedding representations of relevant text features are closer to each other in the embedding space than the corresponding embedding representations of less relevant or irrelevant text features.

[0048] Landing page features 208 may include one or more of text from the title of the landing page, significant terms that appear on the landing page, text from the URL of the landing page, and data identifying the domain of the landing page. In addition, examples of features extracted from the landing page may include the date the page was first crawled or updated, data characterizing the author of the landing page, the language of the landing page, keywords representing the content of the landing page, features of links to images and landing pages (such as the anchor text or source page of the link), features describing the context of the image in the landing page, etc.

[0049] The landing page features 208 may also include features extracted from the landing page that describe the context of the image in the landing page. Examples of features extracted from the landing page that describe the context of the image in the landing page include data characterizing the location of the image in the landing page, the prominence of the image on the landing page, a textual description of the image on the landing page, and the like. The location of the image in the landing page can be accurately located using pixel-based geometric positions in the horizontal and vertical dimensions, lengths in the horizontal and vertical dimensions based on the user device (e.g., in inches), identifiers similar to XPATH based on HTML / XML DOM, selectors based on CSS, and the like. The relative size of the image displayed on a general device and a specific user device can be used to measure the prominence of the image on the landing page. The textual description of the image on the landing page can include an alternative text label for the image, text surrounding the image, and the like.

[0050] Similar to the image search query embedding neural network 204, the pairwise embedding neural network 210 can be a deep neural network that includes multiple embedding sub-networks for each of the multiple image-landing page pair features. Each embedding sub-network can generate an embedding representation of an example of the corresponding feature. For example, the domain embedding sub-network can generate an embedding representation of the page domain features, and the text embedding sub-network can generate an embedding of the text data of the image URL. The output of each embedding sub-network can be a numeric vector. For example, the numeric vector can be a vector of length 128 containing floating-point numbers.

[0051] Similar to the image search query embedding neural network 204, the outputs of each embedding sub-network are combined through operations such as concatenation or addition to generate an embedding representation for the image-landing page pair. For example, the outputs from multiple embedding sub-networks may include multiple embedding vectors of length 128 for the page title single word / double word, page salient terms, page URL single word / double word, image URL single word / double word, and image domain. In some embodiments, a 128-bit embedding vector for the image content may be obtained from an index database. Multiple N-bit embedding vectors may be concatenated together to generate a vector of length 128xN that summarizes the features of the image-landing page pair. Similar to the image search query embedding neural network 204, in some embodiments, the combined features are processed through one or more fully connected layers, which further extract features from the combined features to generate a final pairwise numerical embedding 182 for the image-landing page pair. The pairwise numerical embedding 182 and the query numerical embedding 184 are in the same embedding space.

[0052] In some embodiments, the outputs of the embedding subnetworks may be partially merged or unmerged. Instead of merging the outputs of the embedding subnetworks and generating a single embedding representation for an image-landing page pair, the outputs of the embedding subnetworks may be merged into two or more embedding representations for the image-landing page pair. Thus, the corresponding final pairwise numerical embedding 182 may include two or more embedding representations in the same embedding space as the query numerical embedding 184.

[0053] In some embodiments, the image search query embedding neural network 204 and the pairwise embedding neural network 210 share at least some parameters. For example, two or more of the subnetworks (such as the query text embedding subnetwork, the landing page title embedding subnetwork, the landing page significant term embedding subnetwork, the landing page URL embedding subnetwork, and the image URL embedding subnetwork) can share parameters because these features are extracted from the same vocabulary. Sharing parameters between two neural networks means that the two neural networks are constrained to have the same value for each shared parameter.

[0054] In some embodiments, the image search query embedding neural network 204 and the pairwise embedding neural network 210 may be jointly trained to facilitate training of shared parameters between these networks. More details on training the embedding neural network will be referred to below. Figure 4 Describe in more detail.

[0055] The prediction layer 212 compares the paired numerical embedding 182 with the query numerical embedding 184 in the same embedding space. In some embodiments, the prediction layer 212 can output a distance value that can measure the closeness of the paired numerical embedding 182 and the query numerical embedding 184. For example, the prediction layer 212 can include a dot product between the paired numerical embedding 182 and the query numerical embedding 184.

[0056] The output from the prediction layer 212 can be used differently during training of the embedding neural network 200 and during image search. During image search, the retrieval engine 135 can identify candidate image search results for a search query based on the output from the prediction layer 212, which measures the closeness of the embedding representation of the image-landing page pair to the embedding representation of the search query. When training the embedding neural network 200, the training engine 160 can jointly train the pairwise embedding neural network and the image search query embedding neural network to minimize a loss function (e.g., a dot product) that depends on the output from the prediction layer 212.

[0057] Figure 3 is a flow chart of an example process 300 for generating image search results from an image search query. For convenience, process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, an image search system (e.g., Figure 1A The image search system 114) can perform process 300.

[0058] An image search system receives an image search query from a user device (302). In some cases, the image search query is submitted through a dedicated image search interface provided by the image search system (i.e., a user interface for submitting image search queries). In other cases, the search query is submitted through a general Internet search interface, and image search results are displayed in response to the image search query along with other types of search results, i.e., search results identifying other types of content available on the Internet.

[0059] Upon receiving an image search query, the image search system identifies an initial image-landing page pair 304. For example, the system can identify the initial image-landing page pair from among the image pairs indexed in the search engine index database based on a signal measuring the quality of the image pair, the relevance of the image pair to the search query, or both.

[0060] For each image-landing page pair, the system determines a corresponding pairwise numerical embedding (306) that is a numerical representation of the image-landing page pair in the embedding space. In some embodiments, the system can access an index database that associates image-landing page pairs with corresponding pairwise numerical embeddings that have been previously generated using a pairwise embedding neural network. This can save image search time because the pairwise numerical embeddings have been previously calculated and stored.

[0061] In some other embodiments, the system can process the features of each image-landing page pair using a pairwise embedding neural network to generate a corresponding pairwise numerical embedding for the image-landing page pair. The features of each image-landing page pair can include features of the image and features of the landing page. These features can come from an index database or other data maintained by the system that associates images and landing pages with corresponding features. These features can be represented categorically or discretely. In addition, additional related features can be created from pre-existing features. For example, the system can create a relationship between one or more features through a combination of addition, multiplication, or other mathematical operations.

[0062] The system obtains features of the image search query (308) and processes the features of the image search query using an image search query embedding neural network (310). The image search query embedding neural network can generate a query numerical embedding for the image search query. The generated query numerical embedding is a numerical representation of the image search query in the same embedding space as the pairwise numerical representation of the image-landing page pair.

[0063] The system identifies a subset of the initial image-landing page pairs as first candidate image search results (312). The subset of the initial image-landing page pairs has pairwise numerical embeddings in the embedding space that are closest to the query numerical embedding of the image search query. For example, a nearest neighbor search can be used to select the top K image-landing page pairs from the initial image-landing page pairs that have embedding representations that are closest to the embedding representation of the search query.

[0064] Feature embeddings can model more general semantic relationships between features. The proximity of the numerical embeddings of features can be trained to measure the relevance of a candidate image search result to an image search query. In some embodiments, the proximity of the numerical embeddings can be trained to measure the likelihood that a user submitting a search query will interact with the search result. Numerical embeddings that are closer to each other indicate that the user submitting the search query will find the candidate image search result more relevant and interact with it. Figure 4 Describes training an embedding neural network to generate numerical embeddings.

[0065] The first candidate image search results typically include far fewer candidates than the initial image search results. For example, the number of first candidate image search results may be limited to fewer than a hundred results. This is much fewer than the initial image search results, which may include thousands or millions of image search results.

[0066] In some embodiments, upon receiving the first candidate image search results, the system then generates a plurality of second candidate image search results that include at least some of the first candidate image search results. For example, the system may obtain additional candidate words retrieved by a keyword-based term-based search system. The system may merge the term-based candidates and the embedding-based candidates and send the merged candidates for a second round of relevance scoring. After the second round of relevance scoring, a second candidate image search result may be selected from the embedding-based first candidate image search results and the term-based candidate image search results.

[0067] The system ranks the plurality of second candidate image search results using a ranking engine (314). The ranking engine can generate relevance scores based on scores stored in an index database or scores calculated at query time, and ranks the plurality of second image-landing page pairs based on the corresponding ranking scores. The relevance scores of the candidate image-landing page pairs reflect the relevance of the image-landing page pairs to the received search query, the quality of the given image-landing page pairs, or both. The system ranks the image search results based on the relevance scores of the corresponding image-landing page pairs.

[0068] The system generates an image search results presentation (316) showing the image search results ordered according to the ranking, and provides the image search results presentation (318) for presentation by sending the search results presentation over a network to a user device from which the image search query was received in a form that can be presented to the user.

[0069] Figure 4 is a flow chart of an example process 400 for training an embedded neural network. For convenience, process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, an image search system (e.g., Figure 1A The image search system 114 of the system may perform process 400.

[0070] The system receives a set of training image search queries and, for each training image search query, receives training image search results for the query (402). Each training image search result can be identified as a positive training example or a negative training example. In some implementations, when a user interacts with a search result that identifies a training image-landing page pair after submitting a training image search query, the system identifies the training image search query and the training image-landing page pair as a positive training example.

[0071] For each of the training image search queries, the system generates training examples using features of the image search query (404). For each of the training image search results, the system generates training examples using features of the image-landing page pair (408). For each training pair, the system identifies (i) features of the image search query, (ii) features of the image, and (iii) features of the landing page. Extracting, generating, and selecting features can occur before training or using other embedding models. Examples of features are provided above with reference to Figure 2 Described.

[0072] The system trains a pairwise embedding neural network (410) and jointly trains an image query embedding neural network (406). The system jointly trains the two neural networks to minimize a loss function that depends on a dot product between (i) the query numerical embeddings of the training image search queries and (ii) the pairwise numerical embeddings of the training image-landing page pairs. For example, the loss function can encourage the dot product to be higher when the training image search query and the training image-landing page pair have been identified as positive training examples than when the training image search query and the training image-landing page pair have been identified as negative training examples.

[0073] In some embodiments, the image search query embedding neural network can be pre-trained for other embedding representation tasks. For example, the image search query embedding neural network can be implemented using a lookup table with pre-determined or trained parameters. The numerical representation of the training image search query can be calculated by indexing the lookup table using the token representation of the training image search query. In some embodiments, the pairwise embedding neural network can be pre-trained for other embedding representation tasks.

[0074] In some embodiments, the pairwise embedding neural network and the image search query embedding neural network can share at least some parameters. For example, the pairwise embedding neural network and the image search query embedding neural network can share parameters corresponding to any features extracted from the same vocabulary. The shared neural network parameters can be efficiently trained using the joint training method described above.

[0075] In some embodiments, the system can implement a loss function using any of a variety of loss functions available for training embedded neural network models to improve the efficient use of the large amount of data available. Examples of loss functions that can be used to train models include softmax with cross entropy loss, sampled softmax loss (Jean, Sébastien, et al. "On using very large target vocabulary for neural machine translation." arXiv preprint arXiv:1412.2007.2014), contrastive loss functions, or a combination of two or more thereof.

[0076] In some embodiments, the system can train the embedded neural network model in several stages, and the system can implement different types of loss functions at each stage of the training process. For example, the system can use a softmax loss function in the first stage and can use a contrastive loss function or an asymmetric scaled sigmoid loss function in subsequent stages. In some embodiments, in one or more stages after the first stage, difficult samples (e.g., training samples with large loss values ​​in one or more previous training stages) can be used during training to improve the convergence rate of the training process or improve the performance of the final model being trained.

[0077] For example, the system receives a set of 4096 training image search queries, and for each training image search query i , receive the image search result SelectedImage of the query i , i.e., the selected image-landing page pairs. Here, index i = 1, 2, ..., 4096. For each training image search query query i , the system generates a positive training example (query i ,SelectedImage i ) and generate 4095 negative training examples (query i ,SelectedImage j ), where i≠j. During training, for each positive or negative training example, the embedding neural network can output a dot product that can be normalized to the range [0, 1] by a softmax function. The system can then calculate each training image search query query using the normalized dot product calculated from its corresponding 4095 negative training examples and one positive training example. iBecause the number of training image search queries is very large, the sampled softmax loss only considers a subset of the training examples to calculate the loss, rather than calculating the softmax loss for all 4096 training image search queries. The total loss is the sum of the losses calculated for each of the 4096 training image search queries.

[0078] The system trains the embedded neural network by minimizing a loss function. For example, the system can train the embedded neural network model to determine training values ​​of weights of the neural network from initial values ​​of the weights by repeatedly performing the neural network training process to calculate the gradient of the loss function with respect to the weights (e.g., using backpropagation) and determining updates to the weights from the gradients (e.g., using an update rule corresponding to the neural network training process).

[0079] This specification uses the term "configured" in conjunction with system and computer program components. For a system consisting of one or more computers, being configured to perform a particular operation or action means that the system has installed thereon software, firmware, hardware, or a combination thereof that, in operation, causes the system to perform those operations or actions. For one or more computer programs being configured to perform a particular operation or action, it means that the one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0080] Embodiments of the subject matter and functional operations described in this specification may be implemented in digital electronic circuits, in tangibly implemented computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more thereof. Alternatively or additionally, program instructions may be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical, optical, or electromagnetic signal) that is generated to encode information for transmission to a suitable receiving device for execution by a data processing apparatus.

[0081] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, equipment, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. An apparatus may also be or include dedicated logic circuitry, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). In addition to hardware, an apparatus may optionally include code that creates an execution environment for a computer program, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these.

[0082] A computer program, which may also be referred to or described as a program, software, software application, app, module, software module, script or code, may be written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages); and it may be deployed in any form (including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment). A program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data, for example, in a markup language document, in a single file dedicated to the program, or in one or more scripts in multiple coordinated files (e.g., files storing one or more modules, subroutines or portions of code). A computer program may be deployed to be executed on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected by a data communications network.

[0083] In this specification, the term "database" is used broadly to refer to any collection of data: the data need not be structured in any particular way, or at all, and it may be stored on a storage device in one or more locations. Thus, for example, an index database may include multiple collections of data, each of which may be organized and accessed differently.

[0084] Similarly, throughout this specification, the term "engine" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, an engine will be implemented as one or more software modules or components installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a specific engine; in other cases, multiple engines may be installed and run on the same computer or computers.

[0085] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs, which perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, such as an FPGA or ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0086] A computer suitable for executing a computer program can be based on a general or special microprocessor or both, or any other type of central processing unit. Typically, a central processing unit will receive instructions and data from a read-only memory or random access memory or both. The basic elements of a computer are a central processing unit for executing or running instructions and one or more storage devices for storing instructions and data. The central processing unit and memory can be supplemented by or incorporated into a dedicated logic circuit. Typically, a computer will also include one or more large-capacity storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably coupled to receive data from them or transmit data to them, or both. However, a computer does not need to have such a device. In addition, a computer can be embedded in another device (e.g., to name a few examples, a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive).

[0087] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks.

[0088] To provide for user interaction, embodiments of the subject matter described in this specification can be implemented on a computer having a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide for user interaction; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including sound, voice, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's device in response to a request received from the web browser. In addition, a computer can interact with a user by sending text messages or other forms of messages to a personal device (e.g., a smartphone running a messaging application) and receiving response messages from the user in return.

[0089] The data processing apparatus for implementing machine learning models may also include, for example, dedicated hardware accelerator units for processing the common and computationally intensive parts of machine learning training or production, i.e., inference, workloads.

[0090] The machine learning model can be implemented and deployed using a machine learning framework (e.g., the TensorFlow framework, the Microsoft Cognitive Toolkit framework, the Apache Singa framework, or the Apache MXNet framework).

[0091] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0092] A computing system may include a client and a server. The client and the server are typically remote from each other and typically interact via a communication network. The relationship between the client and the server is generated by computer programs running on their respective computers and having a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device, for example, to display data to a user interacting with the device acting as a client and to receive user input from the user interacting with the device acting as a client. Data generated on the user device (e.g., the result of the user interaction) can be received from the device at the server.

[0093] Although this specification contains many specific implementation details, these should not be interpreted as limitations on the scope of any invention or on the scope of what is claimed, but rather as descriptions of specific features of specific embodiments of specific inventions. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented individually in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as working in certain combinations, and even initially claimed as such, in some cases, one or more features from a claimed combination may be deleted from that combination, and a claimed combination may be directed to a sub-combination or variation of a sub-combination.

[0094] Similarly, although operations are depicted in a particular order in the drawings and referenced in a particular order in the claims, this should not be construed as requiring that the operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0095] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. A method for image search query, comprising: receiving an image search query; determining a corresponding pairwise numerical embedding for each of a plurality of image-landing page pairs, each image-landing page pair comprising a corresponding image and a corresponding landing page for the corresponding image, wherein each pairwise numerical embedding is a numerical representation in an embedding space, wherein each pairwise numerical embedding is generated based on one or more image features of the corresponding image and one or more landing page features of the corresponding landing page, wherein the one or more landing page features describe a context of the corresponding image within the corresponding landing page; processing features of the image search query using an image search query embedding neural network to generate a query numerical embedding for the image search query, wherein the query numerical embedding is a numerical representation in the same embedding space; as well as Image search results identifying a subset of image-landing page pairs having a pairwise numerical embedding in the embedding space that is closest to a query numerical embedding of the image search query are identified as first candidate image search results for the image search query.

2. The method according to claim 1, further comprising: ranking a plurality of second candidate image search results including at least some of the first candidate image search results; generating an image search result presentation displaying second candidate image search results sorted according to the ranking; as well as The image search result presentation is provided for presentation on a user device.

3. The method of claim 2 , wherein determining a corresponding pairwise numerical embedding for each of a plurality of image-landing page pairs comprises: An index database is accessed that associates image-landing page pairs with corresponding pairwise numerical embeddings that have been generated for the image-landing page pairs using a pairwise embedding neural network.

4. The method of claim 2 , wherein determining a corresponding pairwise numerical embedding for each of a plurality of image-landing page pairs comprises: The features of each image-landing page pair are processed using a pairwise embedding neural network to generate a corresponding pairwise numerical embedding for the image-landing page pair.

5. The method according to any one of claims 3 or 4, wherein the pairwise embedding neural network and the image search query embedding neural network have been jointly trained.

6. The method of claim 5, wherein the pairwise embedding neural network and the image search query embedding neural network have been jointly trained to minimize a loss function that depends on the dot product between (i) query numerical embeddings of training image search queries and (ii) pairwise numerical embeddings of training image-landing page pairs.

7. The method of claim 6, wherein the loss function causes the dot product between (i) the query numerical embedding of the training image search query and (ii) the pairwise numerical embeddings of the training image-landing page pairs to be higher when the training image search query and the training image-landing page pairs have been identified as positive training examples than when the training image search query and the training image-landing page pairs have been identified as negative training examples.

8. The method according to claim 7, further comprising: When a user interacts with a search result identifying the training image-landing page pair after submitting the training image search query, the training image search query and the training image-landing page pair are identified as positive training examples.

9. The method of claim 8, wherein the pairwise embedding neural network and the image search query embedding neural network share at least some parameters.

10. The method of claim 9, wherein the pairwise embedding neural network and the image search query embedding neural network share parameters corresponding to two features extracted from the same vocabulary. The method of claim 10 , wherein the characteristics of the image search query include data characterizing a location from which the image search query was submitted.

12. The method of claim 10, wherein the characteristics of the image search query include text of the image search query.

13. The method of claim 10, wherein the characteristics of each image-landing page pair include a combination of characteristics of the landing page and characteristics of the image.

14. The method of claim 13, wherein the characteristics of the landing page include one or more of text from a title of the landing page, prominent terms appearing on the landing page, text from a URL of the landing page, or data identifying a domain of the landing page.

15. The method of claim 14, wherein the features of the image include one or more of pixel data of the image or an embedding of the image.

16. The method of claim 15, wherein the characteristics of the image include one or more of data identifying a domain of the image or text from a URL of the image.

17. A system for image search queries, comprising one or more computers and one or more storage devices storing instructions, which, when executed by the one or more computers, cause the one or more computers to perform corresponding operations of the method of any one of the preceding claims.

18. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform corresponding operations of the method of any one of claims 1-16.

Images