Methods for learning embedding spaces using cross-examples

Abstract

Techniques are disclosed that are capable of learning an embedding space using cross- examples, where distances between queries and electronic resources in the embedding space provide an indication of the relevance of the electronic resources to the queries. Various implementations include learning an embedding space using cross-example Softmax techniques. Various implementations include learning an embedding space using cross-example negative mining. Techniques are disclosed that are capable of determining additional or alternative electronic resources for a query based on comparing a query vector (e.g., an embedding space representation of the query) to a set of pre-stored candidate electronic resource vectors (e.g., embedding space representations of a set of candidate electronic resources).

Description

Background Technology

[0001] Information retrieval systems can rely on neural network models to learn embedding spaces, where distances encode the relevance between a given query and candidate responses to that query. These embedding spaces can be trained using conventional methods (e.g., sampling softmax, stochastic negative mining, etc.) by optimizing the relevance between the relative rankings of candidate responses for a given query. Summary of the Invention

[0002] The implementation described in this paper aims to learn the embedding space using cross-examples, so that the distances in the learned embedding space are globally calibrated across queries. Distances generated using conventional techniques may not be comparable across queries, which can make it difficult to determine, for example, how relevant a document is to a query based on distance. For example, in a conventionally learned embedding space (e.g., an embedding space learned using sampling softmax, random negative mining, etc.), a first distance between a first query and a document relevant to that first query may be greater than the distance between a second query and a document unrelated to that second query. In other words, using conventional techniques, it is impossible to compare the distance between a first query and its corresponding relevant document with the distance between a second query and its corresponding relevant document and / or (or multiple) unrelated documents. Furthermore, using conventional techniques, it may be possible to discover the closest query to a given document in terms of distance, but it may not be possible to discover how truly relevant a given document is to a query (e.g., whether it is a truly close match or merely a superficial match).

[0003] In contrast, the implementation described in this paper aims to learn an embedding space(s) using cross-examples, such that distances within the learned embedding space are globally calibrated. For example, the distance between a first query and its first corresponding document should not be greater than the distance between a second query and (multiple) documents(s) unrelated to that second query. In other words, it is possible to compare the distance between a first query and one or more candidate documents with the distance between a second query and one or more candidate documents. Furthermore, the distances between a given query and candidate documents are meaningful and reflect the true relevance of those candidate documents to the given query.

[0004] In some implementations, cross-examples can be used to train one or more neural network models to learn the embedding space. For example, an input model can be trained to generate a query vector by processing a query, where the query vector is an embedding space representation of the query. Additionally or alternatively, a resource model can be trained to generate an electronic resource vector by processing electronic resources (e.g., images, documents, web pages, bounded boxes, and / or (multiple) additional resources), where the electronic resource vector is an embedding space representation of the electronic resource.

[0005] In some implementations, batches of training data can include ground truth query / electronic resource pairs, where each query in the batch of training data has a single corresponding electronic resource, and each electronic resource has only a single corresponding query. In some implementations, an input model can be used to process each query to generate a corresponding query vector. Similarly, in some implementations, a resource model can be used to process each electronic resource to generate a corresponding electronic resource vector. For each query / electronic resource ground truth pair, a relevance score (e.g., distance in the embedding space) can be generated based on the corresponding query vector and the corresponding electronic resource vector. For example, a relevance score can be generated by determining the dot product between the corresponding query vector and the corresponding electronic resource vector. Additionally or alternatively, a negative relevance score (e.g., distance in the embedding space) can be determined for each given query and each electronic resource that is not paired with the ground truth of that given query, based on the corresponding query vector and the corresponding electronic resource vector. For example, a negative relevance score can be generated by determining the dot product between the corresponding query vector and the corresponding electronic resource vector that is not paired with the ground truth of a given query. In some implementations, pairwise similarity matrices can be generated based on queries in batches of training data and electronic resources in batches of training data.

[0006] In some implementations, a query loss can be determined for each query in a batch of training data. For example, each query loss can be based on one or more of the relevance score of the corresponding query and the negative relevance scores of at least one additional query in the batch of training data. In other words, each query loss can be based on the relevance score of the query and at least one negative relevance score of one or more cross-examples (i.e., one or more additional queries). In some implementations, a training batch loss can be determined for each query in the batch based on the query loss. This training batch loss can then be used to update (e.g., backpropagation) one or more parts of the input model, one or more parts of the resource model, and / or one or more parts of the additional models(s).

[0007] In some implementations, a cross-sample Softmax method can be used to generate each query loss, where each query loss is based on each of the relevance score corresponding to the query and each of the generated negative query losses. In other words, the query loss is based on each of the relevance score for the corresponding query, each of the negative relevance scores generated for the corresponding query, and each of the negative relevance scores generated for each additional query in the batch. In contrast, a regular Softmax method (e.g., regular sampling Softmax) can be used to determine the query loss based on the relevance score generated for the corresponding query and the negative relevance score generated for the corresponding query, rather than based on the negative relevance scores generated for additional queries in the batch of training data.

[0008] Additionally or alternatively, a cross-example negative mining approach can be used to generate the loss for each query, where each query loss is based on a subset of the relevance score generated for the corresponding query and the negative relevance scores generated for batches of training data (e.g., a subset of negative relevance scores generated for the query and / or additional queries). In some implementations, a subset of negative relevance scores can be selected based on whether the negative relevance scores satisfy one or more conditions. For example, a subset of negative relevance scores can be selected to include negative scores with the highest values ​​(e.g., the top k negative relevance scores with k highest values). In contrast, a query loss generated using a stochastic negative mining approach is based on the relevance score generated for the corresponding query and a subset of the negative relevance scores generated for the corresponding query, excluding negative relevance scores generated for additional queries.

[0009] In some implementations, a trained input model can be used to determine the corresponding electronic resource. For example, a query vector can be generated by processing a query using the trained input model. In some implementations, the query vector can be compared with pre-stored candidate electronic resource vectors, each of which was previously generated by processing a candidate electronic resource using a resource model. In some implementations, the input model and the resource model can be trained using the same batch loss(s). A candidate electronic resource vector can be selected based on this comparison. For example, a candidate electronic resource vector can be selected based on the minimum distance between the candidate electronic resource vector and the query vector. Additionally or alternatively, the electronic resource corresponding to the query can be determined based on the selected candidate electronic resource vector. In some implementations, the computational system can perform(s) actions(s) based on the determined electronic resource.

[0010] For example, a computing system can be used to determine an image (i.e., an electronic resource) in response to a natural language query (i.e., a query). An input model can process the natural language query to generate a query vector. A pre-stored candidate image vector can be generated by processing candidate images using a resource model. In the illustrated example, the input model used to process the natural language query can be a different model type and / or have a different model structure than the resource model used to process the candidate images. In some implementations, the input model and the resource model can be trained using the same generated training loss(s). In some of those implementations, the input model can be trained simultaneously with the resource model. A candidate image vector can be selected based on the distance between the selected candidate image vector and the query vector. In some implementations, the image corresponding to the natural language query can be determined based on the selected pre-stored candidate image vector. In some implementations, the computing system can perform (multiple) actions based on the determined image, such as displaying the image on the computing system's screen.

[0011] As an additional example, the computational system can be used to determine bounded boxes (i.e., electronic resources) of objects captured in an image (i.e., a query). It can process the image using an input model to generate image vectors. It can generate pre-stored candidate bounded box vectors by processing candidate bounded boxes using a response model. It can compare the image vector with each of the pre-stored candidate bounded box vectors and select a candidate bounded box vector based on the comparison. For example, it can select the candidate bounded box vector with the shortest distance to the image vector. It can determine the bounded box based on the candidate bounded box vectors. In some implementations, the computational system can perform multiple actions based on the determined bounded box, such as displaying the bounded box around an object in the image, identifying objects captured within the bounded box, etc.

[0012] Therefore, various implementations describe techniques for learning the embedding space using cross-examples (e.g., using negative relevance scores generated from additional queries in batches of training data). In contrast, conventional techniques learn the embedding space by excluding additional negative relevance scores generated for additional queries, based on the negative relevance scores generated for a given query. In some cases, the electronic resource most closely corresponding to the query (e.g., the electronic resource closest to the query in the embedding space learned using conventional techniques) is not particularly relevant to the query. Computational resources (e.g., processor cycles, memory, battery power, etc.) can be conserved by providing the user with electronic resources that respond only to the query, based on the distance between the query and the electronic resource in the embedding space learned using cross-examples.

[0013] The above description is provided only as a summary of some implementations disclosed herein. Detailed explanations of these and other implementations of the disclosed techniques are appended below.

[0014] It should be understood that all combinations of the foregoing concepts and the additional concepts described in more detail herein are contemplated as part of the subject matter disclosed herein. For example, all combinations of the claimed concepts appearing at the end of this disclosure are contemplated as part of the subject matter disclosed herein. Attached Figure Description

[0015] Figure 1 The diagram illustrates an example environment in which various implementations disclosed in this article can be carried out.

[0016] Figure 2A The illustration shows examples of generating relevance scores according to the various implementations disclosed in this article.

[0017] Figure 2B The illustration shows examples of generating negative correlation scores according to the various implementations disclosed in this article.

[0018] Figure 2C The illustration shows an example of generating batch loss based on (multiple) correlation scores and (multiple) negative correlation scores according to the various implementations disclosed in this paper.

[0019] Figure 3A and Figure 3B The illustration shows an example embedding space generated using conventional methods.

[0020] Figure 3C and Figure 3D The illustration shows an example embedding space generated according to the implementation method disclosed in this article.

[0021] Figure 4A The diagram illustrates example pairwise similarity matrices generated from batches of training data.

[0022] Figure 4B The illustration shows an example of pairing true values ​​for batches of training data.

[0023] Figure 4C The illustration shows an example of generating query loss using the standard Softmax method.

[0024] Figure 4D The illustration shows an example of generating query loss using a standard random negative mining method.

[0025] Figure 4E The illustration shows an example of generating query loss using the cross-example Softmax method according to the various implementations disclosed in this article.

[0026] Figure 4FThe illustration shows an example of generating query loss using the cross-example negative mining method according to the various implementations disclosed in this article.

[0027] Figure 5 This is a flowchart illustrating an example process of training an input model and / or a response model according to the various implementations disclosed in this document.

[0028] Figure 6 This is a flowchart illustrating an example process of generating query loss using the cross-example Softmax method according to the various implementations disclosed in this paper.

[0029] Figure 7 This is a flowchart illustrating an example process of generating query loss using the cross-example negative mining method according to the various implementations disclosed in this article.

[0030] Figure 8 This is a flowchart illustrating an example process for determining electronic resources based on various implementations disclosed in this document.

[0031] Figure 9 The diagram illustrates an example architecture for a computing device. Detailed Implementation

[0032] Modern image retrieval systems increasingly rely on deep neural networks to learn embedding spaces, where distances encode the relevance between a given query and an image. These embedding spaces can typically be trained by optimizing the relative ranking of documents for a given query. However, this presents a challenge because the resulting absolute distances may not be comparable across queries, making it difficult to determine document relevance solely based on the distance between a document and a query. The techniques disclosed in this paper address this challenge with a cross-example softmax method. In some implementations, in each iteration, the proposed cross-example softmax loss encourages all queries to be closer to their matching images, rather than all queries being closer to all irrelevant images. This leads to a globally more calibrated similarity metric and makes distance more interpretable as a measure of relevance. Additional or alternative techniques focus on cross-example negative mining methods, where each query-document pair is compared against the “hardest” negative comparison across the entire batch. In some implementations, it can be shown that the proposed method effectively improves global calibration and / or enhances retrieval performance.

[0033] The goal of large-scale information retrieval is to efficiently find relevant documents for a given query from billions of potential candidates. A typical example is image search: given a text query, find relevant images. A common implementation could be to learn a real-valued scoring function to rank the set of candidate images that each query might involve. Since computing the relevance of each query-image pair using large neural networks can be extremely expensive, recent deep learning systems have addressed this task by embedding both the query and the images into a shared vector space. By encoding semantic relevance as distances in the vector space, these systems can model complex semantic relationships while still enabling efficient retrieval using approximate nearest neighbor search over a large database of images.

[0034] Conventional deep metric learning methods rely on pairwise or triplet comparisons to position queries close to relevant images and far away from irrelevant ones in the vector space. However, inspired by the success of large-scale classification problems like ImageNet, many deep metric learning models are now able to use Softmax with cross-entropy. Note that, similar to information retrieval systems, models like ResNet or VGG maximize the dot product between the image query representation and a single relevant label from a fixed set of 1,000 labels. The labels can be encoded in the weight matrix of the final layer. Compared to ImageNet training, computing the Softmax likelihood for all documents, potentially billions, in information retrieval systems can be nearly infeasible; alternatively, Softmax can be computed on a small, random subset of documents. This is often referred to as regular sampling Softmax. In practice, this can be achieved by embedding random batches of known-related pairs of queries / documents into their vector representations. For a given query, other documents in the batch can be efficiently reused as negative examples. Since most of these random documents are unlikely to be informative for each query, random negative mining is possible, allowing the use of the most informative negative documents, such as those with the highest similarity scores.

[0035] While methods based on sampling softmax and triples have shown great effectiveness in learning representations that capture relative semantic similarity, key challenges remain. Specifically, these methods are invariant to absolute distances in the vector space because they only optimize the relative distance between a query and its matching documents, compared to irrelevant documents. Therefore, distances cannot be compared across queries and cannot be interpreted as an absolute measure of relevance. Figure 3A and Figure 3BAs illustrated, the relevant image 306 for query A302 is farther from the query than the irrelevant images 316 and 318 are farther from query B312. This lack of calibration can be problematic for retrieval systems that typically use a global confidence threshold to determine which distance results are considered relevant.

[0036] In some implementations, the cross-example softmax method addresses this challenge by directly optimizing retrieval and similarity score calibration, allowing query / document similarity scores to be compared across multiple queries. In other implementations, the cross-example softmax method extends softmax by introducing cross-example comparisons. In some implementations, instead of maximizing the ratio between the distance to a query's matching document and the distance to all other documents, cross-example softmax maximizes the ratio between the distance to a query's matching document and the distance to all unrelated query / document pairs. This encourages any matching pair to be closer in the vector space than any non-matching pair. Figure 3C and Figure 3D The illustration shows the effect of Softmax on the cross-cutting example and how it leads to the calibrated distance score.

[0037] In some implementations, the proposed method further allows the concept of random negative mining to be extended to cross-example negative mining. Instead of mining only the most informative negative documents for a given query, it is possible to select non-matching pairs based on the highest similarity score, even if they are for different queries.

[0038] Metric learning using deep models has been applied to many applications, especially those with very large output spaces. Early methods were based on Siamese networks with contrastive losses on pairwise or relative triplet similarity comparisons. Inspired by the success of large-scale classification tasks on ImageNet, recent models have been trained using sampling softmax loss with cross-entropy. Recently, some works have proposed modifications to the sampling softmax loss by normalization, adding margin, and adjusting the scaling temperature. These methods focus on optimizing the relative ranking of labels given an anchor query. In contrast, the cross-example softmax method disclosed in this paper is able to optimize the score of the correct label for each input query relative to the entire distribution of all possible negative query / label pairs in the entire batch, and even across queries.

[0039] In settings with large output spaces, most documents may be irrelevant for any given query, and therefore including them in the loss function may not be informationally rich for optimization. To address this challenge, several works have proposed mining the most difficult and informationally rich negative labels. However, as an approximation of per-query Softmax, these methods are able to perform negative mining relative to a single query at a time. The cross-example negative mining method disclosed in this paper enables negative example mining across examples, where the system is able to mine the globally most difficult negative query / document comparisons in a batch.

[0040] There has always been interest in score calibration to ensure that scores are consistently normalized or interpretable. A common approach is to interpret the output of the Softmax function applied to the model's logit as a probability. While the output of Softmax is technically a probability distribution in the sense that it is normalized, calculating probabilities for any label requires comparison with all other labels. In settings with large output spaces, this is often impossible because the probability space is too large to compute. To address this challenge, the technique disclosed in this paper focuses on a novel loss function that explicitly encourages calibration of the underlying logit. This can be done during training rather than as a post-identification calibration step. This allows label scores to be compared across queries without needing to compute scores for all other labels.

[0041] Consider a multi-class classification setting with instances x∈X and their associated labels y∈Y, where |Y|=K. In some implementations, the goal could be to learn a scoring function that ranks the labels for each instance based on their relevance. Information retrieval settings can be defined similarly to a query set X and associated related documents Y. In some implementations, the goal may be to learn a scoring function that can rank all documents for a given query based on their relevance.

[0042] In a text-to-image retrieval application example, x i It is a text query and y i This corresponds to the relevant image. In some implementations, to score the relevance between the query and the image, a text encoder can be learned to project the text and image into a shared d-dimensional embedding space. and image encoder In some implementations, querying x i With image y j The correlation score between them can be represented by their vector s i，j = <x i ·y j The dot product between >.

[0043] In a standard multi-class classification setting, the model can be optimized using the Softmax cross-entropy loss over the entire label space; that is, the score for the correct label can be compared with the scores for all other labels. Ideally, in a retrieval setting, the system should also be able to compare the score of the matching document with all other documents in the database. However, this can become daunting given that the number of documents can be in the billions. To address this challenge, the Softmax cross-entropy loss is typically computed only on a random subset of the labels; this is often referred to as sampling Softmax. Specifically, consider N corresponding query / document pairs B sampled uniformly from a batch B over a period of time. t ={(x1, y1), ..., (x N y N Mini-batches. Given vector representations of all text queries and images from that mini-batch, the system is able to compute all possible pairs. The pairwise similarity matrix between them. Figure 4A The diagram illustrates an example of this pairwise similarity matrix for a mini-batch of size N=4.

[0044] When the total number of documents is very large and the random subset of documents within each small batch is relatively small, it can generally be assumed that for a given query x i Within the batch, only its corresponding document y i It is related. In some implementations, it can be assumed that all other documents y sampled within the same batch... j The expression j ≠ i is irrelevant to the query. The matching relationship between the query and the documents within the batch is illustrated in the diagram. Figure 4B In this context, 1 indicates a matching relationship (e.g., a true value relationship), while 0 indicates a no-match. Formally, It can be a query for x i All documents in the batch that do not match, i.e., all documents except the one corresponding to the query. i In addition, the similarity score set between them. Figures 4A-4F In this context, this will be all the scores in the i-th row of S, excluding the relevant documents.

[0045]

[0046] In some implementations, the sampled Softmax cross-entropy can be defined as the relevance score s of the query to its matching document. i，i And the relevance score of the query with all non-matching documents. The relative ranking loss between them. Formally,

[0047]

[0048] Figure 4CThe diagram illustrates the loss per example for the second query. Matching pairs s 2，2 The scores are highlighted in light gray and Emphasized in dark gray. This figure highlights that the loss only considers pairs from the same query.

[0049] In the sampling Softmax method, relevant documents are compared only with a small subset of random documents. As a result, most of these documents will be irrelevant to the query and therefore provide no information to guide optimization. As a means to overcome this, random negative mining selects only the hardest negative document for each query within a randomly sampled subset of documents in a batch. Formally, let... For query x i The top k largest fraction sets within the negative fraction set.

[0050] This allows us to define the modified loss for each query only on the most difficult document.

[0051]

[0052] Figure 4D This scenario is illustrated. The negative example set now only includes the most difficult comparison for a given query. In the graph, the dark gray negative scores are...

[0053] According to equations (2) and (3) and Figure 4C and Figure 4D As illustrated in the diagram, it becomes clear that sampling Softmax only captures the distance of a document relative to a single, given query. Because the loss term is invariant with respect to absolute distances and does not compare distances across queries, the distances in the learned vector space cannot be compared across queries.

[0054] To encourage global calibration that allows distance to be used as an absolute measure of relevance, the technique disclosed in this paper aims to extend the cross-example softmax by introducing negative cross-examples. The proposed loss encourages all queries to get closer to their matching documents, rather than all queries getting closer to all irrelevant documents.

[0055] In some implementations, It could be batch B. t All queries in S are paired comparisons with their corresponding, unrelated documents in the same batch. In some implementations, it can be assumed that queries are only related to their corresponding documents, which can correspond to all off-diagonal entries in S. In some implementations,

[0056]

[0057] In some implementations, using equation (4), the cross-example Softmax cross-entropy can be defined as

[0058]

[0059] Figure 4E The diagram illustrates a cross-tabulation example of Softmax, where the loss for a single query can include data from... All negative scores, even from query / document pairs from different queries.

[0060] In some implementations, random negative mining can be expanded using cross-example negatives to mine the hardest negative comparisons across the entire batch. Similar to the formula above, let... This represents the set of the top k largest scores within the entire batch's negative score set. In some implementations, the cross-example negative mining loss can be defined as:

[0061]

[0062] The negative digging loss of this crossover example is illustrated in the diagram. Figure 4F In. Figure 4F In the example shown, the negative scores for each query are mined from the entire batch. This means that the mined score set may contain all negative scores from some queries, as shown in row 1 of the figure, and no negative scores from other queries, as shown in row 3 of the figure.

[0063] Now turn to the diagrams. Figure 1 The diagram illustrates a block diagram of an example environment 100 in which the implementations disclosed herein may be carried out. Example environment 100 includes a computing system 102 capable of including a query engine 106, a resource engine 108, and / or (multiple) additional engines (not depicted). Additionally or alternatively, computing system 102 may be associated with one or more user interface input / output devices 104. Furthermore, computing system 102 may be associated with an input model 110, a resource model 112, a training engine 114, batches 116 of one or more training data, resource vectors 118, electronic resources 120, and / or one or more additional components (not depicted).

[0064] In some implementations, computing system 102 may include user interface input / output device 104, which may include, for example, a physical keyboard, a touchscreen (e.g., implementing a virtual keyboard or other text input mechanism), a microphone, a camera, a display screen, and / or a speaker. The user interface input / output device may be integrated with one or more computing systems 102 of a user. For example, a user's mobile phone may include a user interface input / output device; a standalone digital assistant hardware device may include a user interface input / output device; a first computing device may include multiple user interface input devices and separate computing devices may include multiple user interface output devices; etc. In some implementations, all or all aspects of computing system 102 may be implemented on a computing system that also includes user interface input / output devices. In some implementations, computing system 202 may include an automated assistant (not depicted), and all or all aspects of the automated assistant may be implemented on multiple computing devices separate from and remote from the client device containing the user interface input / output device (e.g., all or all aspects may be implemented "in the cloud"). In some of those implementations, aspects of the automated assistant can communicate with the computing device via one or more networks such as a local area network (LAN) and / or a wide area network (WAN) (e.g., the Internet).

[0065] Some non-limiting examples of computing system 102 include one or more of the following: desktop computing devices, laptop computing devices, stand-alone hardware devices at least partially dedicated to autonomous assistants, tablet computing devices, mobile phone computing devices, vehicle computing devices (e.g., in-vehicle communication and entertainment systems, in-vehicle navigation systems), or wearable devices that include a user of the computing device (e.g., a watch for a user with a computing device, glasses for a user with a computing device, virtual or augmented reality computing devices). Additional and / or alternative computing systems may be provided. Computing system 102 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. Operations performed by computing system 102 may be distributed across multiple computing devices. For example, computing programs running on one or more computers in one or more locations can be coupled to each other via a network.

[0066] like Figure 1As illustrated, training engine 114 can be used to train input model 110 and / or resource model 112. In some implementations, training engine 114 can process batches 116 of one or more training data to generate batch loss, which can be used to update one or more portions of input model 110 and / or resource model 112 (e.g., via backpropagation). For example, batches 116 of training data can include query sets, electronic resource sets, and ground truth pairs between queries and electronic resources. In some implementations, each ground truth pair is a corresponding ground truth pair between an electronic resource and a corresponding ground truth pair in a query. In some of those implementations, each electronic resource has only a corresponding single ground truth pair.

[0067] In some implementations, training engine 114 can generate a set of query vectors by processing each query in the query set of batch 116 of training data using input model 110. Each query vector can be an embedding space representation of the corresponding query. Additionally or alternatively, training engine 114 can generate a set of electronic resource vectors by processing each electronic resource in the electronic resource set of batch 116 of training data using resource model 112. Each electronic resource vector can be an embedding space representation of the corresponding electronic resource. In some implementations, the embedding space of the query vectors is a shared embedding space with electronic resource vectors. Additionally or alternatively, training engine 114 can determine the relevance between each query vector and each electronic resource vector corresponding to batch 116 of training data. In some implementations, a relevance score can be determined for each ground truth query / electronic resource pair. For example, training engine 114 can determine the relevance score (e.g., distance in the embedding space) between the corresponding query vector and the corresponding electronic resource vector of a ground truth pair by determining the dot product between the corresponding query vector and the corresponding electronic resource vector. Additionally or alternatively, the training engine 114 is capable of determining a negative correlation score for each query vector and for each electronic resource vector other than the corresponding electronic resource vector that has a relationship with the true value of the query. In other words, in addition to the true value pairing, it is capable of generating a negative correlation score for each query and electronic resource pair. In some implementations, in addition to the true value electronic resource pairing for the query, the training engine 114 is also capable of determining each negative correlation score between the query vector and the corresponding electronic resource vector (e.g., distance in the embedding space) by determining the dot product between the corresponding query vector and the corresponding electronic resource vector.

[0068] Training engine 114 is able to determine the query loss corresponding to a query in the query set of batch 116 of the training data. In some implementations, the query loss can be based on a relevance score generated for the corresponding query and at least one negative relevance score generated for at least one additional query in batch 116 of the training data. In other words, the query loss can be generated using cross-examples (e.g., at least one negative relevance score generated for at least one additional query in batch 116 of the training data). In some implementations, training engine 114 can generate the query loss using a cross-example Softmax method, wherein the query loss is generated based on: (1) the relevance score of the corresponding query, (2) each of the negative relevance scores generated for the corresponding query, and (3) each of the negative relevance scores generated for each additional query in the batch of the training data.

[0069] Additionally or alternatively, training engine 114 can use a cross-example negative mining method to generate the query loss. In some implementations, training engine 114 can select a subset of negative relevance scores for use when generating the query loss using the cross-example negative mining method, wherein training engine 114 can select one or more negative relevance scores that satisfy one or more conditions. For example, training engine 114 can select each negative relevance score above a threshold corresponding to batch 116 of the training data. Additionally or alternatively, training engine 114 can select the top k negative relevance scores corresponding to batch 116 of the training data (e.g., training engine 114 can select the top 10 negative relevance scores, the top 100 negative relevance scores, etc.). In some implementations, training engine 114 can generate the query negative relevance score based on the query relevance score and the selected subset of negative relevance scores. In some of those implementations, the selected subset of negative relevance scores includes one or more negative relevance scores generated for additional queries.

[0070] Training engine 114 is capable of generating batch loss for batches 116 of training data. In some implementations, the batch loss is based on the query loss generated for each batch 116 of training data (e.g., the batch loss is based on the query loss for each query in the query set for the batch of training data). The training engine can be used to update one or more parts of the input model 110 and / or the resource model 112 based on the generated batch loss.

[0071] Resource engine 108 can be used to generate electronic resource vectors 118 corresponding to the set of candidate electronic resources 120. In some implementations, resource engine 108 can process each candidate electronic resource in the set of candidate electronic resources 120 to generate the corresponding candidate electronic resource vector 118. In some implementations, the candidate electronic resource vector 118 can be stored locally at computing system 102. Additionally or alternatively, the candidate electronic resource vector 118 can be stored remotely from computing system 102 and can be accessed by computing system 102.

[0072] The query engine 106 can be used to determine one or more candidate electronic resources 120 corresponding to a received query. In some implementations, queries can be received via one or more user interface input devices 104. For example, natural language text queries can be received via a keyboard, image queries can be captured via one or more cameras, spoken language queries can be captured using one or more microphones, and / or additional or alternative queries can be provided by the user. The query engine 106 can generate a corresponding query vector by processing the query using the input model 110. Additionally or alternatively, the query engine 106 can determine electronic resources for a received query based on the distance between the query vector and the candidate electronic resource vector 118. For example, electronic resources can be determined based on the electronic resource vector that is closest to the query vector in the embedding space.

[0073] Figure 2A-2C The diagram illustrates the training of the input model and resource model according to some implementation methods. Figure 2A The illustration shows an example of generating a correlation score of 214A-N. Figure 2A In the training data, batch 116 includes ground truth query / electronic resource pairs 202A-202N. For example, ground truth pair 202A includes query A 204A and electronic resource A 206A. Similarly, ground truth pair 202N includes query N 204N and electronic resource N 206N. Each query 204A-N can be processed using input model 110 to generate a corresponding query vector (i.e., an embedding space representation of the corresponding query). For example, query A 204A can be processed using input model 110 to generate query vector A 208A. Similarly, query N 204N can be processed using input model 110 to generate query vector N 208N. Additionally or alternatively, each electronic resource 206A-N can be processed using resource model 112 to generate a corresponding electronic resource vector (i.e., an embedding space representation of the corresponding electronic resource). For example, electronic resource 206A can be processed using resource model 112 to generate electronic resource vector A 210A. Similarly, resource model 112 can be used to process electronic resources 206N to generate electronic resource vector N 210N.

[0074] In the illustrated example, the relevance score engine 212 can be used to generate relevance scores for each query in batch 116 of training data. For example, the relevance score engine 212 can process query vector A 208A and electronic resource vector A 210A (i.e., the vector corresponding to the true value pair 202A) to generate a relevance score A 214A. Similarly, the relevance score engine 212 can process query vector N 208N and electronic resource vector N 210N (i.e., the vector corresponding to the true value pair 202N) to generate a relevance score N 214N. In some implementations, the relevance score engine 212 can generate relevance scores by determining the dot product between the query vector and the electronic resource vector. For example, the relevance score engine 212 can be used to generate a relevance score A214A by determining the dot product between query vector A 208A and electronic resource vector A 210A. Similarly, the relevance score engine 212 can be used to generate a relevance score N214N by determining the dot product between the query vector N208N and the electronic resource vector N210N.

[0075] Figure 2B The illustration shows an example of generating a negative correlation score 218A-M. Figure 2B In this implementation, the negative relevance score engine 216 can be used to generate negative relevance scores 218A-K. In some implementations, negative relevance scores can be generated based on a given query and electronic resources that are not true value pairs with that given query. In some implementations, negative relevance scores can be generated for each query-electronic resource pair that is not a true value pair. For example, negative relevance score A 218A can be generated by processing query vector A 208A and electronic resource vector N 210N (i.e., query vectors and electronic resource vectors that do not correspond to true value pairs) using the negative relevance score engine 216. Similarly, negative relevance score K 218K can be generated by processing query vector N 208N and electronic resource vector A 210A (i.e., query vectors and electronic resource vectors that do not correspond to true value pairs) using the negative relevance score engine 216. In some implementations, the negative relevance score engine 216 can generate the corresponding negative relevance score by determining the dot product between the query vector and the electronic resource vector. For example, the negative relevance score engine 216 can generate a negative relevance score A 218A by determining the dot product between the query vector A 208A and the electronic resource vector N 210N. Similarly, the negative relevance score engine 216 can generate a negative relevance score K 218K by determining the dot product between the query vector N 208N and the electronic resource vector A 210A.

[0076] Figure 2CThe diagram illustrates the generation of batch loss 226 and the use of the generated batch loss 226 to update one or more portions of the input model 110 and / or resource model 112 (e.g., using backpropagation for updating). In some implementations, the query loss can be determined for each query in a batch of training data based on a relevance score 214 corresponding to the query and one or more negative relevance scores 218, wherein at least one of the negative relevance scores 218 is generated for additional queries. In some implementations, the query loss engine 220 can use each of the generated negative relevance scores when generating the query loss (i.e., the cross-example Softmax method). For example, the query loss A 222A corresponding to query A 204A can be generated by processing the relevance score A 214A, each negative relevance score generated for query A, and each negative relevance score generated for each additional query in a batch 116 of training data. Similarly, the query loss N222N corresponding to query N204N can be generated by processing the relevance score N214N, each negative relevance score generated for query N, and each negative relevance score generated for each additional query in batch 116 of the training data.

[0077] Additionally or alternatively, in some implementations, the query loss engine 220 can use a subset of the generated negative relevance scores when generating the query loss (i.e., a cross-example negative mining method). In some implementations, the query loss engine 220 can determine a subset of negative relevance scores that satisfy one or more conditions. For example, the query loss engine 220 can determine a subset of negative relevance scores where each negative relevance score in the subset exceeds a threshold, is a positive value, and / or satisfies one or more additional conditions. Additionally or alternatively, the query loss engine 220 can determine a subset of negative relevance scores by selecting the k negative relevance scores with the highest values ​​(e.g., selecting the top 10 negative relevance scores, the top 50 negative relevance scores, the top 100 negative relevance scores, and / or an additional number of the highest negative relevance scores). For example, the query loss engine 220 can use the query loss engine 220 to process a subset of relevance score A 214A and negative relevance score 218 to generate a query loss A222A. Similarly, query loss engine 220 can be used to process subsets of relevance scores N 214N and negative relevance scores 218 to generate query loss N 222N. In some implementations, query loss engine 220 can determine the same subset of negative relevance score values. In some other implementations, query loss engine 220 can determine different subsets of negative relevance score values ​​for one or more queries in a batch of training data.

[0078] In some implementations, query loss engine 220 can be used to generate query losses for each query in a batch of training data. Batch losses can be generated based on one or more of the generated query losses. For example, batch loss engine 224 can process query losses A 222A and query losses N 222N to generate batch loss 226. Batch loss 226 can be used (e.g., using backpropagation) to update one or more parts of input model 110 and / or resource model 112. Figure 2A-2C This is described relative to a batch of training data that includes two queries, two electronic resources, and corresponding ground truth pairs. However, this is merely an example, and the batch 116 of training data can include additional queries, electronic resources, and / or ground truth pairs.

[0079] Figure 3A and Figure 3B The figure shows the embedding space 304 learned using conventional Softmax techniques (e.g., sampled Softmax, random negative mining, etc.). Figure 3A The illustration depicts query A 302 (e.g., the vector representation of query A in embedding space 304) and candidate electronic resources 306, 308, and 310 (e.g., the vector representations of electronic resources 306, 308, and 310 in embedding space 304). In the illustrated example, the system is able to determine that electronic resource 306 responds to query A 302. For example, the system is able to determine that the distance between the vector representation of query A 302 and the vector representation of electronic resource 306 is less than (1) the distance between the vector representation of query A 302 and the vector representation of electronic resource 308 and / or (2) the distance between the vector representation of query A 302 and the vector representation of electronic resource 310.

[0080] Figure 3B The illustration depicts query B 312 (e.g., the vector representation of query B in embedding space 304) and candidate electronic resources 314, 316, 318, and 320 (e.g., the vector representations of electronic resources 314, 316, 318, and 320 in embedding space 304). In the illustrated example, the system is able to determine electronic resource 314 in response to query B 312. For example, the system is able to determine that the distance between the vector representation of query B 312 and the vector representation of electronic resource 314 is less than (1) the distance between the vector representation of query B 312 and the vector representation of electronic resource 316, (2) the distance between the vector representation of query B 312 and the vector representation of electronic resource 318, and / or (3) the distance between the vector representation of query B 312 and the vector representation of electronic resource 320.

[0081] However, embedding space 304 has not yet used cross-examples for global calibration. Although electronic resources can be determined for queries A and B, the distances between queries and their corresponding electronic resources are not comparable. For example, the distance between query A 302 and its corresponding electronic resource 306 is greater than the distance between query B 312 and electronic resource 316 that does not respond to query B.

[0082] In comparison, Figure 3C and Figure 3D The diagram illustrates the embedding space 322, which uses cross-example learning in accordance with some of the implementations described in this paper. Figure 3C The diagram illustrates query A 302 (e.g., the vector representation of query A in embedding space 322) and candidate electronic resources 306, 308, and 310 (e.g., the vector representations of electronic resources 306, 308, and 310 in embedding space 322). Figure 3D The diagram illustrates query B 312 (e.g., the vector representation of query B in embedding space 322) and candidate electronic resources 314, 316, 318 and 320 (e.g., the vector representations of electronic resources 314, 316, 318 and 320 in embedding space 322).

[0083] Similar to Figure 3A and Figure 3B The system is able to determine that electronic resource 306 responds to query A302 (e.g., based on the distance between the vector representation of query A302 and the vector representation of electronic resource 306), and is able to determine that electronic resource 314 responds to query B312 (e.g., based on the distance between the vector representation of query B312 and the vector representation of electronic resource 314).

[0084] However, the embedding space 322 has been globally calibrated using cross-examples, such that distances within the embedding space provide an indication of how relevant the electronic resource is to the query. For example, the distances in the embedding space 322 are globally calibrated so that the distance between query A and the corresponding electronic resource 306 is comparable to the distance between query B and the corresponding electronic resource 314. In the illustrated example, electronic resource 314 is closer to query B 312, rather than electronic resource 306 being closer to query A 302. In some implementations, this can indicate that electronic resource 314 is more responsive to query B 312, rather than electronic resource 306 being more responsive to query A 302.

[0085] Figure 5This is a flowchart illustrating process 500 for training an input model and / or a response model using cross-examples according to the implementation disclosed herein. For convenience, the operations of the flowchart are described with reference to the system performing the operations. This system may include various components of various computer systems, such as one or more components of computing system 102 and / or computing system 910. Furthermore, although the operations of process 500 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0086] At box 502, the system receives a batch of training data comprising a query set, an electronic resource set, and ground truth pairs. In some implementations, each query has a corresponding electronic resource ground truth pair. Alternatively or additionally, each electronic resource has a single corresponding ground truth pair. For example, the batch of training data can include a natural language text query set, an image set (i.e., electronic resources), and ground truth pairs of natural language text queries and images. Additional and / or alternative query sets and / or electronic resource sets can be utilized.

[0087] At box 504, the system generates a query vector for each query in the query set by processing the query using an input model. Additionally or alternatively, the system generates an electronic resource vector for each electronic resource in the electronic resource set by processing the electronic resource using a resource model. For example, the system can use... Figure 1 The input model 110 is used to process the corresponding query and generate a query vector. Additionally or alternatively, the system can use... Figure 1 Resource model 112 is used to process the corresponding electronic resources and generate electronic resource vectors. In some implementations, each query vector is a shared embedding space representation of the corresponding query. Alternatively or additionally, each electronic resource is a shared embedding space representation of the corresponding electronic resource.

[0088] At box 506, the system generates a relevance score for each query in the query set based on (1) the query vector of the query and (2) an electronic resource vector that has a pairing with the ground truth of the query. For example, the system can determine the relevance score for each ground truth pair by determining the dot product between the corresponding query vector (e.g., the query vector generated at box 504) and the corresponding electronic resource vector (e.g., the electronic resource vector generated at box 504).

[0089] At box 508, the system generates a negative correlation score for each query and for each electronic resource other than ground truth pairings, based on (1) the query vector corresponding to the query and (2) the electronic resource vector corresponding to the electronic resource. For example, for each query vector and each electronic resource vector that is not a ground truth pairing with that query vector, the system can generate a negative correlation score by determining the dot product between the corresponding query vector (e.g., the query vector generated at box 504) and the corresponding electronic resource vector (e.g., the electronic resource vector generated at box 504).

[0090] At box 510, the system generates a query loss for each query in the query set based on (1) the relevance score of the corresponding query and (2) one or more negative relevance scores generated for at least one additional query. In some implementations, the system is able to use cross-sample Softmax to generate the query loss for each query. The following are described in this paper. Figure 6 Procedure 600 is an example procedure for generating query loss using cross-example softmax. Additionally or alternatively, the system can use cross-example negative mining for each query to generate query loss. The procedures described in this paper... Figure 7 Process 700 is an example process for generating query loss using cross-example negative mining.

[0091] At box 512, the system generates batch loss for batches of training data based on the generated query loss. In some implementations, the system generates batch loss based on the query loss generated for each query in a batch of training data.

[0092] At box 514, the system updates one or more portions of the input model and / or resource model based on batch loss (e.g., using backpropagation). For example, the system can update one or more portions of the input model used at box 504 to generate a query vector using batch loss. Additionally or alternatively, the system can update one or more portions of the resource model used at box 504 to generate an electronic resource vector using batch loss.

[0093] Figure 6 This is a flowchart illustrating a process 600 for generating a query loss using the cross-example Softmax method according to the implementation disclosed herein. For convenience, the operations in the flowchart are described with reference to the system performing the operations. This system may include various components of various computer systems, such as one or more components of computing system 102 and / or computing system 910. Furthermore, although the operations of process 600 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0094] At box 602, the system selects a query from the query set. In some implementations, the query can be selected from a batch of query sets in the training data. In some of those implementations, the batch of training data can include query sets, electronic resource sets, and true value pairs of queries and electronic resources. For example, the system can select a query from a batch of query sets in the training data. Figure 5 In the process of receiving training data in the batch at box 502 of process 500, select the query.

[0095] At box 604, the system uses cross-example Softmax to generate a query loss for the selected query based on: (1) the relevance score generated for the selected query, (2) each negative relevance score generated for the selected query, and (3) each negative relevance score generated for each additional query. In other words, the system is able to generate a query loss for the selected query based on the relevance score of the selected query and each negative relevance score generated for the batch of training data. In some implementations, it is possible to... Figure 5 The process generates a relevance score at frame 506 of box 500. In some implementations, it is possible to... Figure 5 The process generates each of the negative correlation scores at box 508 in position 500.

[0096] At box 606, the system determines whether to process any additional queries. If so, the system returns to box 602, selects additional queries from the query set, and proceeds to box 604 to generate additional query loss for the selected additional queries. If the system determines that no additional queries should be processed, the process ends. In some implementations, the system can determine not to process any additional queries if no unprocessed queries remain in the query set and / or based on whether (multiple) additional conditions are met.

[0097] Figure 7 This is a flowchart illustrating process 700 for generating query loss using the cross-example negative mining method according to the implementation disclosed herein. For convenience, the operations of the flowchart are described with reference to the system performing the operations. This system may include various components of various computer systems, such as one or more components of computing system 102 and / or computing system 910. Furthermore, although the operations of process 700 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0098] At box 702, the system selects a subset of negative correlation scores from the set of negative correlation scores that satisfy one or more conditions. For example, it is able to... Figure 5In process 500, at box 508, a set of negative relevance scores is generated. For example, the system can select a subset of negative relevance scores based on whether the negative relevance scores exceed a threshold, whether the negative relevance scores do not exceed a threshold, and / or whether additional or alternative conditions are met. Additionally or alternatively, the system can select the top k negative relevance scores as a subset of negative relevance scores. For example, the system can select the top 20 negative relevance scores from the set of negative relevance scores. In some implementations, one or more negative relevance scores corresponding to the query are not selected from the subset of negative relevance scores. In some implementations, all negative relevance scores corresponding to the query are selected from the subset of negative relevance scores. In some implementations, negative relevance scores corresponding to the query are not selected from the subset of negative relevance scores.

[0099] At box 704, the system selects a query from the query set. In some implementations, the query can be selected from a batch of query sets used to generate the relevance score set. In some of those implementations, the batch of training data can include the query set, the electronic resource set, and ground truth pairs of queries and electronic resources. For example, the system can select from... Figure 5 In the process of receiving training data in the batch at box 502 of process 500, select the query.

[0100] At box 706, the system uses cross-example negative mining to generate a query loss based on a selected subset of (1) the relevance score generated for the selected query and (2) the negative relevance score.

[0101] At box 708, the system determines whether to process any additional queries. If so, the system returns to box 704, selects additional queries from the query set, and proceeds to box 706 to generate additional query loss for the selected additional queries. If the system determines that no additional queries should be processed, the process ends. In some implementations, the system can determine not to process any additional queries if there are no unprocessed queries remaining in the query set and / or based on whether (multiple) additional conditions are met.

[0102] Figure 8 This is a flowchart illustrating a process 800 for determining electronic resources according to the implementation disclosed herein. For convenience, the operations of the flowchart are described with reference to the system performing the operations. This system may include various components of various computer systems, such as one or more components of computing system 102 and / or computing system 910. Furthermore, although the operations of process 800 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0103] At box 802, the system receives a query. For example, the system can receive a natural language text query from a user of the computing system.

[0104] At box 804, the system generates a query vector by processing the query using the input model.

[0105] At box 806, the system compares the query vector with a plurality of pre-stored candidate electronic resource vectors. In some implementations, each pre-stored candidate electronic resource vector is previously generated by processing the corresponding candidate electronic resource using a resource model. For example, the system can compare the query vector with a plurality of pre-stored candidate electronic resource vectors, where each candidate electronic resource vector corresponds to a candidate image for a candidate natural language text query. In some implementations, the system determines the distance between the query vector and each of the pre-stored candidate electronic resource vectors by determining the dot product between the query vector and each of the pre-stored candidate electronic resource vectors.

[0106] At box 808, the system selects a pre-stored candidate electron resource vector based on this comparison. In some implementations, the system is able to select the pre-stored candidate electron resource vector that is closest to the query vector (e.g., the pre-stored candidate electron resource vector with the smallest distance determined at box 806).

[0107] At box 810, the system determines the electronic resource based on the selected pre-stored candidate electronic resource vector for the query.

[0108] At box 812, the system enables the computing system to perform one or more actions based on the determined electronic resources. For example, the system can display the determined electronic resources on a screen for the user of the computing system. In some implementations, the system can provide the distance between the corresponding query vector determined at box 806 and the corresponding pre-stored electronic resource vector to one or more downstream components in the computing system. For example, the system can determine a bounded box (e.g., an electronic resource) for a given image (e.g., a query). In some implementations, the system can provide the bounded box to an additional system that can be used to identify objects captured within the bounded box. Additionally or alternatively, the system can make a determination not to present the electronic resource to the user, but instead can request additional information from the user (e.g., requesting information to clarify the query, requesting a new query, etc.). For example, the system can receive an image as a query at box 802. The system can determine one or more bounded boxes of the image based on comparing the corresponding query vector with a plurality of pre-stored candidate electronic resource vectors corresponding to candidate bounded boxes.

[0109] In some implementations, the system can receive a natural language query at box 802. The system can determine one or more images for the query based on comparing the corresponding query vector with multiple pre-stored candidate electronic resource vectors corresponding to candidate images. In some implementations, the system can render one or more electronic resources based on the comparison of the query vector with the pre-stored candidate electronic resource vectors (i.e., based on the relevance scores of the query vector and the multiple pre-stored electronic resource vectors). In some implementations, the system can directly link to a single electronic resource based on a relevance score that satisfies one or more conditions (e.g., the relevance score exceeds a threshold indicating a high relevance between the query and the electronic resource). In some implementations, the system can render electronic resources with corresponding high relevance scores in a manner that, for example, emphasizes the electronic resources (e.g., renders the electronic resources more prominently, presents large segments of the electronic resources, etc.), while additional electronic resources with lower corresponding relevance scores are rendered without emphasis. In some implementations, computational resources (e.g., bandwidth, processor cycles, memory, etc.) can be saved by providing electronic resources to the user based on the relevance scores corresponding to the electronic resources.

[0110] Figure 9 This is a block diagram of an example computing device 910 that may be optionally used to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client computing device and / or (a plurality of) other components may include one or more components of the example computing device 910.

[0111] Computing device 910 typically includes at least one processor 914 that communicates with a number of peripheral devices via a bus subsystem 912. These peripheral devices may include a storage subsystem 924 (including, for example, a memory subsystem 925 and a file storage subsystem 926), a user interface output device 920, a user interface input device 922, and a network interface subsystem 916. The input and output devices allow users to interact with computing device 910. The network interface subsystem 916 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0112] User interface input device 922 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touchscreen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. Generally, the term "input device" is used to encompass all possible types of devices and methods for inputting information into computing device 910 or onto a communication network.

[0113] User interface output device 920 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (“CRT”), a flat panel device such as a liquid crystal display (“LCD”), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. Generally, the term “output device” is used to encompass all possible types of devices and methods for outputting information from computing device 910 to a user or to another machine or computing device.

[0114] Storage subsystem 924 provides some or all of the functional programming and data construction capabilities described in this document. For example, storage subsystem 924 may include functions for performing... Figure 5 , Figure 6 , Figure 7 and / or Figure 8 One or more of the selected aspects and implementations in the process Figure 1 The logic of the various components described in the text.

[0115] These software modules are typically executed by processor 914 alone or in combination with other processors. Memory 925 used in storage subsystem 924 can include multiple memories, including main random access memory (“RAM”) 930 for storing instructions and data during program execution and read-only memory (“ROM”) 932 for storing fixed instructions. File storage subsystem 926 provides persistent storage for program and data files and may include hard disk drives, floppy disk drives and associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing certain functionalities can be stored by file storage subsystem 926 within storage subsystem 924 or in other machines accessible by processor(s) 914.

[0116] Bus subsystem 912 provides a mechanism for enabling the various components and subsystems of computing device 910 to communicate with each other as intended. Although bus subsystem 912 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0117] The computing device 910 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, Figure 9 The description of the computing device 910 depicted is intended only as a specific example for illustrating some implementation methods. Many other configurations of the computing device 910 may have... Figure 9 The computing device depicted in the text has more or fewer components.

[0118] The systems described herein collect personal information about users (or, as often referred to herein, "participants"), or where such personal information may be used, and may provide users with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location) or to control whether and / or how content that may be more relevant to the user is received from a content server. Additionally, certain data may be processed in one or more ways before it is stored or used, such that personally identifiable information is removed. For example, a user's identity may be processed to the point that personally identifiable information cannot be determined for that user, or, where geographic location information is available, the user's geographic location may be generalized (e.g., to the city, zip code, or state level), making it impossible to determine the user's specific geographic location. Therefore, users can control how information about themselves is collected and / or used.

[0119] In some implementations, a method implemented by one or more processors is provided, the method comprising identifying batches of training data including a query set, an electronic resource set, and ground truth pairs, wherein each of the ground truth pairs is a ground truth pair in an electronic resource corresponding to a corresponding one in a query, and wherein each of the electronic resources has a corresponding single ground truth pair in only the ground truth pairs. In some implementations, for each query in the query set, the method comprises generating a corresponding query vector by processing the query using an input model. In some implementations, for each electronic resource in the electronic resource set, the method comprises generating a corresponding electronic resource vector by processing the electronic resource using a resource model. In some implementations, for each query in the query set, the method comprises generating a relevance score based on (1) a corresponding query vector generated for the query and (2) a corresponding electronic resource vector generated for electronic resources having ground truth pairs with the query. In some implementations, the method comprises generating a corresponding negative relevance score for each electronic resource other than electronic resources having ground truth pairs with the query, based on (1) a corresponding query vector generated for the query and (2) an electronic resource vector generated for electronic resources other than electronic resources having ground truth relationships with the query. In some implementations, the method includes generating a query loss based on (1) a relevance score generated for the query and (2) at least one corresponding negative relevance score generated for at least one additional query in the query set. In some implementations, the method includes generating a batch loss based on the generated query loss for batches of training data. In some implementations, the method includes updating one or more portions of the input model and / or resource model based on the generated batch loss.

[0120] These and other implementations of the techniques disclosed herein can include one or more of the following features.

[0121] In some implementations, generating query loss based on (1) a relevance score generated for the query and (2) at least one corresponding negative relevance score generated for at least one additional query in the query set includes generating query loss based on: (1) the relevance score generated for the query, (2) the corresponding negative relevance score generated for the query, and (3) all corresponding negative relevance scores generated for each of the additional queries in the query set.

[0122] In some implementations, generating a query loss based on (1) a relevance score generated for the query and (2) at least one corresponding negative relevance score generated for at least one additional query in the query set includes selecting a subset of negative relevance scores, wherein the selected subset includes at least one negative relevance score generated for at least one additional query in the query set, and wherein the selection of the subset is based on the corresponding negative relevance scores of the subset satisfying one or more conditions. In some implementations, the method includes generating a query loss based on (1) a relevance score generated for the query and (2) a subset of corresponding negative relevance scores.

[0123] In some implementations, each query in the query set is a natural language query, and each electronic resource in the electronic resource set is an image or a web page.

[0124] In some implementations, each query in the query set is an image of the captured object, and each electronic resource in the electronic resource set represents one or more corresponding bounded boxes.

[0125] In some implementations, following updating one or more portions of the input model and / or resource model based on the generated batch loss, the method further includes deploying the trained input model on the computing system. In some implementations, the method includes receiving a user query via one or more user interface input devices of the computing system. In some implementations, the method includes determining a user query vector by processing the user query using the trained input model. In some implementations, the method includes determining a user electronic resource in response to a user query, wherein determining the user electronic resource in response to a user query includes comparing the user query vector with a plurality of pre-stored candidate electronic resource vectors, wherein each of the plurality of pre-stored candidate electronic resource vectors is previously generated by processing the corresponding electronic resource using the resource model. In some implementations, the method includes selecting a pre-stored candidate electronic resource vector based on the comparison. In some implementations, the method includes determining a user electronic resource based on the selected pre-stored candidate electronic resource vector. In some implementations, the method includes causing the computing system to perform one or more actions based on the determined user electronic resource.

[0126] In some implementations, generating a relevance score for each query in the query set, based on (1) a corresponding query vector generated for the query and (2) a corresponding electronic resource vector generated for electronic resources that have a pairing with the true value of the query, includes determining the dot product between (1) the corresponding query vector generated for the query and (2) the corresponding electronic resource vector generated for electronic resources that have a pairing with the true value of the query. In some implementations, the method includes generating the relevance score based on the determined dot product.

[0127] In some implementations, for each query in the query set, generating a corresponding negative correlation score for each electronic resource other than those having a corresponding electronic resource paired with the true value of the query, based on (1) a corresponding query vector generated for the query and (2) an electronic resource vector generated for the electronic resources other than those having a relationship with the true value of the query, includes determining the dot product between (1) the corresponding query vector generated for the query and (2) the electronic resource vector generated for the electronic resources other than those having a relationship with the true value of the query. In some implementations, the method includes generating the negative correlation score based on the determined dot product.

[0128] In some implementations, for each query in the query set, the generated query vector projects the query into a shared embedding space, and for each electronic resource in the electronic resource set, the electronic resource vector projects the electronic resource into the shared embedding space.

[0129] In some implementations, a method implemented by one or more processors is provided, the method including receiving an image of a captured object. In some implementations, the method includes generating an image vector by processing the image using an input model. In some implementations, a bounded box of an object captured in the image is determined, wherein determining the bounded box of the object includes comparing the image vector with a plurality of pre-stored candidate bounded box vectors, wherein each of the plurality of pre-stored candidate bounded box vectors is previously generated by processing the corresponding candidate bounded box using a resource model. In some implementations, the method includes selecting a pre-stored candidate bounded box vector based on the comparison. In some implementations, the method includes determining the bounded box of the object based on the selected pre-stored candidate bounded box vector. In some implementations, the method includes causing a computing device to perform one or more actions based on the determined bounded box of the object.

[0130] Additionally, some implementations include one or more processors (e.g., multiple central processing units (CPUs), multiple graphics processing units (GPUs), and / or multiple tensor processing units (TPUs)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause any of the methods described herein to be performed. Some implementations also include one or more transient or non-transitory computer-readable storage media storing computer instructions that can be executed by the one or more processors to perform any of the methods described herein.

Claims

1. A method implemented by one or more processors, the method comprising: Identify batches of training data comprising query sets, electronic resource sets, and ground truth pairs, wherein each of the ground truth pairs is a corresponding ground truth pair in the electronic resource set and a corresponding ground truth pair in the query set, and wherein each of the electronic resource sets has only a corresponding single ground truth pair in the ground truth pairs. For each query in the query set, a corresponding query vector is generated by processing the query using an input model; For each electronic resource in the electronic resource set, a corresponding electronic resource vector is generated by processing the electronic resource using a resource model; For each query in the query set A relevance score is generated based on the corresponding query vector generated for the query and the corresponding electronic resource vector generated for the electronic resources that have a pairing with the true value of the query; and For each electronic resource other than those that have a pairing with the true value of the query, a corresponding negative correlation score is generated based on the corresponding query vector generated for the query and the electronic resource vector generated for the electronic resources other than those that have a relationship with the true value of the query. The query loss is generated based on the relevance score generated for the query and at least one corresponding negative relevance score generated for at least one additional query in the query set; For each batch of the training data, a batch loss is generated based on the generated query loss; and Update one or more parts of the input model and / or the resource model based on the generated batch loss.

2. The method according to claim 1, wherein, Generating the query loss based on the relevance score generated for the query and the at least one corresponding negative relevance score generated for at least one additional query in the query set includes: The query loss is generated based on the relevance score generated for the query, the corresponding negative relevance score generated for the query, and all the corresponding negative relevance scores generated for each of the additional queries in the query set.

3. The method according to claim 1, wherein, Generating the query loss based on the relevance score generated for the query and the at least one corresponding negative relevance score generated for at least one additional query in the query set includes: A subset of the negative relevance scores is selected, wherein the selected subset includes at least one negative relevance score generated for at least one additional query in the query set, and wherein the selection of the subset is based on the corresponding negative relevance score of the subset satisfying one or more conditions; and The query loss is generated based on the subset of the relevance score and the corresponding negative relevance score generated for the query.

4. The method according to claim 1, wherein, Each query in the query set is a natural language query, and each electronic resource in the electronic resource set is an image or a web page.

5. The method according to claim 1, wherein, Each query in the query set is an image of a captured object, and each electronic resource in the electronic resource set represents one or more corresponding bounded boxes.

6. The method of claim 1, further comprising, after updating one or more portions of the input model and / or the resource model based on the generated batch loss: Deploy the input model on a computing system; The computing system receives user queries via one or more user interface input devices. The user query vector is determined by processing the user query using the input model. Determine the user electronic resources responding to the user query, wherein determining the user electronic resources responding to the user query includes: The user query vector is compared with a plurality of pre-stored candidate electronic resource vectors, wherein each of the plurality of pre-stored candidate electronic resource vectors was previously generated by processing the corresponding electronic resource using the resource model; Based on the comparison, a pre-stored candidate electronic resource vector is selected; and The user's electronic resources are determined based on the selected pre-stored candidate electronic resource vectors; and The computing system performs one or more actions based on the determined user electronic resources.

7. The method according to claim 1, wherein, For each query in the query set, generating the relevance score based on the corresponding query vector generated for the query and the corresponding electronic resource vector generated for the electronic resources having a pairing with the true value of the query includes: Determine the dot product between the corresponding query vector generated for the query and the corresponding electronic resource vector generated for the electronic resource having a pairing with the true value of the query; and The correlation score is generated based on the determined dot product.

8. The method according to claim 1, wherein, For each query in the query set, and for each electronic resource other than those having a pairing with the true value of the query, generating the corresponding negative correlation score based on the corresponding query vector generated for the query and the electronic resource vector generated for the electronic resources other than those having a relationship with the true value of the query includes: Determine the dot product between the corresponding query vector generated for the query and the electronic resource vector generated for the electronic resources other than those having a relationship with the true value of the query; and The negative correlation score is generated based on the determined dot product.

9. The method according to any one of claims 1 to 8, wherein, For each query in the query set, the generated query vector projects the query into a shared embedding space, and for each electronic resource in the electronic resource set, the electronic resource vector projects the electronic resource into the shared embedding space.

10. A computer program product, the computer program comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to perform the method according to any one of claims 1 to 9.

11. A computing system configured to perform the method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing instructions executable by one or more processors of a computing system to perform the method according to any one of claims 1 to 9.

Images