Neural network for search retrieval and ranking

Abstract

A mechanism for identifying and ranking search results using neural networks is described herein. A machine learning model is trained by transforming training data comprising query-document entries into query item-document entries. The query item-document entries are used to train the machine learning model. A set of query items is identified. The query items can be derived from a query history. The trained machine learning model is used to compute document ranking scores for the query items, and the resulting scores are stored in a pre-computed item-document index. A query that searches a document index is decomposed into its constituent items, and an aggregate document ranking score is computed from a weighted sum of the document ranking scores corresponding to the individual query items. Because the item-document index can be pre-computed, it can be downloaded to provide deep learning search capabilities in a computationally limited environment.

Description

TECHNICAL FIELD

[0001] The present application relates generally to the use of neural networks in search retrieval and ranking. More specifically, the present application relates to improvements in neural networks for retrieving and ranking documents in an information retrieval system. BACKGROUND

[0002] Information retrieval systems, such as Internet search systems, operate at a large scale, indexing millions of web pages to make information searchable by user-submitted queries. The goal of such search systems is to quickly identify those documents that are relevant to a user query and return relevant search results. Correctly configured neural networks have been demonstrated in research environments to improve the relevance of search results. However, due to the computational expense of such approaches, and due to the large scale and need for low latency, the use of neural networks in search systems has been limited to re-ranking a limited set of search results outside of research environments.

[0003] The present embodiments arise within this context. BRIEF DESCRIPTION OF DRAWINGS

[0004] Figure 1 An example of a prior art search system architecture is illustrated.

[0005] Figure 2 An example of a prior art neural network is illustrated.

[0006] Figure 3 An example prior art method for training a neural network in a search system is illustrated.

[0007] Figure 4 An example method for transforming training data according to some aspects of the present disclosure is illustrated.

[0008] Figure 5 An example method for training a machine learning model according to some aspects of the present disclosure is illustrated.

[0009] Figure 6 An example diagram depicting pre-computed term-document indices according to some aspects of the present disclosure is illustrated.

[0010] Figure 7 An example flow diagram showing how pre-computed term-documents are computed according to some aspects of the present disclosure is illustrated.

[0011] Figure 8 An example flow diagram showing the use of pre-computed term-document scores to compute document rankings according to some aspects of the present disclosure is illustrated.

[0012] Figure 9representative architectures for implementing the systems and other aspects disclosed herein or for performing the methods disclosed herein are illustrated. DETAILED DESCRIPTION

[0013] The description that follows includes illustrative systems, methods, user interfaces, techniques, instruction sequences, and computing machine program products illustrative of embodiments. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide an understanding of various embodiments of the inventive subject matter. It will be evident, however, to those skilled in the art, that embodiments of the inventive subject matter can be practiced without some or all of these specific details. In general, well-known instruction instances, protocols, structures, and techniques have not been shown in detail.

[0014] SUMMARY

[0015] The following summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features of the claimed subject matter, nor is it meant to be used to limit the claimed subject matter's scope. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

[0016] In search systems, machine learning based ranking models compare the entire query against each document in the entire document collection to be searched, and thus are only used for post-ranking. Embodiments of the present disclosure can be applied to a variety of systems whenever a search system retrieves documents based on a user query. Embodiments of the present disclosure employ machine learning models that operate under a query term independent construction. In the context of the present disclosure, "query term" refers to any set of characters, such as a word, subword, byte string, etc. A "query term" is smaller than the entire query string. Thus each document in the collection can be compared independently, and then scores for individual query terms can be combined into a weighted sum to arrive at a score for the entire query.

[0017] Because machine learning models that use the entire query (or combinations of n-grams from the query) reveal and exploit hidden relationships between query terms, it is expected that while the methods disclosed in embodiments of the present disclosure are more computationally efficient, they will also suffer in ranking effectiveness compared to machine learning models that use the entire query. However, experiments show that using the query term independent methods disclosed in embodiments on state-of-the-art deep learning based ranking models show minimal (or no) ranking effectiveness degradation compared to the entire query methods.

[0018] Further, the query-agnostic approach of the present disclosure allows for pre-computing all item-document scores offline using a deep learning ranking model. At search time, the final query-document scores are computed as a linear sum of the pre-computed item-document scores. Further, the pre-computed scores can be used in combination with inverted indexes to perform very efficient retrieval from web-scale collections. Altogether, this allows deep learning models to be used for retrieval on large collections, such as web indexes.

[0019] Further, the pre-computation of item-document scores allows for deploying deep learning search capabilities in computationally limited environments, such as mobile or other client-side devices. This opens up entirely new scenarios where state-of-the-art ranking and retrieval models can be used in environments where they currently cannot be deployed. Thus, embodiments of the present disclosure have efficiency, storage, and privacy benefits over current approaches.

[0020] DETAILED DESCRIPTION

[0021] Base search system architecture

[0022] Figure 1 An example 100 of a prior art search system architecture is illustrated. Figure 1 The architecture of FIG. 1 is simplified in that it does not discuss all implementation details. However, it provides enough detail to allow the context of the present embodiments to be set and it covers aspects relevant to the present disclosure.

[0023] For a web-scale search system, such as a search system that searches millions of documents, the system first extracts information about the documents to be searched and builds one or more indexes that can be searched against user queries to identify relevant documents. Thus, one or more crawlers 104 identify a collection of documents that will represent the corpus on which the search will be performed. The collection of documents in a web search system is the documents that reside on the Internet 102. Thus, the crawler 104 crawls the web and extracts information about the documents that reside on the web. In this context, a document includes any information accessible on the Internet, such as web pages, images, other types of documents, etc.

[0024] The extracted information is managed by a storage manager 106 that deduplicates the information and stores the information in one or more page stores 108. One or more indexers 110 create document indexes 112 and / or indexes of specific type information 114 from the information in the page stores 108.

[0025] When a user submits a search query via a user machine 116, it is received by a query engine, which searches the indexes 112, 114 and identifies a set of documents relevant to the search query. The set of documents is ranked 120, and a final set of results is returned in response to the query to the user machine 116.

[0026] Documents are typically ranked by multiple tiers of rankers 112, 124 in order to arrive at a final set of search results to be returned to the user. Multiple tiers are used for computational efficiency. The first ranker tier 122 typically employs computationally very efficient methods to quickly narrow down the set of documents under consideration to a set of documents that are likely to be relevant to the search query. The later ranker tier 124 typically employs more complex but less computationally efficient methods that are able to better distinguish relevant documents from non-relevant documents at a higher computational cost. Thus, the goal of the upper tiers is to quickly narrow down the set of documents under consideration, while the goal of the lower tiers is to identify the relative relevance of individual documents.

[0027] Since deep learning methods are computationally expensive, but are able to better determine relative rankings, they are typically only used in the later ranking phase of the system, thus being limited to re-ranking a dataset rather than initial document selection.

[0028] While the above description is made with reference to documents, search results typically do not include the documents themselves, but rather links to the locations where the documents can be found. However, embodiments of the present disclosure are able to return search results that include links to the documents, the documents themselves, and / or combinations thereof. Thus, the present description does not distinguish between returning documents and returning links to documents in search results, and returning a document should be taken to include returning the document, a link to the document, and combinations thereof.

[0029] Example neural network architecture

[0030] Figure 2 An example of a prior art neural network 200 is illustrated. The architecture is representative of a general architecture rather than any specific architecture. Different deep learning model architectures have been developed, and some are better than others on certain tasks. However, Figure 2 The general architecture of

[0031] A deep learning model in a search system compares an input query 202 to candidate documents 206 in order to produce ranking scores 218 for the candidate documents 206 relative to the input query 202. The deep learning model typically includes embedding layers 204, 208 that produce word embeddings for the input query 202 and / or the candidate documents 206. Following the embedding layers are one or more neural networks and / or encoding layers 210, 214. The neural networks and / or encoding layers can include various types of layers depending on the particular model architecture. These are the layers that use and produce the relevance of the features of the input query 202 and / or the candidate documents 206 in the trained model so that the ranking scores 218 that indicate the relevance can be produced. The model has one or more output layers 216 that produce the ranking scores 218. Various output layers 216 can be used such as softmax, sum, sigmoid, etc. as known in the art.

[0032] As discussed herein, neural network architectures for search have several characteristics that limit their usefulness in search systems. First, the computational burden of computing ranking scores for a large number of candidate documents makes them infeasible for large scale search and ranking problems. Thus, outside of research problems, neural network architectures are limited to post-search result re-ranking where the number of candidate documents under consideration is more limited.

[0033] Second, the entire premise of neural network architectures is to uncover "hidden" (e.g. unknown) relevance between features of a query and features of a candidate document to identify documents that are more relevant to a search query and / or to gain more accuracy in ranking scores. The training process adjusts the weights in the neural / encoding layers so that features / relevance that are important for ranking score accuracy are emphasized and those that are not important are minimized. Thus, it is intuitive that the more information that is presented to the neural network, the more relevance can be uncovered and utilized. Thus, when using a neural network model to rank search documents, it is intuitive that the entire query string is better than just entering a few keywords.

[0034] Training a machine learning model

[0035] Figure 3 An example prior art method 300 for training a neural network in a search system is illustrated. In this disclosure, the term "machine learning model" will be used to encompass neural networks, deep learning models, and other forms of learning models suitable for information retrieval such as document identification and ranking in search systems as described herein.

[0036] The machine learning tool operates by adjusting the weights and other parameters of the machine learning model using example training data 304 in order to make data-driven predictions or decisions expressed as outputs or assessments 314. Although example embodiments are presented with respect to several machine learning models, the principles presented herein can be applied to other machine learning models.

[0037] Through the training process, the machine learning model begins to recognize correlations and interactions between features in the data 302, which leads to correct assessments 314 of the input data. A feature is an individual characteristic of an observed phenomenon. Features do not always correspond to easily identifiable aspects of the data, as some important features "discovered" by the machine learning model in machine learning are mixtures or amalgamations of input data features.

[0038] The machine learning algorithm utilizes the training data 304 to find correlations between features of the data 302 that affect the outcome or assessment 314. In some example embodiments, the training data 304 includes labeled data that identifies the correct outcome of the input data. In the context of search queries, the training data includes query-document "pairs" 306. Such query-document pairs typically contain an example search query, a document relevant to the query (called Doc+) and a document not relevant to the query (called Doc-). Doc+ is a positive example of the search query and Doc- is a negative example of the search query. In the context of the present disclosure, a query-document pair will refer to different combinations of a query and an example document. For example, a query-document pair can include a query and an associated Doc+. A query-document pair can also include a query and an associated Doc-. Finally, a query-document pair can include a query, an associated Doc+, and an associated Doc-.

[0039] The machine learning model is trained via a training process 308 using the training data 304. The training process 308 is known in the art and adjusts the weights and other parameters of the machine learning model such that a loss function expressing a model error is minimized with respect to the training data. The training process presents each of the query-document pairs in the training data set 304 to the machine learning model, evaluates the output assessment, compares it to the correct output, and adjusts the weights and other parameters of the model based on the comparison. This process is repeated for the training data set 304 such that the loss function is minimized. The result of the training is a trained machine learning model 312.

[0040] When the machine learning model 312 is used to perform an assessment, new data 310 (e.g., a new query and documents to be evaluated with respect to the query) is provided as input to the trained machine learning model 312 and the trained machine learning model 312 generates an assessment 314 as output. In this case, the assessment 314 will be a ranking score for the documents.

[0041] Machine learning models unrelated to the query item

[0042] As mentioned above, modifying machine learning models to exploit query item independence runs counter to conventional wisdom and practice. However, embodiments of this disclosure utilize query item independence machine learning models. This leads to the surprising and unexpected results discussed herein.

[0043] Machine learning models used for ranking do not assume query item irrelevance. Instead, they learn complex matching functions to compare candidate documents with the complete query discussed above. The parameters of a traditional machine learning model φ are typically learned by minimizing a loss function of the form discussed above:

[0044]

[0045] in,

[0046] In these equations, q is the query, and d is the query. + This represents the document related to the query (i.e., doc+), and d - This represents documents irrelevant to the query (i.e., doc-). Loss Different forms of loss can be used, such as RankNet loss (discussed in Learning to Rank Using Gradient Descent, published by Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender in the proceedings of the 22nd International Conference on Machine Learning in ACM, pages 89-96, 2005), hinge loss (discussed in Large Margin Rank Boundaries for Ordinal Regression, published by Herbrich, Ralf & Graepel, Thore & Obermayer, and Klaus in Advances in Large Margin Classifiers, page 88, 2000), or some other loss.

[0047] RankNet loss is a logarithmic loss of the following form:

[0048]

[0049] As given in equation (2) above, And σ is a parameter of the loss function.

[0050] The hinge loss is a loss function of the form:

[0051]

[0052] where, as given in equation (2) above, and ∈ is a parameter of the loss function.

[0053] Given a machine learning model φ, the corresponding query- independent machine learning model is defined as:

[0054] Φ q，d =∑ t∈q W i φ t，d (5)

[0055] where t is a query item that is present in the query q.

[0056] d is a document under consideration.

[0057] φ t，d is the output of the machine learning model with input parameters t and d. This is the ranking score contribution factor for item t with respect to document d.

[0058] W i is an optional weighting factor for the i-th ranking score contribution factor.

[0059] The new model Φ can retain the same architecture as the original model φ, but estimate the relevance of document d independently with respect to each query item. However, even though the new model can have the same architecture as the original model, it is often possible to simplify the architecture because the new model only needs to deal with a single input item rather than an entire query phrase. If such a simplification is made, the computational burden of the new model can often be reduced relative to the original model.

[0060] The parameters of the new model Φ are learned using a modified loss function of the form:

[0061]

[0062] where,

[0063] The loss can take different forms, such as the ranknet loss, the hinge loss, or some other loss.

[0064] The ranknet loss for the query-independent model will take the form:

[0065]

[0066] where, as given in equation (7) above, and σ is a parameter of the loss function.

[0067] The hinge loss for a query-independent model will take the following form:

[0068]

[0069] where, as given in equation (7) above, and ∈ is a parameter of the loss function.

[0070] Given a collection of documents (sometimes referred to as a corpus) C and a vocabulary of query items V, a trained machine learning model can be used to pre-compute each of the ranking score contribution values for all items t e V and documents d e C. From these ranking contribution values, the query-independent machine learning model for a query q and a document d can be computed as a weighted sum of the ranking contribution values, as given in equation (5). Additionally, the pre-computed ranking contribution values can be stored in an inverted index to perform retrieval from the complete collection using the learned relevance function Φ.

[0071] A number of information retrieval data structures can be used to store the pre-computed ranking contribution values φ t，d The values of φ t，d are always positive. If employed in embodiments of the present disclosure, negative values can prevent proper functioning of these information retrieval data structures. However, the formulas of the present disclosure do not guarantee that the values of φ t，d are always positive. To eliminate the possibility of negative values and to ensure proper functioning of such data structures, a rectified linear unit as an activation function of the model output can be used, such as discussed in Deep Sparse Rectifier Neural Networks by Xavier Glorot, Antoine Bordes, and Yoshua Bengio, published in Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pages 315-323, 2011.

[0072] The ability to pre-compute the ranking contribution values φ t，d allows machine learning document selection and ranking to be utilized in the computing-limited environments described herein. This is a significant advantage of the present approach, as one of the problems in utilizing state-of-the-art machine learning methods in large-scale search systems or computing-limited environments is the computational burden and associated latency in these applications. Thus, embodiments of the present disclosure provide a significant technical advantage over prior art approaches.

[0073] In practice, the total number of combinations of query terms t and documents d can be very large for large scale search environments. However, in this case, embodiments of the present disclosure are able to impose additional constraints on the <t,d> combinations to be evaluated and simply assume no contribution from the remaining pairs. Essentially, this means discarding particular term-document combinations <t,d> from the vocabulary V and / or corpus C or both, discarding terms t and / or discarding documents d, respectively.

[0074] Discarding terms from consideration can be achieved by discarding terms and / or term-document combinations that do not provide discriminatory effects for one reason or another, such as by evaluating term-frequency, inverse document frequency, or a combination thereof. Term-frequency and inverse document frequency are well known metrics. Term-frequency is a count of how often a term occurs in a document. Inverse document frequency is a measure of how much information a term provides, i.e., whether the term is rare or common across all documents. Several formulas for term frequency and inverse document frequency are known, and any of the formulas can be used to help identify terms and / or documents to discard from consideration. For example, terms that are present in all (or almost all) documents have little discriminatory effect on ranking documents according to relevance to a query containing the term. Thus, terms with high term-frequency values (e.g., above a threshold) can be excluded from consideration. Similarly, common terms, as measured by inverse document frequency, can be excluded from consideration. This can again be measured by comparing inverse document frequency values to a threshold and excluding those terms that exceed the threshold.

[0075] Term frequency and inverse document frequency are often combined into a term frequency-inverse document frequency signal by multiplying the two metrics together. This signal can also be used to identify common terms and have little discriminatory effect on ranking values for documents. Comparing the term frequency-inverse document frequency signal to a threshold and excluding those terms that exceed the threshold can be used to exclude terms from consideration.

[0076] There are other mechanisms to exclude item-document combinations, items, and / or documents from consideration to constrain the number of combinations evaluated. Using the representative example of words as query items, words can often have the same or similar meanings. Words with the same or similar meanings can be assigned the same query contribution score. In this sense, words in a "cluster" can be considered the same in terms of query contribution score, and only one score needs to be computed and / or stored for the entire cluster. This approach effectively discards query-items from computation and / or storage. For example, a single query item can be used to compute the query contribution score for that item, and then that score can be stored for all query items in the cluster. As an alternative, the score can be stored for the entire cluster, such as by using a cluster identifier. The index can then be used to convert query items to cluster identifiers, and the query contribution score can be retrieved based on the cluster identifier.

[0077] There are several ways to identify words with the same or similar meanings. For example, words that are very close in an embedding space also have similar meanings. Thus, a nearest neighbor algorithm can be used to group query items into clusters, where all members of the cluster are assigned the same query contribution score. Other methods can be used to identify similarities between query items, such as dictionary-definition synonyms, ontology strategies (such as "Word-Net" which includes a graph of relationships between words), and the like.

[0078] In cases where the query items are not words, but some other subset of query strings, similar strategies can be used.

[0079] Training a machine learning model independent of query items

[0080] Turning now to Figures 4 to 6 the discussion of training a machine learning model independent of query items Φ is presented.

[0081] As mentioned above, the machine learning model independent of query items of the present disclosure is trained using query item-document pairs that include one or both of a query item, a positive relevant document (doc+) example, and / or a negative relevant document (doc-) example. One way to gather such training data is to convert the training data that is used to train the query phrase machine learning model into training data suitable for training the machine learning model independent of query items. Figure 4 One such example 400 is illustrated.

[0082] In this example, the single entry of the query phrase-document 402 is converted into multiple query term-document 408 entries. This is accomplished by training the conversion method 406. How the method 406 operates is that the next query-document pair 402 in the training data set is first retrieved. The query portion of the query-document pair 402 is evaluated to identify all query terms in the query phrase. How the query terms are extracted depends on what the query terms are (e.g., query term definition). For example, in the case where the query terms are single words, this can be accomplished by looking for term separation indicators such as spaces or using other mechanisms to extract individual words from the phrase. In the case where the query terms are sub-words, fixed length strings, or some other portion of the query string, then other mechanisms are used. Extracting fixed length strings is self-explanatory in that the query is broken into fixed length strings, padded as necessary. In the case where the query terms are n-grams, known methods of dividing the query string into n-grams can be used. In the case where the query terms are divided by characters or symbols, the query string can be parsed based on the location of the characters or symbols within the query string and broken into query terms. Other methods can also be used, depending on the particular query term definition being used.

[0083] Once the individual query terms are extracted from the query phrase, any stop words or other terms to be excluded from consideration (such as explained above) can be excluded from the set of extracted query terms.

[0084] For each of the remaining query terms, each query term is paired with an example document (e.g., doc+ and / or doc-) from the original query-document pair 402 to result in multiple query term-document pairs 408.

[0085] As an example, consider the query "plastic recycling strategy" associated with a query-document pair with a doc+ example and a doc- example, as indicated by the query-document pair 404. In this case, the query would be analyzed and the query terms "plastic," "recycling," and "strategy" would be extracted.

[0086] Assuming that the analysis of the query terms does not indicate that any of these terms should be discarded from consideration, the doc+ and doc- examples will be paired with each of the three query terms to produce term-document pairs: {“plastic”, doc+, doc-} 410, {“recycle”, doc+, doc-} 412, {“strategy”, doc+, doc-} 414. These three query term-document pairs will then be added to the training dataset for the query term-agnostic machine learning model. The same strategy will be used if the original query-document pair contains the query and only contains the associated doc+ or doc-, pairing the query term with the associated doc+ or doc-, as the case can be.

[0087] Figure 5 An example method 500 for training a machine learning model is illustrated in accordance with some aspects of the present disclosure. This operation is similar to the previously discussed methods in that Figure 3 the loss function used in the training process 508 to train the query term-agnostic machine learning model and the actual training data 502 used by the training process 508 to train the query term-agnostic machine learning model. Additionally or alternatively, the actual query term-agnostic machine learning model used can be simplified to exclude aspects that are not needed when the model is only using a single query term, as described herein.

[0088] In other words, what turns a machine learning model into a trained query term-agnostic machine learning model is: 1) an optional simplification of the model to exclude aspects that are not needed for a single input query term; 2) that the training data used is query term-document pairs (rather than query-document pairs); and 3) the loss function used during training.

[0089] The optional model simplification is performed by excluding aspects that are not needed all of the time that a single input query term is to be input. There is no general method to do this, but one skilled in the art will be able to do this by analyzing the machine learning model architecture under the assumption of a single input term. What is excluded, if anything, depends on the actual architecture of the machine learning model. However, generally only aspects of the term combination can be excluded entirely. Neural network layers within the machine learning model can also be simplified generally.

[0090] For example, obtaining the correct query term-document pairs 506 of the training data 504 can be done as Figure 4 described.

[0091] In another embodiment, the query term clustering methods described above can be utilized. For example, a machine learning model can be trained using query term cluster-document pairs. Using the representative example of words as query terms, words can often have the same or similar meanings. Words with the same or similar meanings can be grouped together in a cluster, as described above in connection with reducing the number of query terms, documents, and / or query term-document pairs. For training purposes, the words in a cluster can be treated as the same. Thus, a query term can be checked for membership in a cluster. If the query term is a member of the cluster, one of two methods can be employed. In a first method, a query term-document pair for each query term in the cluster is created by pairing each query term in the cluster with the doc+ and / or doc- associated with the query phrase. For example, if the query-document pair is {“plastic material”, doc+} and “plastic” belongs to a query term cluster with “pliable”, “elastic”, and “ductile”, the resulting query term-document pairs would be: {“plastic”, doc+}, {“pliable”, doc+}, {“elastic”, doc+}, {“ductile”, doc+}, and {“material”, doc+}.

[0092] In a second method, a single representative term of the cluster can be used for training any term in the cluster. Thus, with the same query term cluster above, if “plastic” is the representative term, the query-document pair {“ductile material”, doc-} would result in query term-document pairs of: {“plastic”, doc-} and {“material”, doc-}.

[0093] The loss function used by method 508 is given by equations (6) and (7), with equations (8) and (9) as specific examples discussed above. With this loss function, the machine learning model, independent of the query term, is trained using methods well known in the art to minimize the selected loss function(s).

[0094] The trained machine learning model 512 can be used to compute the ranking contribution value φ t，d for a given term t and document d. Thus, the new data 510 is the term t and the document d, and the evaluation is the corresponding ranking contribution value φ t，d .

[0095] With these changes, the remainder of the figures operate as described Figure 3 with the necessary modifications.

[0096] The pre-computed term-document index

[0097] The ranking contribution value φ t，dcan be pre-computed and stored in the query term-document index. This aspect, including the use of an inverted index on the query term-document index, has been discussed to some extent above. Figure 6 Figure 6 illustrates an example diagram 600 depicting a pre-computed term- document index, in accordance with some aspects of the present disclosure.

[0098] A machine learning model 602 is selected. The machine learning model can be any machine learning model designed to produce a ranking score for a given document and an input query. An example state of the art deep learning model is discussed below in the section presenting surprising test results achieved from embodiments of the present disclosure. However, any desired machine learning model can be used.

[0099] In optional operation 604, the selected machine learning model 602 can be simplified by analyzing the machine learning model 602 under the condition that only a single term, not the entire query phrase, will be input, and excluding and / or simplifying any aspects of the machine learning model 602 that are unnecessary. This process has been discussed above. The result of operation 602 is a simplified machine learning model 606.

[0100] The machine learning model 602 or the simplified machine learning model 606, as the case can be, is then trained in operation 608 using training data 610 as discussed above. As also discussed above, the training data 610 includes query term-document pairs, which include a query term and an associated doc+ and / or an associated doc. As discussed above, the training process minimizes one or more selected query term- independent losses.

[0101] The resulting trained machine learning model 612 can be used to compute a query term-document index 616 using a set of query terms 614 and a corpus of documents 615. A representative process is discussed in Figure 7 The process computes a ranking contribution value φ t，d for a given term t and document d. t，d The ranking contribution values φ

[0102] The query term-document index 616 can be sent to one or more systems for document selection and ranking. Because the query terms are pre-computed, and the document ranking scores are computed by a weighted sum of the ranking contribution values, the query term-document index 616 can be used in a compute-limited context, such as a user device 618. It can also be deployed as part of any search system 620.

[0103] Figure 7An example flowchart 700 showing how a pre-computed item-document index is computed is illustrated in accordance with some aspects of the present disclosure. The method starts in operation 702 and proceeds to an optional loop which starts in operation 704 and includes operation 706 and ends in operation 708. If the input to the process is a list of queries rather than a set of individual query items, the loop starting in operation 704 loops over all queries in the query set and extracts individual query items from the queries and assembles a list of unique query items (operation 706) such that the ranking contribution value φ t，d can be computed.

[0104] If the input is a set of query items rather than a set of queries, the method 700 can start in operation 710 which loops over all query items.

[0105] Operation 710 starts a loop which loops over all items in the set to be used for the query item-document index.

[0106] Operation 712 starts a loop over all documents in the corpus for the query item under consideration.

[0107] Operation 714 computes the ranking contribution value φ t，d for the query item and document combination for the current loop iteration. t，d Also referred to herein as the query item-document ranking score or more simply the item-document score.

[0108] Operation 716 stores the ranking contribution value φ t，d in the item-document index.

[0109] Operations 718 and 720 close the corresponding loops to select the next document, query item and query respectively.

[0110] The method ends in operation 722.

[0111] As a representative example of how this works, consider a set of three queries (q1...q3):

[0112] (1) "plastic recycling strategy"

[0113] (2) "garbage recycling options"

[0114] (3) "sort garbage for recycling"

[0115] and a corpus of n documents D1...Dn.

[0116] The loop starting in operation 704 will iterate over the three queries ql...q3 and extract the unique query terms. In this case, the list of unique query terms is in the first column of Table 1.

[0117] Table 1: Term-Document Scores

[0118]

[0119]

[0120] The loop starting in operation 708 will iterate over each term in the query. The loop starting in operation 710 will iterate over all documents D1...Dn. Operation 714 computes the term-document score, and operation 716 stores the score in the index. Let S t，d be the score for term t and document d, after the method completes, Table 1 will look like Table 2.

[0121] Table 2: Term-Document Scores

[0122] D1 D2 ... Dn Plastic [SA 1，1 ]] [SA 1，2 ]] ... [SA 1，n ]]> Recycling [SA 2，1 ]]> [SA 2，2 ]]> ... [SA 2，n ]] Strategy [SA 3，1 ]] [SA 3，2 ]] ... [SA 3，n ]]> Trash [SA 4，1 ]] [SA 4，2 ]]> ... [SA 4，n ]] Option [SA 5，1 ]] [SA 5，2 ]] ... [SA 5，n ]] Classification [SA 6，1 ]] [SA 6，2 ]] ... [SA 6，n ]]

[0123] Each score S t，d is a ranking contribution value φ t，d computed using a trained query-term independent machine learning model (e.g., 512, 612).

[0124] Using Machine Learning to Select and Rank Documents

[0125] Figure 8 Figure illustrates an example flowchart 800 showing the use of pre-computed term- document scores to compute document rankings, according to some aspects of the present disclosure. For purposes of illustration, Table 2 above will be used as an example of a pre-computed query term index and scores therein. Additionally, the sample query “sort plastics before recycling” will be used as a sample query.

[0126] The method begins in operation 802, and proceeds to operation 804, where a query is received. In the example used herein, the query is “sort plastics before recycling.”

[0127] Operation 806 begins a loop over each document in the corpus. In our example, D1...Dn.

[0128] Operation 808 begins a loop over each term in the query. For our sample query, the stop word “to” is discarded, and the loop starting in operation 808 will loop once for the terms “sort,” “plastics,” “before,” and “recycling.”

[0129] The document loop and the query term loop can be executed in any order. However, if the documents are outer loops, then the score aggregation operation 814 can easily aggregate the scores for each document, as explained below.

[0130] Operation 810 looks up the corresponding score for the query term under consideration and the document in the term-document index. On the first pass through the loop started in operation 806 and operation 808, the document under consideration will be Dl, and the term under consideration will be "classify." The corresponding score from Table 2 is S 6,1 .

[0131] Operation 812 represents optional weighting factors for the terms and / or the documents. In some embodiments, it can be desirable to adjust the weighting factors based on knowledge that is currently known but was not known at the time the scores were pre-computed. For example, over time, through observation of user behavior, the importance of certain words can become apparent. Assume that a user typed the following queries in order:

[0132] (1) "classify rates"

[0133] (2) "plastic sorting machine for recycling"

[0134] (3) "sort plastics before recycling"

[0135] Further assume that after the first two queries, the user did not find what they were looking for (e.g., determined by the user running the next query without looking at the search results for the previous queries). It can be assumed that the new terms in the current query are more important than the repeated terms in the previous queries. In this case, it can be desirable to give the new terms a greater weight than the repeated terms. Similarly, perhaps some words are not very discriminative in some contexts, but are more discriminative in other contexts. The context can then be used to identify which terms should be given more weight and / or less weight in the context of a given context.

[0136] Operation 814 aggregates the scores for the documents of the current term.

[0137] The loop ends in operations 816 and 818. After the loop ends in operation 816, the aggregated score for the document under consideration will be known. Using our example queries, and assuming that operation 812 did not adjust the scores, the aggregated scores will be as shown in Table 3.

[0138] Table 3: Aggregated document scores

[0139]

[0140]

[0141] Operation 820 can rank the documents by the aggregated scores and return a subset of the documents as a ranked set of documents. The subset can be determined by a selection criterion based at least in part on the aggregated scores. Thus, documents corresponding to the top N documents can be selected and returned. In this context, the subset is all documents or less than all documents.

[0142] Test Results

[0143] To test the performance of embodiments of the present disclosure, a known passage ranking dataset was selected and the state-of-the-art deep neural network model was adjusted and trained according to the methods described herein. The performance of the model used in the query-item agnostic manner of the present disclosure was compared to the performance of the model used in the traditional manner. In making this comparison, it was expected that the model used in the traditional manner would outperform the model used in the query-item agnostic manner of the present disclosure in terms of ranking and selection accuracy, but the model used in the query-item agnostic manner proposed in the present disclosure would outperform the model used in the traditional manner in terms of computational efficiency. In other words, it was expected that the approach employed by the present disclosure would be more efficient from a computational perspective, but less efficient from an accuracy perspective. Surprisingly, there was little statistical significance between the accuracy of embodiments in the present disclosure and the state-of-the-art deep learning methods. Additionally, the pre-computed item-document index had a significant computational advantage over the state-of-the-art deep learning methods. Thus, the test data showed no trade-off in accuracy for the expected computational efficiency.

[0144] The test used the MS MARCO dataset, as described in Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. MS MARCO: A Human Generated Machine Reading Comprehension Dataset. arXiv:1611.09268v3 [cs.CL]. At the time of writing, the dataset was available from http: / / www.msmarco.org. The dataset includes over 1 million queries extracted from search logs, 8.8 million passages extracted from web documents, and 400,000 positively labeled query-passage pairs for training. The search log was extracted from search logs, 8.8 million passages extracted from web documents, and 400,000 positively labeled query-passage pairs for training, and is one of the few large datasets available today for benchmarking deep neural information retrieval methods.

[0145] The MS MARCO dataset contains a passage ranking task, each query includes one thousand passages, and the information retrieval model being evaluated should be re-ranked. For each query, one or several passages are annotated by human editors as containing an answer relevant to the query. The ranking list produced by the model is evaluated against the ground truth annotations using the Mean Reciprocal Rank (MMR) metric. The MS MARCO training dataset is used to train all models used in the tests.

[0146] The dataset also includes a leaderboard (with 18 entries as of March 3, 2019) which can be a useful catalog of methods showing the state-of-the-art information retrieval performance. In other words, the leaderboard allows methods with the highest accuracy on the dataset to be adopted for comparison with embodiments of the present disclosure.

[0147] These tests compare three different state-of-the-art deep learning neural network methods for information retrieval. The three models are BERT, Duet, and CKNRM, and are described in more detail below.

[0148] BERT achieves the state-of-the-art performance on the MS MARCO dataset as described in Rodrigo Nogueira and Kyunghyun Cho, Passage Re-Ranking with BERT, arXiv: 1901.04085v2 [cs.IR]. The authors report retrieval performance based on fine-tuning of the BERT base and BERT large models pre-trained on the MS MARCO dataset. The tests conducted use the BERT base model.

[0149] The Duet model estimates the relevance of a document to a query by combining (1) a pattern that checks for exact matches of query terms in the document, and (2) a computation of the similarity between learned latent representations of the query and the document. The Duet model is described in Bhaskar Mitra, Fernando Diaz, and Nick Craswell, Learning to Match Using Local and Distributed Representations of Text for Web Search, Proc. WWW, pp. 1291-1299, 2017. A version of Duet that produces results on the MS MARCO leaderboard modifies the original Duet of the above paper. At the time of writing, these modifications can be found at https: / / github.com / dfcf93 / MSMARCO / blob / master / Ranking / Baselines / Duet.ipynb. This version of Duet is used for comparison purposes.

[0150] The CKNRM model combines kernel pooling based soft matching and a convolutional architecture for comparing n-grams. CKNRM uses kernel pooling to extract ranking signals from the interaction matrix of query and passage unigrams, bigrams, and trigrams. A publicly available implementation of this model found at https: / / github.com / thunlp / Kernel-Based-Neural-Ranking-Models at the time of writing is used for comparison purposes.

[0151] Table 4 below compares three models, which are in their original configuration (i.e., using full queries) and trained and used as query term agnostic models, as specified herein.

[0152] Table 4: Ranking effectiveness comparison

[0153]

[0154] In Table 4, “full” indicates models trained and used in the prior art way (i.e., with full queries). “Query term agnostic” indicates models trained and used as described in embodiments of the present disclosure (i.e., in the query term agnostic way described herein). BM25 is a non-machine learning method of ranking search results that uses the assumption of query term agnosticism and is well known in the art. It is presented here as a comparison to machine learning methods.

[0155] The Duet and CKNRM based models were trained five and eight times, respectively, using different random seeds for the model parameters. Table 4 reports the mean and median MRR. For the BERT based model, a single training and evaluation run was performed. As mentioned in the table, there is little or no statistically significant difference in effectiveness when using the models in the embodiments of the present disclosure compared to using the models on the entire query phrase. The BERT model does show a statistically significant difference in a single run, but it is currently unknown how the statistics of multiple training runs would compute. As mentioned herein, this is a surprising result and unexpected given the nature and purpose of machine learning models. This means that the embodiments of the present disclosure have little or no accuracy, but gain computational efficiency.

[0156] Table 5 below shows the results of a smaller scale retrieval experiment based on a random sample of 395 queries from the test set. The Duet model was selected to be evaluated and used in a query independent manner in the embodiments described herein to pre-compute ranking contribution values φ t，d (e.g. a pre-computed item-document index). To constrain the dimensionality of the item- document index, the ranking contribution values were only computed for items that 1) occur at least once in a document and 2) do not occur in more than 5% of the documents in the document collection. In the table, BM25 model, BM25 model with Duet as final re-ranker (used for initial selection) and Duet model independent of query items.

[0157] Table 5: Retrieval results

[0158]

[0159] As mentioned here, the recall @1000 of the Duet independent of items is improved by 6.25% compared to BM25 and BM25+Duet. This is a statistically significant improvement. Using the full Duet model in the above test is computationally infeasible because the full Duet model has to be run for every query and document combination.

[0160] Example machine architectures and machine readable media

[0161] Figure 9 A representative machine architecture suitable for implementing a system or the like or for performing a method disclosed herein is illustrated. Figure 9The machine of FIG. 1 is shown as a standalone device, which is suitable for implementation of the above concepts. For the server aspects described above, multiple such machines operating in a data center, part of a cloud architecture, etc. can be used. Not all of the illustrated functionality and devices are utilized in the server aspects. For example, while a system, device, etc. used by a user to interact with the server and / or cloud architecture can have a screen, touch screen input, etc., the server typically does not have a screen, touch screen, camera, etc. and typically interacts with the user through a connected system having appropriate input and output aspects. Thus, the following architecture should be considered to encompass multiple types of devices and machines, and individual aspects can or can not be present in any particular device or machine, depending on its form factor and use (e.g. a server rarely has a camera, while a wearable device rarely includes a disk). However, Figure 9 The example explanations are suitable to allow those skilled in the art to determine how to implement the previously described embodiments with appropriate combinations of hardware and software, and to make appropriate modifications to the illustrated embodiments of the specific devices, machines, etc. used.

[0162] While only a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

[0163] Examples of the machine 900 include, at least, one processor 902 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an advanced processing unit (APU), or a combination thereof), one or more memory, such as a main memory 904, a static memory 906, or other type of memory, which communicate with each other via a link 908. The link 908 can be a bus or other type of connection channel. The machine 900 can include other optional aspects, such as a graphics display unit 910 including any type of display. The machine 900 can also include other optional aspects, such as an alphanumeric input device 912 (e.g., a keyboard, a touch screen, etc.), a user interface (UI) navigation device 914 (e.g., a mouse, a trackball, a touch device, etc.), a storage unit 916 (e.g., a disk drive or other storage device(s)), a signal generation device 918 (e.g., a speaker), sensor(s) 921 (e.g., a global positioning sensor, accelerometer(s), microphone(s), camera(s), etc.), an output controller 928 (e.g., a wired or wireless connection to one or more other devices to connect and / or communicate with, such as a universal serial bus (USB), near field communication (NFC), infrared (IR), serial / parallel bus, etc.), and a network interface device 920 (e.g., wired and / or wireless) to connect and / or communicate over one or more networks 926.

[0164] Executable instructions and machine storage media

[0165] Various memories (i.e., 904, 906 of the processor(s) 902 and / or the storage unit 916) and / or storage units 916 can store one or more instruction sets and data structures (e.g., software) 924 embodying or utilized by any one or more of the methodologies or functions described herein. These instructions, when executed by the processor(s) 902, implement various operations to achieve the disclosed embodiments.

[0166] As used herein, the terms "machine-storage medium," "device-storage medium," "computer-storage medium" and "device-storage medium" mean the same thing and can be used interchangeably in this disclosure. The terms refer to a single or multiple storage devices and / or media (e.g., a centralized or distributed database, and / or associated caches and servers) that store executable instructions and / or data structures. Thus, the term should be understood to include the presence of one or more storage devices and / or media storing executable instructions and / or data structures. The term includes, for example, solid-state memories, optical and magnetic media, including a non-exhaustive list of: nonvolatile memory such as semiconductor memory devices (e.g., Erasable Programmable Read-Only Memories (EPROM), electrically Erasable Programmable Read-Only Memories (EEPROM), FPGAs, and flash memory devices); magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The term machine-storage medium does not encompass transitory media, which are instead included in the term "signal medium" as discussed below.

[0167] Signal Medium

[0168] The term "signal medium" should be taken to include any form of modulated data signal, carrier wave, and so on. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.

[0169] Computer-Readable Medium

[0170] The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and can be used interchangeably in this disclosure. The terms include a machine-storage medium and a signal medium. The terms include both a storage device / media and a carrier wave.

[0171] Example Embodiments

[0172] Example 1. A method for information retrieval, comprising:

[0173] providing access to a pre-computed item-document index, each ranking score in the pre- computed item-document index provided by a machine learning model trained using a training data set comprising query item-document pairs;

[0174] receiving a query;

[0175] splitting the query into constituent query items of the query;

[0176] for each constituent query item:

[0177] retrieving a corresponding ranking score for each document in the set of documents based on the constituent query item from the pre-computed item-document index; and

[0178] aggregating the ranking scores of the constituent query items by document;

[0179] ranking the set of documents by the corresponding aggregated ranking scores;

[0180] selecting a subset of the ranked set of documents as a set of search results; and

[0181] returning the set of search results in response to the query.

[0182] Example 2. The method of example 1, wherein the pre-computed item-document index is created using a method comprising:

[0183] identifying a set of queries;

[0184] for each query in the set of queries:

[0185] splitting the query under consideration into constituent query items of the query; and

[0186] for each constituent query item:

[0187] computing a ranking score for each document in the set of documents using the trained machine learning model; and

[0188] storing the ranking scores in the pre-computed item-document index.

[0189] Example 3. The method of example 1, wherein the pre-computed item-document index is created using a method comprising:

[0190] identifying a set of queries;

[0191] splitting each query in the set of queries into constituent query items of the query to form a set of query items;

[0192] selecting a subset of query items from the set of query items;

[0193] selecting a subset of documents from the collection of documents;

[0194] for each query item in the subset of query items:

[0195] computing a ranking score for each document in the collection of documents using the trained machine learning model; and

[0196] storing the ranking scores in a pre-computed item-document index.

[0197] Example 4. The method of example 1, 2, or 3, wherein the pre-computed item- document index is received from another system.

[0198] Example 5. The method of example 1, 2, 3, or 4, wherein the pre-computed item- document index is computed on the system executing the method.

[0199] Example 6. The method of example 1, 2, 3, 4, or 5, wherein the machine learning model is trained using a method comprising:

[0200] identifying first training data comprising a collection of query-document pairs;

[0201] splitting each query-document pair into a collection of query item-document pairs, thereby creating second training data;

[0202] identifying a loss function that is independent of the query item;

[0203] training the machine learning model by minimizing the loss function that is independent of the query item over the second training data.

[0204] Example 7. The method of example 6, wherein the loss function that is independent of the query item is a weighted sum of item-document scores.

[0205] Example 8. The method of example 6, wherein the loss function that is independent of the query item is a function of a weighted sum of differences between item-document scores for positive document examples and item-document scores for negative document examples.

[0206] Example 9. The method of example 1, 2, 3, 4, 5, 6, 7, or 8, wherein each item- document pair comprises:

[0207] a query item from the query-document pair; and

[0208] one of a positive document example from the query-document pair or a negative document example from the query-document pair.

[0209] Example 10. The method of example 1, 2, 3, 4, 5, 6, 7, or 8, wherein each item- document pair comprises:

[0210] query terms from the query-document pair;

[0211] positive document examples from the query-document pair; and

[0212] negative document examples from the query-document pair.

[0213] Example 11. The method of example 1, 2, 3, 4, 5, 6, 7, 8, 9, or 10, further comprising selecting the set of documents from a larger set of documents such that the set of documents is a subset of the larger set of documents.

[0214] Example 12. The method of example 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, or 11, further comprising:

[0215] dynamically selecting a weight for the ranking scores that make up the query term; and

[0216] wherein the aggregating includes weighting the ranking scores that make up the query term by the dynamically selected weight.

[0217] Example 13. The method of example 12, wherein the dynamically selected weight is selected based on a context of the query phrase.

[0218] Example 14. An apparatus comprising means for performing the method as in any of the preceding examples.

[0219] Example 15. A machine-readable storage including machine-readable instructions for implementing the method or implementing the apparatus as in any of the preceding examples when executed.

[0220] Example 16. A method for information retrieval, comprising:

[0221] providing access to a pre-computed term-document index, each ranking score in the pre-computed term-document index provided by a machine learning model trained using a training data set, the training data set including query term-document pairs;

[0222] receiving a query;

[0223] splitting the query into constituent query terms of the query;

[0224] for each constituent query term:

[0225] retrieving a corresponding ranking score for each document in the set of documents based on the constituent query term from the pre-computed term-document index; and

[0226] aggregating the ranking scores of the constituent query terms by document;

[0227] ranking the collection of documents by corresponding aggregate ranking scores;

[0228] selecting a subset of the ranked collection of documents as a search result collection; and

[0229] returning the search result collection in response to the query.

[0230] Example 17. The method of example 16, wherein the pre-computed item- document index is created using a method comprising:

[0231] identifying a collection of queries;

[0232] for each query in the collection of queries:

[0233] splitting the query under consideration into constituent query items of the query; and

[0234] for each constituent query item:

[0235] computing a ranking score for each document in the collection of documents using the trained machine learning model; and

[0236] storing the ranking scores in the pre-computed item-document index.

[0237] Example 18. The method of example 16, wherein the pre-computed item- document index is received from another system.

[0238] Example 19. The method of example 16, wherein the pre-computed item- document index is computed on the system executing the method.

[0239] Example 20. The method of example 16, wherein the machine learning model is trained using a method comprising:

[0240] identifying first training data comprising a collection of query-document pairs;

[0241] splitting each query-document pair into a collection of query item-document pairs, thereby creating second training data;

[0242] identifying a loss function that is agnostic to query items;

[0243] training the machine learning model by minimizing the loss function that is agnostic to query items over the second training data.

[0244] Example 21. The method of example 20, wherein the loss function that is agnostic to query items is a weighted sum of item-document scores.

[0245] Example 22. The method of example 20, wherein the loss function that is independent of the query items is a function of a weighted sum of the difference between the item-document scores of positive document examples and the item-document scores of negative document examples.

[0246] Example 23. The method of example 20, wherein each item-document pair comprises:

[0247] a query item from the query-document pair; and

[0248] one of a positive document example from the query-document pair or a negative document example from the query-document pair.

[0249] Example 24. The method of example 20, wherein each item-document pair comprises:

[0250] a query item from the query-document pair;

[0251] a positive document example from the query-document pair; and

[0252] a negative document example from the query-document pair.

[0253] Example 25. The method of example 16, further comprising selecting the set of documents from a larger set of documents such that the set of documents is a subset of the larger set of documents.

[0254] Example 26. The method of example 16, further comprising:

[0255] dynamically selecting weights for the ranking scores that make up the query items.

[0256] Example 27. A system comprising a processor and computer-executable instructions that, when executed by the processor, cause the system to perform operations comprising:

[0257] creating a pre-computed item-document index with operations comprising:

[0258] identifying a set of query items;

[0259] providing access to a machine learning model trained using a training data set comprising query item-document pairs;

[0260] for each query item in the set of query items:

[0261] computing a ranking score for each document in the set of documents using the trained machine learning model; and

[0262] storing the ranking scores in the pre-computed item-document index;

[0263] Selecting a set of search results using operations comprising:

[0264] Receiving a query;

[0265] Splitting the query into constituent query items of the query;

[0266] For each constituent query item:

[0267] Retrieving a corresponding ranking score for each document in a set of documents based on the constituent query item from a pre-computed item-document index; and

[0268] Aggregating the corresponding ranking scores of the constituent query items by document;

[0269] Ranking the set of documents by the corresponding aggregated ranking scores;

[0270] Selecting a subset of the ranked set of documents as the set of search results; and

[0271] Returning the set of search results in response to the query.

[0272] Example 28. The system of example 27, further comprising selecting the set of documents from a larger set of documents such that the set of documents is a subset of the larger set of documents.

[0273] Example 29. The system of example 27, further comprising training the machine learning model using operations comprising:

[0274] Identifying first training data comprising a set of query-document pairs;

[0275] Splitting each query-document pair into a set of query item-document pairs, thereby creating second training data;

[0276] Identifying a loss function that is independent of the query items;

[0277] Training the machine learning model by minimizing the loss function that is independent of the query items over the second training data.

[0278] Example 30. The system of example 29, wherein the loss function that is independent of the query items is a function of a weighted sum of item-document scores.

[0279] CONCLUSION

[0280] In view of the principles of the application and the many possible embodiments that can be applied thereto, it should be recognized that the examples described herein are merely illustrative and should not be considered limiting on the scope of the application. Thus, the present application described herein contemplates all such embodiments as can come within the scope of the following claims and any equivalents thereto.

Claims

1. A method for information retrieval, comprising: It can receive queries that include one item and another item; Find a ranking score for the project and document in the index, wherein the ranking score is pre-calculated by a machine learning model based on the project and document; Find another ranking score for the other item and the document in the index, wherein the other ranking score is pre-calculated by the machine learning model based on the other item and the document; Calculate the aggregate score of the document relative to the query, wherein the aggregate score is based on the aggregation of the one ranking score and the other ranking score; The documents are ranked using the aggregated scores; and Returns a ranked collection of documents in response to the query.

2. The method of claim 1, wherein the index is created using a method comprising the following steps: Identifier query set; Each query set is broken down into its constituent query items to form a query item set; Select a subset of query items from the set of query items; Select a subset of documents from the document collection; For each query item in the subset of query items: The ranking score is calculated for each document in the subset of documents using the trained machine learning model; as well as The ranking score is stored in the index.

3. The method of claim 1, wherein the index is received from another system.

4. The method of claim 1, wherein the index is calculated on the system performing the method.

5. The method of claim 1, wherein the machine learning model is trained using a method comprising the following steps: The identifier includes the first training data for the query-document pair set; Each query-document pair is split into a set of query item-document pairs to create a second set of training data; Identify loss functions that are irrelevant to the query item; as well as The machine learning model is trained by minimizing the loss function that is independent of the query item on the second training data.

6. The method of claim 5, wherein the loss function independent of the query item is a weighted sum of item-document scores.

7. The method of claim 5, wherein the loss function independent of the query item is a function of the weighted sum of the differences between the item-document scores of positive document examples and the item-document scores of negative document examples.

8. The method of claim 5, wherein each project-document pair comprises: Query items derived from query-document pairs; as well as One of the positive document examples from the query-document pair or one of the negative document examples from the query-document pair.

9. The method of claim 5, wherein each project-document pair comprises: Query items derived from query-document pairs; Example of a positive document from the query-document pair; as well as Example of a negative document from the query-document pair.

10. A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform actions, the actions including: It can receive queries that include one item and another item; Find a ranking score for the project and document in the index, wherein the ranking score is pre-calculated by a machine learning model based on the project and document; Find another ranking score for the other item and the document in the index, wherein the other ranking score is pre-calculated by the machine learning model based on the other item and the document; Calculate the aggregate score of the document relative to the query, wherein the aggregate score is based on the aggregation of the one ranking score and the other ranking score; The documents are ranked using the aggregated scores; and Returns a ranked collection of documents in response to the query.

11. The computer-readable storage device of claim 10, further comprising: Training the machine learning model, wherein training the machine learning model includes: The identifier includes the first training data for the query-document pair set; Each query-document pair is split into a set of query item-document pairs to create a second set of training data; A loss function that identifies items irrelevant to the query; and The machine learning model is trained by minimizing the loss function that is independent of the query item on the second training data.

12. The computer-readable storage device of claim 11, wherein the loss function independent of the query item is a function of the weighted sum of item-document scores.

13. The computer-readable storage device of claim 11, wherein the query-item-independent loss function is a function of the weighted sum of the differences between the item-document scores of positive document examples and the item-document scores of negative document examples.

14. The computer-readable storage device of claim 11, wherein each item-document pair comprises: Query items derived from query-document pairs; as well as A positive document example from the query-document pair or a negative document example from the query-document pair.

Images