Interactive layer neural network for search, retrieval and ranking

Abstract

Interactive layer neural networks for search, retrieval, and ranking. A language system includes a controller. The controller can be configured to receive a query and a document, tokenize the query into a sequence of query tokens and tokenize the document into a sequence of document tokens, generate a token pair matrix for each query and document token, retrieve, for each entry in the token pair matrix, a pre-computed similarity score produced by a neural conditional translation probability network, wherein the neural network has been trained in a ranking task using a corpus of paired queries and corresponding relevant documents, aggregate, via a sum product of each similarity score for a respective query, a ranking score for each document relative to each query; and output the document and the associated ranking score for the document.

Description

Technical Field

[0001] This application generally relates to the use of neural networks in multi-stage retrieval systems. More specifically, this application relates to improvements in the interaction layers of neural networks for document retrieval and document ranking in information retrieval systems. Background Technology

[0002] Linguistics is the scientific study of language, involving the analysis of language form, meaning, and context, as well as the analysis of social, cultural, historical, and political factors that influence language. Information retrieval systems, such as internet search systems, operate on a massive scale, indexing millions of web pages to make information searchable based on user-submitted queries. The goal of such search systems is to quickly identify documents relevant to a user's query and return relevant search results. Typical text retrieval systems rely on simple term / tag matching techniques to generate an initial list of candidates, which can be further reordered using a learned model. Therefore, mismatches between query and document terms negatively impact retrieval performance; this is known as the lexical gap problem. Two decades ago, Berger and Lafferty proposed reducing the lexical gap and thus improved retrieval efficiency using a dictionary translation model called IBM Model 1. IBM Model 1 performs strongly when applied to finding answers in English question-and-answer (QA) archives using questions as queries, as well as cross-linguistic retrieval. In previous work, Model 1 was trained on question-document pairs of similar length, which simplified the task of finding useful associations between query terms and terms in relevant documents. Furthermore, these models have been primarily trained in translation tasks using variants of the traditional Expectation Maximization (EM) algorithm that produces nonparametric models. Summary of the Invention

[0003] A language system includes a controller. The controller can be configured to receive queries and documents, tokenize queries into a sequence of query tokens and tokenize each document into a sequence of document tokens, convert query tokens into query embeddings and document tokens into document embeddings, generate an embedding pair matrix for each query and document embedding, compute a similarity score for each entry in the embedding pair matrix via a neural conditional translation probabilistic network, wherein the neural network has been trained on a ranking task using a corpus of paired queries and corresponding related documents, generate a ranking score for each document relative to each query by aggregating the sum of the similarity scores for the corresponding queries, rank the documents according to the ranking scores, and output the document and the document association ranking score.

[0004] A language system includes a controller. The controller can be configured to receive queries and documents, tokenize queries into a sequence of query tokens and documents into a sequence of document tokens, generate a token pair matrix for each query and document token, retrieve a pre-computed similarity score for each entry in the token pair matrix generated by a neural conditional translation probabilistic network, wherein the neural network has been trained on a ranking task using a corpus of paired queries and corresponding related documents, aggregates the sum of each similarity score for the corresponding query to generate a ranking score for each document relative to each query, and outputs the document and the document association ranking score.

[0005] A non-transitory computer-readable medium includes instructions for ranking using a neural conditional translation probabilistic network. When executed by a processor of a computing device, the instructions cause the computing device to perform operations. The operations include receiving a query and documents; tokenizing the query into a sequence of query tokens; tokenizing the document into a sequence of document tokens for each document; converting the query tokens into query embeddings and the document tokens into document embeddings; generating an embedding pair matrix for each query and document embedding; calculating a similarity score for each entry in the embedding pair matrix via a neural conditional translation probabilistic network, wherein the neural network has been trained on a ranking task using a corpus of paired queries and corresponding related documents; generating a ranking score for each document relative to each query by aggregating the sum and product of the similarity scores for each corresponding query; and outputting the order of the documents based on their ranking scores. Attached Figure Description

[0006] Figure 1 This is a block diagram of a search / retrieval system.

[0007] Figure 2 This is a flowchart of an existing sorting neural network.

[0008] Figure 3 It is a flowchart of a sorting neural network with neural conditional translation probability network and sum-product aggregation.

[0009] Figure 4 This is a flowchart of the network sparsification process for neural conditional translation probabilities, which uses a context-free label embedding network.

[0010] Figure 5 This is a block diagram of an electronic computing system. Detailed Implementation

[0011] Embodiments of this disclosure are described herein as needed. However, it will be understood that the disclosed embodiments are merely examples of the invention, which may be embodied in various forms and alternative forms. The figures are not necessarily to scale; some features may be enlarged or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to employ the embodiments in various ways.

[0012] The term "substantially" is used herein to describe disclosed or claimed embodiments. The term "substantially" may modify values ​​or relative characteristics disclosed or claimed in this disclosure. In such cases, "substantially" may specify that the value or relative characteristic it modifies is within the range of that value or relative characteristic. Within 0%, 0.1%, 0.5%, 1%, 2%, 3%, 4%, 5% or 10%.

[0013] First, the utility of a dictionary translation model (e.g., IBM Model 1) for English text retrieval is considered, particularly its end-to-end trained neural variant. Then, the neural variant of IBM Model 1 is used as an interaction layer applied to query / document embedding sequences generated in a context-free or contextualized manner. This novel approach to designing neural ranking systems is crucial for its benefits in terms of effectiveness, efficiency, and interpretability. Specifically, this disclosure demonstrates that adding a neural Model 1 layer on top of a bidirectional encoder representation from contextualized embeddings based on Transformer (BERT) does (1) reduce accuracy and / or efficiency; and (2) overcomes the limitation of existing BERT models on maximum sequence length. While the dictionary-based (i.e., context-free) neural Model 1 is less efficient than BERT-based ranking models, it can run efficiently on controllers, processors, CPUs, GPUs, or other computational or logic circuits (without requiring expensive indexing time preprocessing).

[0014] Typical text retrieval systems rely on simple term matching techniques to generate an initial list of candidates, which can then be further reordered using a learned model. Therefore, mismatches between query and document terms negatively impact retrieval performance; this is known as the lexical gap problem. Two decades ago, a method was proposed to reduce the lexical gap, thus improving retrieval efficiency with the help of a dictionary translation model called IBM Model 1 (also known as Model 1). Model 1 performs strongly when applied to finding answers in English question-answering (QA) archives using questions as queries, as well as cross-linguistic retrieval. However, little is known about its effectiveness on real-world monolingual English queries, partly because training Model 1 requires a large, previously unpublished query set.

[0015] Model 1 can be trained on question document pairs of similar length, which simplifies the task of finding useful associations between query terms and terms in related documents. However, if the query is much shorter than the corresponding related documents, for example, by two orders of magnitude, then training Model 1 should be considered. Furthermore, these models have been primarily trained in translation tasks using variants of the traditional Expectation-Maximization (EM) algorithm that produce nonparametric models.

[0016] This disclosure presents a better option by parameterizing the conditional translation probabilities using a neural network and performing end-to-end learning of the model in a ranking (rather than translation) task.

[0017] Further research was conducted on two recent MS MARCO sets containing hundreds of thousands of real user queries. Specifically, a novel class of ranking models was considered, where an interpretable neural model layer 1 was applied to the output of a labeled embedding neural network. This layer produces all pairwise similarities between the query and the BERT word fragments of the document. The word fragments are combined using a simple sum-product formula without any learned weights:

[0018]

[0019] in Q It's a query. D It's a document. q It is a query tag. d It is a document tag. It is a query tag q Conditional translation probability (with document markup) d (probability of alignment) It is the ranking score (interpreted as the conditional probability of a query given a document), and It is a marker d In the document D The maximum likelihood estimate of the probability of occurrence.

[0020] Sorting score It is the product of the scores of individual query word fragments, which makes it easy to pinpoint the word fragment that contributes the most. Similarly, for each query word fragment, it is possible to identify the document word fragment that contributes the most to its score. This makes the published model more interpretable compared to previous work.

[0021] The resulting composite network (including the labeled embeddings) is learned end-to-end using a ranking objective. Consider two scenarios: context-independent labeled embeddings and context-based labeled embeddings generated by BERT. Note that the method presented in this disclosure is general and can also be applied to other embedding networks.

[0022] The findings of this work can be summarized as follows:

[0023] First, adding an interpretable neural model layer on top of BERT makes it necessary to achieve almost no loss in accuracy and efficiency compared to the less interpretable vanilla BERT sorter. In fact, for long documents, the BERT-based interpretable neural model may outperform the baseline model applied to truncated documents, thus overcoming the limitation of existing preprocessing Transformer models on the maximum sequence length.

[0024] The fusion of nonparametric model 1 with the BM25 score can outperform the baseline fusion model, but the gain is very small (≈3%). In this case, the context-independent neural model 1 can be much more effective than its nonparametric variant.

[0025] Context-free Model 1 can be sparsified, exported as a nonparametric format, and executed on a CPU, achieving a speedup of 10% compared to running a BERT-based sorter on a GPU. 3 More than twice as fast. Therefore, it can improve the candidate generation stage without requiring expensive indexing-time pre-computation methods such as doc2query.

[0026] The use of statistical translation methods in text retrieval has been successfully applied to finding answers in QA archives by using questions as queries. In this case, the model is trained on a large set of monolingual questions paired with user-provided answers, i.e., on a parallel monolingual corpus. Model 1 is also useful for cross-lingual retrieval.

[0027] Model 1 is a nonparametric translation model where the context-independent translation probabilities of lexemes (or tokens) are learned using an expectation-maximization (EM) algorithm from a parallel corpus. A general approach to improving the performance of nonparametric statistical learning models is to parameterize the corresponding probabilities using neural networks. An early successful implementation of this idea in language processing was the hybrid HMM DNN / RNN (Hidden Markov Model Deep Neural Network / Recurrent Neural Network) system for speech recognition. More specifically, this disclosure uses Neural Model 1 as the final network layer, which is based on an LSTM-CRF (Long Short-Term Memory-Conditional Random Field) and CEDR (Contextualized Embedding for Document Ranking) architecture to compute the likelihood of a query document.

[0028] Consider a context-relevant dictionary neural translation model for cross-language retrieval. This method uses context-relevant translation probabilities from a bilingual parallel corpus for dictionary translation tasks. Given a document, the highest translation probability, along with its corresponding tag, is pre-computed and stored in the index. One would need to train the model on aligned sentences of similar length. However, in the case of monolingual retrieval, this disclosure does not have such fine-grained training data because queries are only paired with much longer, relevant documents.

[0029] Neural ranking models have been a hot topic in recent years. However, the success of earlier methods before BERT was controversial. This changed with the adoption of large neural models preprocessed in a self-supervised manner, especially after the introduction of Transformer models and the release of BERT. BERT-based models are now considered to significantly outperform all other methods. BERT and its variants also dominate the MS Marco leaderboard.

[0030] Prior to the Transformer, LSTM (Long Short-Term Memory) and other recurrent neural networks were the dominant methods for sequence data classification and translation. The Transformer has no inherent limitation on sequence length. However, recently proposed Transformer models employ a self-attention mechanism, where each sequence position can pay attention to all positions in previous layers. Because the self-attention complexity is quadratic with respect to sequence length, Transformer models (including BERT) only support inputs of finite length. Several proposals exist to alleviate this constraint; for example, the Conformer-kernel ordering model uses so-called separable attention. This research line complements this work.

[0031] To process long documents using existing pre-trained models, the system must split the document into several blocks, process each block separately, and aggregate the results by, for example, calculating a maximum value or weighted prediction score. Such models cannot be trained end-to-end on complete documents. Furthermore, the training process must assume that each block in a relevant document is also relevant, which is not always accurate. To improve upon simple aggregation methods, combined outputs from several document blocks using three simpler models have been introduced: KNRM (Kernel-Based Neural Ranking Model), PACRR (PACRR: Location-Aware Neural IR Model for Relevance Matching), and DRMM (Deep Relevance Matching Model). However, none of these aggregator models are interpretable.

[0032] The interpretation and explanation of statistical models has become a busy research area. However, the vast majority of methods rely on training separate interpretive models or utilizing saliency / attention maps. This is problematic because interpretations provided by unrelated models are difficult to verify and trust. Furthermore, saliency and attention maps reveal which parts of the data the model is processing, rather than how the model is processing them. One approach advocates that its computation be done by designing a transparent network, rather than building unreliable post-hoc explanations. When a fully transparent network is not feasible, there is still the benefit of interpretability at the last layer. In text retrieval, consider using a kernel-based formula to compute soft-match counts through contextualized embeddings. Because each pair of query document tokens produces several soft-match values ​​corresponding to different thresholds, aggregating these values ​​in an interpretable way is problematic. While this approach does provide insights into the model's decisions, the aggregation formula is quite complex. It is essentially a two-layer neural network with a non-linear (logarithmic) activation function after the first layer. In contrast, an interpretable model with one layer produces similarity scores for each pair of query and document sub-token tokens. These scores are combined using an explicit sum-product formula that does not have any learned weights (see Equation 1).

[0033] Methods that accelerate query-time ranking by deferring some computations to indexing time can be divided into two groups. The first group involves generating partially unpacked re-representations, which can be quickly combined at query time. These methods make little to no performance degradation necessary. The second group of methods produces or enhances independent query and document representations, which can be compared by computing inner products. Although at the cost of expensive indexing and some inefficiency, representations—whether dense or sparse—have been shown to improve first-stage retrieval.

[0034] In sparse representations, DeepCT (Deep Contextualized Term Weighting) uses BERT to generate importance weights for both documents and query terms, doc2query learns to augment documents with the most probable query terms, and the Epic model combines document augmentation with term reweighting. These models are less efficient than the vanilla BERT ranking engine and require expensive indexing time. Due to the sparsity of the data generated by the term augmentation and reweighting models, it can be stored in a traditional inverted index to improve performance in the first retrieval stage. While the model's augmentation and reweighting decisions are difficult to interpret because the first retrieval stage typically employs transparent ranking formulas such as BM25, it is easy to determine which term matches contribute most to the query document score. This makes the augmentation and reweighting models partially interpretable.

[0035] Moving on to tokenized embeddings, Vanilla BERT, and CEDR, let's assume the input text is broken down into small chunks of text called tokens. This process is called tokenization. Tokens can be complete English words, word segments, or lexical units (word units). [Document] d The length is measured by the number of tags, and is expressed as... Labeling. Because neural networks cannot directly manipulate text data, label sequences are used. First, it is transformed into a sequence of d-dimensional vectors through an embedding subnetwork. These vectors are also called embeddings. Initially, the embedding subnetworks were context-independent, meaning each tag always mapped to the same vector. Recently, the superiority of context-dependent, i.e., contextualized embeddings, has been demonstrated, producing multi-layer bidirectional LSTM preprocessing on large corpora in a self-supervised manner. These were soon surpassed by large pre-trained Transformers.

[0036] This disclosure includes two types of embeddings: vanilla context-free embeddings and BERT-based contextualized embeddings. Several important considerations regarding the BERT architecture are:

[0037] Contextualized tag embeddings are the values ​​of the last hidden state;

[0038] BERT operates on word fragments rather than complete words;

[0039] The vocabulary contains nearly 30,000 tags and includes two special symbols: [CLS] (aggregator) and [SEP] (delimiter); and

[0040] [CLS] is always preceding each labeled sequence, and its embedding is used as a sequence representation for classification and sorting tasks.

[0041] The “vanilla” BERT sorter uses a single fully connected layer as the prediction head, which converts the [CLS] vector into a scalar. It makes predictions based on the following tag sequence: ,in q It is a query, and This refers to documents. Long documents and queries need to be truncated so that the total number of tags does not exceed 512. To overcome this limitation, consider a method that:

[0042] Longer documents d Break it down into several blocks: ;

[0043] generate m A sequence of tags ;and

[0044] Each sequence is processed using BERT, generating regular labels and contextualized embeddings for [CLS].

[0045] This result is... m [CLS]-vectors and n context vectors Each document tag One. There are several ways to combine these contextualized vectors.

[0046] One approach is to extend the vanilla BERT sorter by making predictions based on the average [CLS] label: .

[0047] The second approach is to use contextualized embeddings as a direct replacement for context-independent embeddings in the following neural architectures: KNRM, PACRR, and DRMM.

[0048] In the third approach to introducing the CEDR architecture, [CLS] tags / vectors are additionally incorporated into KNRM, PACCR, and DRMM in a model-specific manner, which further improves performance.

[0049] Nonparametric and Neural Model 1. This includes a proposal to recast the retrieval as a translation problem, with the objective of learning a soft-matching function that assigns non-zero weights to relevant but distinct terms (e.g., synonyms). For this purpose, a simple Model 1 is considered, which is a term-independent and context-agnostic model.

[0050] let Document identification D With query q The relevant probability.

[0051] Use Bayesian rules Conveniently rewritten as Furthermore, assuming the document occurrence probability If there is a uniform prior, then the correlation probability is equal to... Proportional This can then be used as a translation from document to query for calculation.

[0052] let It is a query tag q Document tags d The probability of translation, and It is a marker d From the document D The probability of "generating". Then, query. Q It is a document D The probability of translating can be calculated as the product of the likelihood of individual query terms:

[0053] .

[0054] Making Model 1 effective requires multiple techniques, which differ significantly between nonparametric and neural variables. For nonparametric Model 1, ——Query terms q Likelihood – using parameters With set probability Linear combination:

[0055] .

[0056] Set probability and the probability of terms within the document Both are maximum likelihood estimates, i.e., normalized frequency of occurrence. For non-lexical terms... q , Set to a small number (e.g., 10) -9 Translation probability The EM algorithm, implemented in MGIZA, is used for computation. MGIZA models spurious insertions (i.e., translations from empty words), but these can be discarded. Several additional measures are then considered to improve the effectiveness of Model 1:

[0057] A parallel corpus is created by splitting documents and paragraphs into contiguous small chunks whose length is comparable to the query length.

[0058] Conditional translation probability It was learned from a symmetric corpus;

[0059] The threshold for discarding, based on empirical findings, is approximately 10. -3 All translation probabilities below And retain a maximum of 10 6 The most frequent marker; and

[0060] Self-translation probability Set to a positive value based on empirical findings, and rescale the probability. , making .

[0061] In the case of neural translation models, consider the simpler Equation 1, which deals with non-smooth translation and set probabilities. Translation probability It is computed by a simple neural network, whose input is a label. q and d Context-independent or context-based embeddings. For context-independent embeddings, Equals the term in the document d The number of times it appears divided by However, the contextualization of tokenized embeddings results in tokens that are inherently non-repeating. Therefore, we set up specific embeddings for BERT-based embeddings. .

[0062]

[0063] in Q It's a query. D It's a document. q It is a query tag. d It is a document tag. It is the embedding (vector) of the query tag. Document tags d i Embedding (vector) T It is the conditional translation probability, and It is the ranking score, which is interpreted as a given document D Query Q The conditional probability.

[0064] Similarly, in contextualized embedding settings,

[0065]

[0066] Where Q represents the query, D represents the document, q is the query tag, and d is the document tag. It is a contextualized embedding of query tags. Document tags d i The contextualized embedding is T, where T is the conditional translation and P is the ranking score.

[0067] Note that the context-free neural model 1 can be sparsified and derived into a nonparametric format, which allows for extremely efficient execution on the CPU. This is achieved by pre-computing all lexical tag pairs. This is accomplished by discarding all values ​​less than the threshold.

[0068] Further calculations in Equation 3 are performed using a simple and efficient neural network. Network "consumption" marker q and d Context-independent or contextualized embeddings, and produce within the scope The value in [the variable name]. To incorporate the self-translation probability—which is crucial for good convergence of context-free models—we set [the value]. and multiply all other probabilities by However, scaling the conditional probability ensures That is impractical. Therefore, It is a similarity function, but not a true probability distribution.

[0069] It can be shown that using a small self-translation probability This significantly improves the convergence of the context-free model. In this case, the conditional probability is appropriately scaled so that for any label t2, we have This is inefficient for context-independent embeddings and even insensitive to contextualized embeddings (because embedding vectors with the same label are different in different contexts). Therefore, instead of normalizing the conditional probabilities, the output of the neural network is normalized. Constraints in scope In addition, settings And multiply all other probabilities by .

[0070] Considering Several methods for designing neural parameterization. Consider using... and The embeddings of query and document tags are labeled separately. Perhaps the simplest approach is to learn separate sets of embeddings for queries and documents and compute the probabilities as scaled cosine similarities: However, this neural network lacks sufficient expressive power, and the resulting context-free model 1 is inferior to the nonparametric model 1 learned via EM.

[0071] As can be shown, a key factor for good performance is the connection between the embedding and its Hadamard product, which we believe helps the following layers discover better interactive features. This is then passed through one or more fully connected linear layers, where ReLU is followed by a sigmoid function:

[0072]

[0073] in P q , P d and F i It is a fully connected linear projection layer; It is a vector connection; and the layer norm is the layer normalization. It is a vector x and y The Hadama pile.

[0074]

[0075] Dataset: When experimenting with the MS MARCO collection, it includes data for paragraph and document retrieval tasks. Each MS MARCO collection contains a large number of real user queries (see Table 1). In this respect, no other collection is comparable to MS MARCO. The large query set is sampled from the logs of the search engine Bing. In this case, the dataset creators ensured that all queries could be answered using short text fragments. These queries are sparsely judged only (approximately one relevant paragraph per query). The sparse judgment is binary: relevant documents have a rank of one, and all other documents have a rank of zero.

[0076] In addition to a large query set with sparse judgments, two evaluation sets from the TREC 2019 / 2020 deep learning trajectories were used. These query sets are very small, but they have been thoroughly judged by NIST evaluators for document and paragraph retrieval tasks, respectively. The TREC NIST judgments range from zero (irrelevant) to three (fully relevant).

[0077] The publicly available training and validation sets were randomly split into the following subsets: a small training set for training linear fusion models (training / fusion), a large set for training neural models and nonparametric models 1 (training / modeling), a development set (development), and a test set containing up to 3K queries (test). Detailed dataset statistics are summarized in Table 1. Note that the training subset was obtained from the original training set, while the new development and test sets were obtained from the original development set.

[0078] Next, Spacy 2.2.3 was used to process the collection, extracting tokens and lexicons (terms) from the text. Frequently occurring tokens and lexicons were filtered out using Indri's stop word list, which was expanded to include abbreviations such as "n't" and "ll". Lexicons were indexed using Lucene 7.6. Sub-word tokens, i.e., BERT word fragments, were generated using the HuggingFace Transformers library. The stop word list was not applied to BERT word fragments.

[0079] Basic Setup. Experiments were conducted on a Linux server equipped with a six-core (12-thread) i7-6800K 3.4 GHz CPU, 125 GB of RAM, and four GeForce GTX 1080 TI GPUs. It used the text retrieval framework FlexNeuART, implemented in Java. It generated an initial candidate list using Lucene 7.6 with a BM25 scorer, which could be further reordered using either a traditional or neural reorderer. The traditional reorderers, including nonparametric model 1, were also implemented in Java. They ran in multi-threaded mode (12 threads) and made full use of the CPU. The neural reorderer was implemented using PyTorch 1.4 and Apache Thrift. The neural reorderer operated as a standalone, single-threaded server.

[0080] Ranking speed is measured by total CPU / GPU throughput per document / paragraph, not latency. Ranking accuracy is measured using trec_eval, a standard utility provided by the TREC organizers. Statistical significance is calculated using a two-tailed t-test with a threshold of 0.05.

[0081] All ranking models were applied to the candidate list generated by the adjusted BM25 scorer. The BERT-based model re-ranked the 100 entries with the highest BM25 scores: using a larger candidate pool impaired both efficiency and accuracy. All other models, including the neural context-agnostic model 1, re-ranked 1000 entries: further increasing the number of candidates did not improve accuracy.

[0082] Training the Model. The neural model was trained using pairwise margin loss. Training pairs were obtained by combining known relevant documents with 20 negative examples selected from the top 500 candidates returned by Lucene. In each epoch, one positive and one negative example were randomly sampled for each query. The BERT-based model was first preprocessed with a target corpus using masked language modeling and next-statement prediction. It was then trained on a ranking task over one epoch. A batch size of 16 was used, simulated via gradient accumulation. Context-free model 1 was trained from scratch for 32 epochs using a batch size of 32. Nonparametric model 1 was trained for five epochs using MGIZA. Further increasing the number of epochs did not significantly improve the results.

[0083] Use small weight decay (10) -7 The learning rate is linearly increased from zero over 10-20% of the steps until it reaches the base learning rate. The optimizer includes AdamW. For BERT-based models, different base rates are used, with fully connected prediction heads (…) ) and the main Transformer layer ( For context-free model 1, the basic rate is It decays by 0.9 after each period. This is the same for all parameters. This is achieved by pre-calculating all pairwise translation probabilities and discarding those less than 10. -4 Based on the probability, the trained neural model 1 was exported as a nonparametric model. The export took three minutes, and the exported model was executed using the same Java code as the nonparametric model 1.

[0084] Each neural model and the sparse model1 were trained and evaluated using five seeds (from zero to four). To do this, values ​​for each query, along with query-specific seed values ​​and averages, were computed. All hyperparameters were tuned on the development set.

[0085] Because the context-free Model 1 orderers are inherently weak, they are evaluated in a fusion mode. First, Model 1 is trained on the training / fusion subset. Then, the model's score is linearly combined with the BM25 score. Optimal weights are computed on the training / fusion subset using the coordinate ascent algorithm from RankLib. To improve the effectiveness of this linear fusion, using Model 1, the log score is normalized by the number of query tokens. Furthermore, the BM25 score is normalized by the sum of the IDF values ​​of the query terms. A fusion of BM25 scores, also trained on the training / fusion subset via RankLib, is also used for different tokenization methods.

[0086] Model Overview. Several models are compared (see Table 2). First, the BM25 score computed for lexicalized text is used, hereinafter referred to as BM25 (lexical). Second, several variants of the context-free model 1 are evaluated. The nonparametric model 1 is trained on both the original tokens and BERT word segments: the corresponding models are labeled as model 1 (tok) and model 1 (b / tok). The neural context-free model 1—labeled as NN-model 1—is used only with BERT word segments. This model is sparsified and derived into a nonparametric format, which runs efficiently on the CPU. It is labeled as NN-model 1-exp. Note that the context-free model 1 sorters are not powerful on their own; therefore, we evaluate them in a fusion mode by combining their scores with BM25 (lexical).

[0087] Crucially, all context-free models incorporate accurate term matching signals either via self-translation probabilities or via explicit smoothing with probabilities of the tag set (see Equation 2). Therefore, these models should be compared not only to BM25 but also to fusion models that incorporate BM25 scores from the original tags or BERT word segments. These models are denoted as BM25(lexical) + BM25(tok) and BM25(lexical) + BM25(b / tok), respectively.

[0088] Contextualized Model 1 applies the first layer of Neural Model 1 to the contextualized embeddings generated by BERT. This model is labeled BERT-Model 1. Due to the limitations of existing pre-trained Transformer models, long documents need to be broken into chunks, each of which is processed separately, i.e., contextualized. This is done in BERT-Model 1 (full), BERT-vanilla (full), and BERT-CEDR models. Another approach is to make predictions on truncated document text. This is done in BERT-Model 1 (short) and BERT-vanilla (short). In paragraph retrieval tasks, all paragraphs are short and do not require truncation or chunking. For BERT-based models, the base model, the 12-layer Transformer model, is used because it is more practical than the 24-layer BERT-large and performs comparably to BERT-large on the MS MARCO dataset.

[0089] Several hypotheses were tested using a two-tailed t-test:

[0090]

[0091] Table 2 illustrates the evaluation results: b / tok denotes BERT word segments, lexical units denote text lexical units, and tok denotes the original tokens. NN-Model1 and NN-Model1-exp are context-free neural model 1 and its derived variants running on a CPU. Both use only b / tok. Statistical significance is shown using... and Marking.

[0092] The context-independent neural model 1 is the same as the corresponding fusion baseline, for example, comparing BM25(lexical) + model 1(tok) with BM25(lexical) + BM25(tok);

[0093] BERT-Model 1 (full) is identical to the baseline BERT-vanilla (short);

[0094] The BERT-CEDR model is identical to the baseline BERT-vanilla (short);

[0095] BERT-vanilla (full) is identical to the baseline BERT-vanilla (short); and

[0096] The BERT-Model 1 (full) is identical to the baseline BERT-Model 1 (short).

[0097] The primary purpose of these tests was to evaluate whether the specific aggregation layers (including neural model 1) could be more accurate compared to models run on truncated documents. In Table 2, statistical significance is indicated by special symbols: the last two hypotheses use... All other assumptions use .

[0098] Results Discussion. The results are summarized in Table 2. First, note that there is less consistency in the results on the TREC 2019 / 2020 test set compared to the test set. In this case, some statistically significant differences (in the test) "vanish" on TREC2019 / 2020. The TREC 2019 / 2020 query set is very small and more likely to yield spurious results (compared to the test). Furthermore, the fusion model BM25(terms) + Model 1 (b / tok) is either worse than the baseline model BM25(terms) + BM25(b / tok) or the difference is not significant. BM25(terms) + Model 1 (tok) is mostly better than the corresponding baseline, but the gain is very small. In contrast, the fusion of neural model 1 with the BM25 scores of BERT word segments is more accurate on all query sets. On the test set, it is 15-17% better than BM25(terms). These differences are significant for both models on the test set and on the TREC 2019 / 2020 test set for paragraph retrieval tasks. Sparsification of Neural Model 1 does not lead to a significant decrease in accuracy. In this case, the sparsed model—executing on a CPU—is 10 times faster than the BERT-based sorter (running on a GPU). 3 More than double. For large candidate sets, the calculation of Model 1 scores can be further improved. Therefore, BM25 (terms) + NN - Model 1 - exp may be useful in the first retrieval stage.

[0099] Finally, the accuracy of BERT-based Neural Model 1 is compared with BERT-CEDR and BERT-vanilla models. By comparing BERT-vanilla (short) with BERT-Model 1 (short) and BERT-vanilla (long) with BERT-Model 1 (long) on ​​the test set (for document retrieval tasks), it can be shown that the Neural Model 1 layer actually has no efficiency or accuracy loss. In this case, on the test set, BERT-Model 1 (full) – which operates on full documents – is about 1% more accurate than both BERT-Model 1 (short) and BERT-vanilla (short), however, the difference in the latter is not statistically significant. The same is true for BERT-CEDR-PACRR in this case, which is also 1% better than BERT-vanilla. This does not invalidate the results, which show that when comparing BERT-CEDR-PACRR with the “full” variant of the vanilla BERT orderer, BERT-CEDR-PACRR is better than BERT-vanilla, which makes predictions for the average [CLS] embedding. However, in our experiments, this model significantly outperformed the simpler BERT-vanilla (short) model—which operates on truncated text—by 4.2%, and the difference was statistically significant. It is believed that obtaining more conclusive evidence regarding the effectiveness of the aggregation layer requires different datasets where document and paragraph relevance is more difficult to predict from truncated documents.

[0100] In experiments utilizing dictionary-based translation models—particularly combining Neural Model 1 with a tag embedding network (both context-aware and context-free)—the latter was found to be beneficial for both efficiency and interpretability. Furthermore, Neural Model 1 is the only neural model that can be sparsified and derived into a "non-neural" format, allowing it to run efficiently on CPUs (faster than BERT on GPUs). This approach eliminates the need for costly indexing time pre-computation. The effectiveness of this method can be further improved, for example, by designing better parameterization of conditional translation probabilities. An interpretable neural model 1 layer could also be useful for efficient ranking of long documents, but conclusive experiments require a dataset with characteristics different from MS MARCO. It can be shown that even when queries and documents have significantly different lengths, nonparametric model 1 can be trained via EM, but better results are obtained by training a context-independent neural model 1 and deriving it into a nonparametric format.

[0101] This disclosure proposes a mechanism for ranking search results using a special neural network layer, namely a dictionary translation model (e.g., IBM Model 1). First, the query and document are encoded using a context-free or context-aware neural network (e.g., BERT). Thus, the query and document tags are represented by a sequence of vectors. Then, the special dictionary neural translation layer ingests these vectors and produces a matrix of similarity scores, such as query term probabilities. These scores are then combined in an interpretable manner using a simple sum-product formula. When using context-aware networks, efficient execution typically requires specialized accelerators, such as graphics processing units (GPUs) or tensor processing units (TPUs). However, for context-free encoding networks, this mechanism relies on the pre-computation and sparsification of the query-document tag probability matrix. This allows for efficient execution on lower-performance systems, processors, or controllers.

[0102] Typical text retrieval systems rely on simple term / tag matching techniques to generate an initial list of candidates, which can then be further reordered using a learned model. Therefore, mismatches between query and document terms negatively impact retrieval performance, a problem known as the lexical gap. To reduce the lexical gap and improve retrieval efficiency, a dictionary translation model called IBM Model 1 was introduced. IBM Model 1 demonstrates strong performance when applied to finding answers in English question-answering (QA) archives using questions as queries, as well as cross-language retrieval. Model 1 is trained on question-document pairs of substantially similar length, which simplifies the task of finding useful associations between query terms and terms in relevant documents. These models have been trained primarily on translation tasks using variants of the traditional expectation-maximization (EM) algorithm that generate nonparametric models. Key steps in this disclosure include parameterizing the conditional translation probabilities using a neural network and performing end-to-end learning of the model on ranking (rather than translation) tasks.

[0103] The concepts described below are in a simplified form and are not exhaustive and are not intended to limit the scope of this disclosure, but rather to present a basic description of the concepts that are discussed in more detail throughout this disclosure.

[0104] This disclosure provides a multi-stage retrieval system applicable to various domains, including but not limited to mobile and desktop search on public and private networks (e.g., the World Wide Web). In such a system, documents flow through a series of "funnels" that discard hopeless candidate documents using increasingly sophisticated and accurate ranking components. Both queries and documents are fragments of text that can be represented by sequences of tokens. Tokens can be complete words, lexical units (terms), word segments, or multi-word phrases. Breaking text fragments down into tokens is called tokenization or tokenization.

[0105] In retrieval systems, neural ranking models can be applied to queries and documents to generate estimates of relevance scores. These models are more complex and expensive, and therefore, they are used in later stages of the retrieval pipeline. While these estimates computed using neural ranking models can be very accurate, they are difficult to interpret. This disclosure presents an interpretable neural network layer—a neural dictionary translation model (e.g., IBM Model 1)—that can be applied to state-of-the-art neural networks such as BERT, along with benefits of efficiency and effectiveness.

[0106] In particular, context-free dictionary translation models can be fully pre-computed (e.g., for all query and document tag pairs), sparsified, and executed efficiently on lower-performance systems, processors, or controllers (e.g., CPUs that do not require expensive accelerators, such as GPUs and TPUs). Although such models have poorer accuracy for the final ranking, they can still improve earlier stages of retrieval.

[0107] Figure 1 This is a block diagram of the search retrieval system 100. This is a simplified description and does not include all implementation details. The search retrieval system 100 is designed to provide simple access to a document collection 102, which can be stored in various media (e.g., RAM, magnetic, optical, and solid-state storage devices) in various formats (e.g., text, rich text format (rtf), Microsoft Word, PDF, etc.). It can also be organized in different ways: stored as files in a file system, a collection of hyperlinked documents, or internal fields in a relational database table. The collection can be processed in whole or in part. In particular, a document discovery engine 104 (e.g., a web crawler or database query application) is first used to discover documents that are worth including in the index. A text processing engine 106 identifies the document format, assigns them (usually) unique identifiers, and extracts the text data. The documents are then stored in (1) a direct (or forward) index 110 that allows fast retrieval of the document text using its identifiers and (2) a reverse (or backward) index 108 that allows fast retrieval of documents containing one or more tags.

[0108] When a user submits a query (i.e., a search request for indexed documents) using a user device (e.g., a desktop computer, laptop computer, smartphone, etc.) 112, a multi-stage retrieval pipeline is employed. Although the image shows a keyboard and monitor, input to user device 112 can include a touchscreen, keyboard, microphone, camera, scanner, image file, or other file or input device. Furthermore, output to user device 112 can include a monitor, projector, speaker, or file or other output device 200. The first stage uses a lightweight query engine 114, which retrieves a relatively large set of candidate documents 116. For this, it relies on simple tag / term matching techniques and employs a previously created inverted index 108. Subsequent stages employ zero or more non-neural lightweight ranking components 118 to refine the original candidate set. These reorderers can be very efficient, and they run on the CPU, meaning they do not require expensive accelerators for neural networks such as GPUs or TPUs. Furthermore, since neural network sorters 120 are typically computationally expensive, they are only used in the most recent stage of the retrieval pipeline.

[0109] Figure 2 This is a flowchart of a sorting neural network 200. Consider providing a query... Q 202 and Documents D Query 202 and document 204 both involve assigning relevance scores. Both are text fragments that can be represented by sequences of tokens. Tokens can be complete words, lexical units (terms), word segments, or multi-word phrases. Query tokens are... q Identifier (possibly with an index), and document markup. d Identifier (possibly with an index). Document. D The length is measured by the number of tags, and is expressed as... Marking.

[0110] Because neural networks cannot directly manipulate text data, the labeled sequence... First, it is transformed into a sequence of d-dimensional embedding vectors through an embedding neural network. This embedded neural network consists of one or more components.

[0111] Typically, such a network first generates a context-free embedding 206 for query tags and a context-free embedding 208 for document tags. The context-free embeddings for the same query (or document tags) are always the same, regardless of their position and the surrounding tags. However, they may differ between queries and documents.

[0112] After generating context-independent embeddings 206 and 208, an optional neural network 210, such as a Transformer model (e.g., BERT), generates context-defined query token embeddings 212 and context-defined document token embeddings 216. These embeddings depend on the context and are never the same unless we have precise tokens in the same context.

[0113] In addition to contextualized token embeddings, network 210 can generate one or more (pooled) embeddings 214 that represent a large portion of the document. For example, for each sequence of 512 tokens, BERT generates an embedding vector for the so-called class token [CLS], which represents the entire sequence.

[0114] When the optional contextualization network 210 is not used, 212 is the same as 206 and 216 is the same as 208.

[0115] Finally, embeddings 212, 214, and 216 (or, at the moment when the optional neural contextualization layer 210 is unavailable, embeddings 206 and 208) are fed into an interactive neural network that produces the final query document ranking score 220. The interactive neural network 218 can be a multi-layer interactive neural network 218.

[0116] Training: One method for training a neural network involves compiling pairs (or lists) of positive and negative documents, which are relevant and irrelevant to a given query, respectively, and forcing relevant documents to rank higher than irrelevant documents. This is done by feeding the scores (or their differences) into a special comparison function called the loss function and minimizing the loss using various optimization techniques. The loss function is specified to be high when irrelevant documents rank higher than relevant documents.

[0117] A key aspect presented in this disclosure is that, unlike systems that use parallel corpora to train neural dictionary translation models for translation tasks, this method allows the system to train models in monolingual ranking tasks. Parallel corpora are sets of pairs of short documents (mostly single-sentence documents) in different languages.

[0118] Special neural network layers: Figure 3 This is a flowchart of a neural sorter architecture 300, which receives a query 302 and a document 304, and generates context-independent embeddings 306 and 308, which are optionally passed through a contextualized neural network 310. Embeddings 312 and 314 are then processed by aggregation components 316 and 318 to produce a final sorting score 320.

[0119] Another key aspect presented in this disclosure is the replacement of partially explainable aggregate components 316 and 318. Figure 2 Interaction layer 218.

[0120] First, neural network 316 generates translation scores for the probability of generating images for all query and document tag pairs. As explained below, computation is performed via a neural network. Then, these values ​​are combined using a simple sum-product formula 318, which multiplies the scores of all the individual tags:

[0121] .

[0122] In formula (5), The document embedding indicates the probability that it was "generated" from a document. For context-independent embeddings, it is the maximum likelihood estimate, i.e., the normalized frequency of term occurrences. In the case of contextualized embeddings, it is set to... .

[0123] Switch to using neural networks to calculate dictionary translation probabilities Another key step in this disclosure involves parameterizing the conditional translation probabilities using a neural network. This is done as follows:

[0124] First, the system tokenizes queries and documents.

[0125] Next, the system adopts Figure 3 The embedding network is used to generate the embedding 312 of the query token (denoted as...). w q Embedding of document tags 314 (marked as) w d This method is effective for both context-independent and context-based embedding neural networks.

[0126] Finally, using the embeddings of query and document tags, the system employs a feedforward neural network to calculate the translation probability. There are multiple ways to do this. In one implementation, the system proceeds as follows:

[0127]

[0128] This is a method for calculating the conditional translation probability of a query tag for a given document tag via a neural conditional translation probability network, based on the tag embeddings calculated in steps 406 and 408.

[0129] Now consider the pre-computation and sparsification (neural derivation) of the query document tag probability matrix. When Figure 3When the ranking system combines the neural dictionary translation layer with contextualized embeddings, as described in this disclosure, the overall system achieves state-of-the-art performance. However, it can also be computationally expensive. Using context-independent embeddings leads to a significant performance degradation. However, the resulting model can be (1) sparsified and pre-computed, and (2) combined with simple TF*IDF models (such as BM25), and executed efficiently on a CPU to improve performance. Figure 1 The earlier stage sorting component 118.

[0130] Figure 4 This is a flowchart of model pre-computation and sparsification 400, which uses a context-free tag embedding network. First, the processor or controller generates embeddings 406 and 408 for all frequent combinations of query 402 and document tag 404. Then, the controller or processor uses the context-free model... (For example, using a neural network to calculate dictionary translation probabilities) to calculate all possible pairwise translation probabilities / scores.

[0131] In step 410, the processor or controller calculates the conditional translation probability of the query tag for a given document tag via a neural conditional translation probability network based on the tag embeddings calculated in steps 406 and 408.

[0132] The result of step 410 is a dense query document tag translation probability matrix that can be stored in step 412. For example, in step 412, the output of step 410 from the GPU is stored in CPU memory.

[0133] The sparsification step 416 discards all scores below an empirically chosen threshold. As described in this disclosure, this allows for efficient computation of equation (1) on a CPU. Examples of thresholds include a probability of one in a thousand (0.1% or 0.001), 0.2%, 0.5%, 0.75%, 1%, or 2% for one tag to match another. The result is a sparse query document tag translation probability matrix 418.

[0134] Example machine architecture and machine-readable media. Figure 5 It is a block diagram of an electronic computing system suitable for implementing the system disclosed herein or performing the methods disclosed herein. Figure 5The machines are shown as standalone devices suitable for implementing the above concepts. Regarding the servers described above, multiple such machines can be used, operating within data centers, as part of a cloud architecture, etc. On the server side, not all illustrated functions and devices are utilized. For example, while systems and devices used by users to interact with servers and / or cloud architectures may have screens, touchscreen inputs, etc., servers typically do not have screens, touchscreens, cameras, etc., and usually interact with users through connection systems with appropriate input and output aspects. Therefore, the architecture below should be considered as encompassing multiple types of devices and machines, and depending on their form factors and uses, various aspects may or may not be present in any particular device or machine (e.g., servers rarely have cameras, and wearable devices rarely include disks). However, Figure 5 The illustrative examples are suitable for enabling those skilled in the art to determine how to utilize appropriate combinations of hardware and software to implement the previously described embodiments, and to make appropriate modifications to the illustrated embodiments of the particular devices, machines, etc. used.

[0135] Although only a single machine is illustrated, the term "machine" should also be understood to include any collection of machines that individually or jointly execute one or more sets of instructions to perform any one or more of the methods discussed herein.

[0136] Examples of machine 500 include at least one processor 502 (e.g., a controller, microcontroller, central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), advanced processing unit (APU), or a combination thereof), one or more memories, such as main memory 504, static memory 506, or other types of memory, which communicate with each other via link 508. Link 508 may be a bus or other type of connection channel. Machine 500 may include further optional aspects, such as a graphics display unit 510 including any type of display. Machine 500 may also include other optional aspects such as alphanumeric input device 512 (e.g., keyboard, touch screen, etc.), user interface (UI) navigation device 514 (e.g., mouse, trackball, touch device, etc.), storage unit 516 (e.g., one or more disk drives or other storage devices), signal generation device 518 (e.g., speaker), one or more sensors 521 (e.g., GPS sensor, one or more accelerometers, one or more microphones, one or more cameras, etc.), output controller 528 (e.g., wired or wireless connection for connecting to and / or communicating with one or more other devices, such as Universal Serial Bus (USB), Near Field Communication (NFC), Infrared (IR), serial / parallel bus, etc.), and network interface device 520 (e.g., wired and / or wireless) for connecting to and / or communicating through one or more networks 526.

[0137] Various memories (i.e., the memories of 504, 506, and / or one or more processors 502) and / or storage units 516 may store one or more sets of instructions and data structures (e.g., software) 524, which embody or are utilized by any one or more of the methods or functions described herein. When executed by one or more processors 502, these instructions cause various operations to implement the disclosed embodiments.

[0138] As used herein, the terms “machine storage medium,” “device storage medium,” and “computer storage medium” refer to the same thing and may be used interchangeably in this disclosure. The terms refer to single or multiple storage devices and / or media (e.g., centralized or distributed databases, and / or associated caches and servers) that store executable instructions and / or data. Therefore, the terms should be understood to include storage devices such as solid-state memory, as well as optical and magnetic media including memory internal or external to a processor. Specific examples of machine storage media, computer storage media, and / or device storage media include non-volatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), FPGAs, and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms machine storage medium, computer storage medium, and device storage medium specifically and explicitly exclude carrier waves, modulated data signals, and other such temporary media, at least some of which are included in the term “signal medium” discussed below.

[0139] The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" refer to the same thing and are used interchangeably in this disclosure. The term is defined to include both machine storage media and signal media. Therefore, the term includes both storage devices / media and carrier / modulated data signals.

[0140] Program code embodying the algorithms and / or methods described herein can be distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform aspects of one or more embodiments. Inherently non-transitory computer-readable storage media can include volatile and non-volatile, removable and non-removable tangible media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technologies, portable optical disc read-only memory (CD-ROM) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be read by a computer. Computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device or another device, or downloaded via a network to an external computer or external storage device.

[0141] Computer-readable program instructions stored in a computer-readable medium can be used to direct a computer, other type of programmable data processing apparatus, or other device to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of art including instructions that implement the functions, actions, and / or operations specified in a flowchart or diagram. In some alternative embodiments, the functions, actions, and / or operations specified in the flowcharts and diagrams can be reordered, processed sequentially, and / or concurrently, consistent with one or more embodiments. Furthermore, any of the flowcharts and / or diagrams may include more or fewer nodes or blocks than those illustrated consistent with one or more embodiments.

[0142] While the invention has been fully described through various embodiments, and these embodiments have been described in considerable detail, the applicant does not intend to limit the scope of the appended claims or restrict them in any way to such details. Additional advantages and modifications will readily occur to those skilled in the art. Therefore, the invention is not, in its broader aspects, limited to the specific details, representative apparatuses and methods, and the illustrative examples shown and described. Consequently, deviations from such details may be made without departing from the spirit or scope of the overall inventive concept.

Claims

1. A language system, comprising: The controller is configured as follows: Receive queries and documents; The query is tokenized into a sequence of query tokens; Each token in the document is converted into a sequence of document tokens; Convert query tags into query embeddings; Convert document tags into document embeddings; Generate an embedding pair matrix for each query and document embedding; A similarity score is computed for each entry in the embedding pair matrix via a neural conditional translation probabilistic network, which has been trained on a ranking task using a corpus of pair queries and corresponding relevant documents. Each document's ranking score relative to each query is generated by aggregating the sum of the similarity scores for the corresponding queries; The documents are ordered based on their sorting scores; and Output the document and its association ranking score.

2. The language system according to claim 1, wherein, The neural conditional translation probability network has been trained in a single language.

3. The language system according to claim 1, wherein, The query embedding or document embedding is generated using a Transformer model.

4. The language system according to claim 1, wherein, The sum-product aggregation is non-contextual and is expressed as in Q It's a query. D It's a document. q It is a query tag. d It is a document tag. embed q (q) It is a query embedding of query tags. embed d (d i ) Document tags d i Document embedding, T It is conditional translation, and P It is the ranking score.

5. The language system according to claim 1, wherein, The sum-product aggregation is contextual and is expressed as in Q It's a query. D It's a document. q It is a query tag. d It is a document tag. embed q (q,d i ) It is a contextualized embedding of query tags. embed d (q,d i ) Document tags d i Contextualized embedding, T It is conditional translation, and P It is the ranking score.

6. The language system according to claim 1, wherein, There is more than one controller.

7. The language system according to claim 1, wherein, The query was entered via microphone.

8. The language system according to claim 7, wherein, The output is made via a speaker.

9. A language system, comprising: The controller is configured as follows: Receive queries and documents; The query is tokenized into a sequence of query tokens; The document is tokenized into a sequence of document tags; Generate a tag pair matrix for each query and document tag; For each entry in the labeled matrix, a pre-computed similarity score generated by a neural conditional translation probabilistic network is retrieved, where the neural conditional translation probabilistic network has been trained on a ranking task using a corpus of pairwise queries and corresponding relevant documents; Each document is ranked relative to each query by aggregating the sum of the similarity scores for each query. and Output the document and its association ranking score.

10. The language system according to claim 9, wherein, Sum and product aggregation is in Q It's a query. D It's a document. q It is a query tag. d It is a document tag. T ( q │ d ) is a query tag q With document tags d Alignment pre-computed conditional translation probabilities P ( Q │ D ) is the sorting score, and P ( d │ D ) is a marker d In the document D The maximum likelihood estimate of the probability of occurrence.

11. The language system according to claim 9, wherein, The neural conditional translation probability network is trained in a single language.

12. The language system according to claim 9, wherein, The neural conditional translation probabilistic network incorporates context-independent embeddings for query tagging or document tagging.

13. The language system according to claim 12, wherein, The dense query document tag translation probability matrix, generated by a neural conditional translation probability network, is sparsified by discarding probabilities below a threshold.

14. The language system according to claim 12, wherein, The query is input via microphone, and the output is made via speaker.

15. A non-transitory computer-readable medium comprising instructions for ranking neural conditional translation probabilistic networks, the instructions, when executed by a processor of a computing device, causing the computing device to perform operations including: Receive queries and documents; The query is tokenized into a sequence of query tokens; For each document, the document is tokenized into a sequence of document tokens; Convert query tags into query embeddings; Convert document tags into document embeddings; Generate an embedding pair matrix for each query and document embedding; A similarity score is computed for each entry in the embedding pair matrix via a neural conditional translation probabilistic network, which has been trained on a ranking task using a corpus of pair queries and corresponding relevant documents. Each document is ranked relative to each query by aggregating the sum of the similarity scores for each corresponding query. and Output the order of the documents based on their sorting scores.

16. The medium according to claim 15, wherein, The neural conditional translation probability network has been trained in a single language.

17. The medium according to claim 15, wherein, The query embedding or document embedding is generated using a Transformer model.

18. The medium according to claim 15, wherein, The sum-product aggregation is non-contextual and is expressed as in Q It's a query. D It's a document. q It is a query tag. d It is a document tag. embed q (q) It is a query embedding of query tags. embed d (d i ) Document tags d i Document embedding, T It is conditional translation, and P It is the ranking score.

19. The medium according to claim 15, wherein, The sum-product aggregation is contextual and is expressed as in Q It's a query. D It's a document. q It is a query tag. d It is a document tag. embed q (q,d i ) It is a contextualized embedding of query tags. embed d (q,d i ) Document tags d i Contextualized embedding, T It is conditional translation, and P It is the ranking score.

Images