Lookahead strategy for trie-based bundle search in generative retrieval

Abstract

Systems and methods are provided for generating a sequence of keywords from an input query. A first sequence of text corresponding to the input query can be received and encoded into a source sequence representation using an encoder of a machine learning model. A keyword sentence can then be generated from the source sequence representation using a decoder of the machine learning model. The decoder can generate modified generation scores for a plurality of predicted tokens, where the modified generation scores are based on respective predicted token generation scores and maximum generation scores for suffixes of each predicted token. The decoder can then select a predicted token of the plurality of predicted tokens based on the modified generation scores and add the selected predicted token to a previously decoded partial hypothesis provided by the decoder.

Description

BACKGROUND

[0001] Search queries from users can be expanded or otherwise modified to include keywords that are relevant to media content providers. Since media content providers tend to rely on triggered keywords to display their content when a user clicks on the content, it is of great interest to expand or modify search queries from users to keywords in the library of content providers that are relevant. Although the previous keyword needs to be identical to the user input query to trigger the display of content to the user, methods such as information retrieval with quality filtering are often used to invoke more relevant keywords. However, traditional information retrieval techniques are unable to bridge the semantic gap between the query and the keyword. Thus, it is of primary interest to search engine service providers to develop more semantically relevant keywords from user queries rather than generating keywords from traditional information retrieval techniques. It is with respect to these and other general considerations that embodiments have been made. Also, although a relatively specific problem has been discussed, it should be understood that the embodiments should not be limited to solving the specific problem identified in the background. SUMMARY

[0002] According to examples of the present disclosure, a method for generating a sequence of keywords from an input query is provided. The method can include receiving a first sequence of text corresponding to the input query and encoding the first sequence of text into a source sequence representation using an encoder of a machine learning model, generating the sequence of keywords from the source sequence representation using a decoder of the machine learning model. Further, the method can include generating a plurality of predictive tokens based on the source sequence representation and a previously decoded partial hypothesis provided by the decoder, generating modified generation scores for the plurality of predictive tokens based on respective predictive token generation scores and maximum generation scores of suffixes of each predictive token, selecting a predictive token of the plurality of predictive tokens based on the modified generation scores, and adding the selected predictive token to the previously decoded partial hypothesis provided by the decoder. Content can then be selected based on the selected predictive token and provided to a computing device in response to the input query, where the selected predictive token matches at least one keyword associated with the content to be provided to the computing device. By modifying the score of the next predictive token with the scores of future tokens that look ahead, the search space can be optimized, thereby allowing for recall of more relevant keywords compared to past sequence prediction models. Further, future information associated with future tokens can be leveraged to alleviate common problems such as, but not limited to, common prefixes that are not part of the desired target sequence, or words that encounter "noise" or spelling errors in the target sequence.

[0003] In some examples, a computer-readable storage medium is provided. The computer-readable storage medium can include instructions that, when executed by a processor, cause the processor to receive a first text sequence corresponding to an input query, encode the first text sequence into a source sequence representation using an encoder of a machine learning model, generate a keyword sequence from the source sequence representation using a decoder of the machine learning model, generate generation scores for a plurality of predicted word pieces, where the plurality of predicted word pieces are based on the source sequence representation and a previously decoded partial hypothesis provided by the decoder, generate modified generation scores for the plurality of predicted word pieces, where the modified generation scores are based on the respective predicted word piece generation score and a maximum generation score for a suffix of each of the plurality of predicted word pieces, select a predicted word piece of the plurality of predicted word pieces based on the modified generation scores, and add the selected predicted word piece to the previously decoded partial hypothesis provided by the decoder. In some examples, a system including a processor and a memory is provided. The processor can include instructions for performing a method that includes receiving a first text sequence corresponding to an input query and encoding the first text sequence into a source sequence representation using an encoder of a machine learning model, and generating a keyword sequence from the source sequence representation using a decoder of the machine learning model. Further, the processor can generate generation scores for a plurality of predicted word pieces, where the plurality of predicted word pieces are based on the source sequence representation and a previously decoded partial hypothesis provided by the decoder, generate modified generation scores for the plurality of predicted word pieces, where the modified generation scores are based on the respective predicted word piece generation score and a maximum generation score for a suffix of each of the plurality of predicted word pieces, select a predicted word piece of the plurality of predicted word pieces based on the modified generation scores, and add the selected predicted word piece to the previously decoded partial hypothesis provided by the decoder. The content can then be selected based on the selected predicted word piece and provided to a computing device in response to the input query, where the selected predicted word piece matches at least one keyword associated with the content to be provided to the computing device. By modifying the score of the next predicted word piece with the score of the future word piece, the search space can be optimized, thereby allowing more relevant keywords to be recalled compared to past sequence prediction models. Further, future information associated with future word pieces can be leveraged to alleviate common issues such as, but not limited to, common prefixes that are not part of the desired target sequence, or words that encounter "noise" or spelling errors in the target sequence.

[0004] In some aspects of the above-described methods, systems, and / or computer-readable storage media, the generation score for a suffix of each of the plurality of predicted word pieces can be based on a generation score for a word piece that is a suffix of the one or more predicted word pieces. Thus, future information for word pieces beyond the immediate suffix of the predicted word piece can be used to optimize the search space and allow more relevant keywords to be recalled compared to past sequence prediction models.

[0005] In some aspects of the above described methods, systems, and / or computer- readable storage media, the plurality of predicted word pieces and the suffix of each predicted word piece are accessed from and / or constrained to a Trie. As the number of possible predicted word pieces from which a selection can be made can be significantly reduced when using a Trie, the methods described herein can not encounter the overhead associated with searching through a search space larger than the Trie. For example, the amount of resources utilized, such as memory and computation, can be reduced. In some aspects, the Trie can be specific to the input query, such that a keyword-specific Trie can be generated.

[0006] This summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0007] Non-limiting, non-exhaustive examples are described with reference to the following figures.

[0008] Figure 1 An example system for performing a look-ahead strategy for a Trie-based generation retrieval model is depicted in accordance with examples of the present disclosure.

[0009] Figure 2A Details of a first Trie are depicted.

[0010] Figure 2B Details of a second Trie are depicted.

[0011] Figure 2C Details of a third Trie are depicted.

[0012] Figure 2D Details of a fourth Trie are depicted.

[0013] Figure 2E Details of a fifth Trie are depicted.

[0014] Figure 3 Details of a decoding process associated with a Seq2Seq model are depicted in accordance with examples of the present disclosure.

[0015] Figure 4A Details of a decoding process associated with a Seq2Seq model are depicted in accordance with examples of the present disclosure.

[0016] Figure 4B Additional details of a decoding process associated with a Seq2Seq model are depicted in accordance with examples of the present disclosure.

[0017] Figure 4CAdditional details of the decoding process associated with the Seq2Seq model are depicted in accordance with examples of the present disclosure.

[0018] Figure 4D Additional details of the decoding process associated with the Seq2Seq model are depicted in accordance with examples of the present disclosure.

[0019] Figure 4E Additional details of the decoding process associated with the Seq2Seq model are depicted in accordance with examples of the present disclosure.

[0020] Figure 5 Details of the decoding process associated with the Seq2Seq model are depicted in accordance with examples of the present disclosure.

[0021] Figure 6 Additional details of the decoding process associated with the Seq2Seq model are depicted in accordance with examples of the present disclosure.

[0022] Figure 7 Additional details of the search engine service, the sponsored content service, and the Trie server are depicted in accordance with examples of the present disclosure.

[0023] Figure 8 Details of the first method are depicted in accordance with examples of the present disclosure.

[0024] Figure 9 Details of the second method are depicted in accordance with examples of the present disclosure.

[0025] Figure 10 Details of the third method are depicted in accordance with examples of the present disclosure.

[0026] Figure 11 Details of the fourth method are depicted in accordance with examples of the present disclosure.

[0027] Figure 12A A mobile computing device that can be used to practice embodiments of the present disclosure is shown.

[0028] Figure 12B Additional details of a mobile computing device that can be used to practice embodiments of the present disclosure are shown.

[0029] Figure 13 One aspect of an architecture for a system for processing data is shown. DETAILED DESCRIPTION

[0030] In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration specific embodiments or examples. These aspects can be combined, other aspects can be utilized, and structural changes can be made without departing from the scope of the present disclosure. Embodiments can be practiced as methods, systems, or devices. Accordingly, the embodiments are not to be restricted to the precise details of the description given herein and the accompanying drawings. The following detailed description is therefore not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0031] In a sponsored search engine, a search query from a user is extended or otherwise modified to include keywords related to a sponsor, such as an advertiser. The sponsor can bid on the triggered keywords to display its content and can pay the sponsored search engine on a per-click basis when a user clicks on the content. Since the primary revenue of a sponsored search engine is to provide content that a user can be interested in, the sponsored search engine provider is very interested in the extension or modification of a search query from a user to a keyword in the pool of keywords related to a sponsor. Although the past keywords need to be the same as the user entered query to trigger the display of content to the user, more relevant keywords are often invoked using methods such as information retrieval with quality filtering. However, traditional information retrieval techniques are unable to bridge the semantic gap between the query provided by the sponsor and the keywords. Therefore, what the sponsored search engine service provider is interested in is to develop more semantically relevant keywords from the user query, rather than generating keywords from traditional information retrieval techniques.

[0032] In some examples, an initial user query is rewritten to include a series of intermediate queries; the results of the intermediate queries can then be combined. Query rewriting strategies are often used widely, but tend to be inefficient because many of the keywords generated from the extended query rewriting strategy are not included in the keyword library associated with a sponsor or other media content provider. Examples of query rewriting can include utilizing a natural language generation (NLG) sequence-to-sequence (Seq2Seq) model to output one or more text segments based on an input text segment, such as a user query. The NLG Seq2Seq model can receive an input, such as a query, and generate multiple output segments one word at a time during a decoding process. That is, each output segment can include multiple wordpieces or words generated at various time steps throughout the decoding process. At each decoding time step, multiple predicted wordpieces or predicted words are generated, where each predicted wordpiece or predicted word is associated with a generation score. The generation score indicates a conditional probability or likelihood that the predicted wordpiece or predicted word is the next element in the output segment based on the input and a partial hypothesis of the output segment that includes the previously predicted wordpieces or words generated at previous time steps. The partial hypothesis of the output segment that is decoded is the partial output segment that includes the previously predicted wordpieces or words generated at previous time steps. Thus, once the decoding process has completed, the decoder can have predicted one or more most-decoded partial hypotheses. However, in some examples, the NLG Seq2Seq model can be efficient at generating fluent sentences, but can lack the accuracy of generating queries that include keywords found in a library or keywords that are otherwise of interest to a sponsor, advertiser, or other media content provider.

[0033] In some examples, a Trie-based Seq2Seq model generates a word or wordpiece at each time step, where the word or wordpiece exists on a path of the Trie. A Trie (also known as a prefix tree) is a tree-like data structure that has a node for each keyword in a library. By constructing the nodes in a particular way, a text segment that includes one or more words can be retrieved from the Trie by traversing a branching path of the tree. Thus, in a Trie-based Seq2Seq model, all generated words or wordpieces are constrained to the Trie; thus, where the Trie is composed of keywords from a library of content keywords associated with one or more sponsors, the output segments produced by the Trie-based Seq2Seq model can be limited to the library of keywords. While the Trie-based Seq2Seq model constrains the generation of keywords to the search space, i.e., those keywords that appear in the Trie, previous Trie-based Seq2Seq models do not utilize global information and thus suffer from noise, common prefix, and wordpiece selection problems.

[0034] That is, when constraining a Seq2Seq model to a Trie, the model can generate a word or word piece that is already in the Trie but has a low generation score. Relying on such a word piece with a low generation score can lead to incorrect search paths in the Trie and can result in inefficient outcomes in which the Seq2Seq model generates a text segment that is semantically dissimilar to the user query or to the keywords in the content keyword library associated with one or more sponsors. Such a generated word piece with a low generation score can be referred to as a noise word piece.

[0035] A second problem with previous Trie-based Seq2Seq models occurs when a common prefix is encountered in the Trie that does not have a future object. For example, in a Trie-based Seq2Seq model, a common prefix can lead to a relevant keyword; however, an expected suffix of the common prefix can not exist in the Trie and, thus, the desired keyword field can not be generated. Simply removing the common prefix from the Trie does not solve this problem because these prefixes are common in desired keyword fields and, thus, need to generate such desired keyword fields. Moreover, a Trie-based Seq2Seq model also encounters a word piece selection problem when several predicted suffixes have similar high generation scores. While each of the suffixes with a high generation score can be used to compose a fluent sentence, keyword expansion using such suffixes can not lead to a desired keyword field because previous Trie-based Seq2Seq models do not have enough information to determine which suffix will lead to a desired path in the Trie and, thus, to a desired keyword field.

[0036] According to examples of the present disclosure, by using a Trie-based NLG model to simultaneously predict several word pieces ahead, future generation scores related to future word pieces in the Trie can be used to look ahead in the Trie and optimize the search space. That is, for a Trie-based search, the next word piece to be generated by the model is constrained to a possible suffix that exists in the Trie, where a possible suffix is a suffix of one or more word pieces in the partial hypothesis of the decoding.

[0037] More specifically, while a NLG Seq2Seq model can predict a next word piece or word based on a generation score of the word or word piece, examples of the present disclosure modify such a generation score to a weighted sum of the generation score and a future word piece generation score to optimize the search space. Each predicted word piece or word can then be ordered according to the modified score and can be selected as the next element in the segment or output. However, the original generation score of the selected word piece or word can be used when predicting ahead based on the partial hypothesis of the decoding.

[0038] Figure 1An example system 100 for performing a lookahead strategy for a Trie-based generative retrieval model is depicted in accordance with examples of the present disclosure. The system 100 can allow a user 102 to interact with a computing device 104 that renders content 106 at a display of the computing device 104. The rendered content 106 can be received from a computing device or server 114 via a network 112 and can be displayed as, for example, a webpage, a portion of an application or app. The rendered content 106 can allow a user, such as the user 102, to input a query 108 including, for example, one or more words, phrases, text segments, or symbols, and provide the query 108 to a search engine service 116 via the network 112, where the search engine service 116 can be running at or otherwise hosted by the computing device or server 114. The computing device or server 114 can include one or more computing devices that provide a distributed or cloud-based service, such as the search engine service 116. The search engine service 116 can provide content to the computing device 104 based on the query 108. For example, the search engine service 116 can receive the query 108 and identify content that matches or otherwise is similar to the query 108. In some examples, a search index that indexes content can be utilized to determine content results that match or otherwise are similar to the query 108. The search engine service 116 can then assemble the results and provide the assembled results as, for example, content 110 to the computing device 104. The content 110 can be rendered at the display of the computing device 104 and can include, for example, results 115A, 115B, 115C that match or otherwise are similar to the query 108. The content 110 can also include sponsored content 134 and 135, which includes sponsored content, also referred to as advertisements, and is rendered at the display of the computing device 104 in response to the query 108.

[0039] To determine sponsored content 134 and / or 135 to provide in response to query 108, search engine service 116 can include query keyword expander 118; query keyword expander 118 can receive query 108 and expand query 108 into a plurality of expanded text segments 128. In some examples, sponsored content 134 and / or 135 can be advertisements. Query keyword expander 118 can utilize a Seq2Seq model that employs an encoder to encode a query into a source sequence representation and a decoder that provides one or more text segments as a conditional likelihood based on the source sequence representation and a previously predicted segment (also referred to as a decoded partial hypothesis). The plurality of expanded text segments 128 can be provided to sponsored content service 130, which receives the plurality of expanded text segments 128 and uses query matching service 132 to identify sponsored content associated with the plurality of expanded text segments 128. For example, query matching service 132 can identify sponsored content 134 based on the plurality of expanded text segments 128 and a plurality of keywords 136 associated with sponsored content 138 in sponsored content repository 140. The plurality of keywords 136 can correspond to a plurality of keyword bids that have been placed by or otherwise associated with a sponsor; thus, sponsored content associated with a sponsor that has the highest bid for one or more keywords matching one or more of the plurality of expanded text segments 128 can be provided in response to query 108. Accordingly, sponsored content service 130 can provide sponsored content 142 or a position associated with the sponsored content to search engine service 116, such that search engine service 116 can provide sponsored content 142 as part of a response to user-entered query 108. Accordingly, sponsored content can be provided as part of content 110 and can be rendered to a display of computing device 104.

[0040] In some examples, query keyword expander 118 can provide or generate the plurality of expanded text segments 128 based on vocabulary 126A, 126B, where vocabulary 126A and 126B can include a plurality of text segments. The plurality of text segments can correspond to key fields as arranged in Trie 124. As previously noted, a Trie is a tree-like data structure that has nodes storing each keyword. By structuring the nodes in a particular way, a text segment including one or more words can be retrieved from the Trie by traversing a branching path of the tree. Vocabulary 126A, 126B containing Trie 124 can be accessed via network 112; alternatively or additionally, vocabulary 126A containing Trie 124 can reside in or otherwise be part of query keyword expander 118.

[0041] Figures 2A-2EAdditional examples of a Trie are provided in accordance with examples of the present disclosure. More specifically, Figure 2A Details of the first Trie 200A are depicted. As a non-limiting example, the first Trie 200A can include keywords or text segments 202 that can be associated with keyword bids provided by one or more sponsors. For example, a text segment can include the phrase "best hotel of tokyo" that, when generated as part of a query, such as query 108, will return content, such as an advertisement, of a sponsor associated with such a phrase. As another example, a text segment can include the phrase "texas hotels." When a query, such as query 108, is generated for "texas hotels," sponsor-specific content can be displayed, where the sponsor can have a bid, or otherwise have provided some means of compensation, for the content to be displayed when the query or an expanded text segment associated with the query matches the desired text segment.

[0042] Each text segment 202 is represented as a plurality of word units or nodes. That is, each word in a text segment 202 can be arranged as an ordered word unit in the first Trie 200A. As an example, the first Trie 200A can include a start word unit 204 that generally identifies a start node of the Trie. A next word unit 206 associated with the word "the" can follow the start word unit 204. Since "the" is a commonly used word, additional word units can be arranged as suffixes of "the." For example, "texas," "best" 208, and "flight" are suffixes of the word unit "the." As further depicted in the first Trie 200A, each word in each text segment 202 can be present in an ordered manner. That is, the phrase "best hotel of tokyo" can include word units 208, 212, 216, and 220. In some examples, an end node or leaf node can include a sentence end (e.g., a period) that indicates an end of a particular text segment or sentence. <eos>tokens 224. Thus, a phrase or sentence can include a sequence of tokens 224 beginning with a start token 204 and ending with a leaf node or <eos>All lexical units between lexical unit 224.

[0043] like Figure 2B As shown, for example, keyword 236 can be added to an existing first Trie 200A to generate a second Trie 200B. Because the terms "best" and "hotel" are already included in the first Trie 200A, the terms "in" 240 and "toronto" 244 can be added to the "hotel" terminus 212. As another example, keyword 252 can be added to an existing second Trie 200B to generate a third Trie 200, such as... Figure 2C As shown. Since the "best" lexical unit, "hotel" lexical unit, and "in" lexical unit are already included in the second Trie 200B, the lexical unit "texas" 256 can be added to the "in" lexical unit 240. As another example, text segment 264 can be added to the existing third Trie 200C to generate the fourth Trie 200, as shown. Figure 2D As shown. Since the "best" morpheme is already included in the third Trie 200C, the "wine" morpheme 268 can be added to the "best" morpheme. Of course, if needed, <eos>Tokens can be added to represent the end of a text segment. In examples, a text segment can correspond to a keyword and / or a sequence or segment of keywords.

[0044] According to some examples of the present disclosure, such as Figure 2E A Trie, such as the Trie shown in FIG. 11, can be used to constrain the output predictions made by the Seq2Seq model at one or more time steps to tokens that appear within the Trie. Thus, the text segment that is most similar to an input query, such as query 108, can be generated by the Seq2Seq model based on the vocabulary of the Trie.

[0045] Figure 3 Additional details of a decoding process 300 associated with a Seq2Seq model 302 according to examples of the present disclosure are depicted. A Seq2Seq model is an encoder-decoder based natural language generation model that maps an input sequence to an output sequence. The Seq2Seq model transforms a received sequence into an output sequence. The Seq2Seq model can use one or more of a recurrent neural network (RNN), a long short-term memory (LSTM) neural network, a gated recurrent unit (GRU) neural network, or other machine learning structures. The main components of a Seq2Seq model include an encoder 308 and a decoder 312. The encoder 308 transforms each item input to the encoder into a source sequence representation 318 that contains the item and its context, where the context of each item can be the output of a previous step and can correspond to a decoded partial hypothesis. The decoder 312 generally reverses the process to transform the source sequence representation 318 into an output item using the previous output or decoded partial hypothesis as an input context for each step. The Seq2Seq model 302 can be jointly trained to maximize the conditional probability of a target sequence given an input sequence.

[0046] An input sequence 304, which can correspond to a query 108, can be received by an encoder 308 of a Seq2Seq model 302; the input sequence 304 can be input one word at each time step 316, 320 to generate a source sequence representation 318. In an example, the encoder 308 can include a multi-layer Transformer encoder with multi-head self-attention mechanisms, such as the encoder discussed by Yan et al. in the publication “ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training,” which is incorporated by reference herein in its entirety for all it teaches and for all purposes. The encoder 308 can encode the input sequence 304 into a source sequence representation 318. The encoded source sequence representation 318 can then be provided to a decoder 312. The decoder 312 can include an LSTM+Tree model, as suggested by Chen et al. in the publication “An end-to-end generative retrieval method for sponsored search engine decoding efficiently into a closed target domain,” which is incorporated by reference herein in its entirety for all it teaches and for all purposes. In some examples, the decoder 312 model can implement a copy mechanism to emphasize the generation scores of those words that appear in the input sequence 304. That is, the output words typically have some overlap with the input query, and the copy mechanism allows the model to select word tokens directly from the input sequence 304 when synthesizing the answer.

[0047] According to examples of the present disclosure, with the input sequence 304 and the source sequence representation 318 generated by the encoder 308, the decoder 312 can generate an output sequence by predicting one word token or word at a time for each next predicted word and for one or more look-ahead or future predictions; each prediction can include a probability distribution for all possible words for each respective position in the output sequence (e.g., next prediction, future prediction). In other words, for each word that the decoder 312 is predicting, it will generate an array of probabilities that indicate how likely a particular word is to be the next output token and / or how likely a particular word is to be a future output token. Each of these probabilities can be considered a generation score.

[0048] As previously described, the predicted next token and the predicted future tokens can be constrained to words in the Trie, and more specifically, to paths along the Trie. Thus, the decoder 312 can determine generation scores for the token in the Trie that corresponds to the next token in the Trie and one or more future tokens in the Trie. For example, assuming "the" 317 is selected at time step 316, "best" 321 is selected at the third time step 322, and "hotel" 323 is selected at the third time step 322, generation scores for the predicted tokens "in" and "of" can be generated at time step 324, where the predicted tokens "in" and "of" correspond to suffixes of the last predicted word (e.g., "hotel" 323) in the decoded partial hypothesis (e.g., "the best hotel") that occur along a path of the Trie. According to examples of the present disclosure, the decoder 312 can also generate generation scores for the future tokens "toronto," "texas," and "tokyo" as look-ahead predictions, where the look-ahead predictions can influence the selection of the predicted token "in" or "of" as the next word in the output sequence.

[0049] According to examples of the present disclosure, the modified generation score for the predicted token "in" can be based on the likelihood that "in" is the next word in the output sequence given the previous predicted segment "the best hotel" and the maximum likelihood of the next future token provided by the Trie (e.g., the maximum of the likelihood that "toronto" is the next word in the output sequence given the previous predicted segment "the best hotel" or the likelihood that "texas" is the next word in the output sequence given the previous predicted segment "the best hotel"). "Toronto" and "texas" are the next suffix nodes (e.g., at time step 7 326) that correspond to the "the best hotel" route through the Trie when the predicted token "in" is selected. The generation score for the token "of" can be based on the previous predicted segment "the best hotel" and the maximum likelihood of the next future token(s) provided by the Trie (e.g., the maximum of the likelihood that "tokyo" is the next word in the output sequence given the previous predicted segment "the best hotel"). "Tokyo" is the next suffix node (e.g., at time step 7 326) that corresponds to the "the best hotel" route through the Trie when the predicted token "of" is selected. Although Figure 3 A single look-ahead at time step 326 is depicted, but additional look-ahead time steps are also contemplated.

[0050] Figures 4A-4E Additional details are provided regarding the decoding process associated with a Seq2Seq model according to examples of this disclosure. Encoder 402 may be the same as or similar to encoder 308 previously discussed. Encoder 402 may receive input sequence 401 and encode the input sequence into source sequence representation h 404. Decoder 406 may be the same as or similar to decoder 312 previously discussed. According to examples of this disclosure, the generation score of the next predicted word can be modified based on the scores of one or more future words anticipated from previous predicted segments. As previously described, problems associated with prediction can be reduced where the input includes common prefixes but does not return the desired output, or where the input may include noise or include spelling errors in the target sequence.

[0051] Figures 4A-4E The decoding example provided uses information related to the suffix of the predicted lexical to modify the generation score of the predicted lexical, where the predicted lexical and future lexical are constrained to lexical units in the Trie. Decoder 406 can look ahead. There are several steps, where for an n-gram generative model, It can be equal to n-1. Therefore, when determining the prediction of the next output in the output sequence, generation scores can be generated simultaneously for the next predicted word and n-1 future words; thus, the next first predicted word and n-1 future words can be used as the lookahead for this prediction. The residual weight λ can be set to control the weights of the generation score of the next word and its lookahead scores.

[0052] like Figure 4A As shown, decoder 406 can simultaneously generate scores for both the word to be predicted [tbp] 408 and the future word to be predicted [tbp] 410. The generated score for the word to be predicted [tbp] 408 can be called... Furthermore, the generation score of the future predictable word unit [tbp]410 can be called... The previously decoded sequence can be called a seq, such as... Figure 4A As shown, it may include <start>or statement begin <bos>The first suffix of tokens 412. seq can be referred to as si, and for each node p1 in si, one look ahead step suffix of p1 is denoted s2. The generation score of the next first token p1 is modified according to Equation 1 to obtain a modified generation score.

[0053] Equation 1

[0054] Thus, as shown, decoder 406 can generate a generation score Figure 4A for next predicted token 414 and a generation score for next predicted token 416 Figure 2E where predicted tokens 414 and 416 are constrained to a Trie, such as the Trie of That is, the generation score Figure 4A may equal the probability that token 414 "the" is the next token in the output sequence given the input sequence 401 and the partially decoded output sequence, as shown, which can equal <start>or statement begin <bos>Token or label. Generating a score may equal the probability that the token 416 "a" is the next token in the output sequence given the input sequence 401 and the partially decoded output sequence, as Figure 4A shown, which can equal <start>or statement begin <bos>tokens or tokens. In addition, the decoder 406 can generate a generation score for the future token 420 and a generation score for the future token 422 where the tokens 418, 420, and 422 are suffixes of the predicted token 414, as indicated by the Trie. The generation score may equal the probability that the token 418 "texas” is the next token in the output sequence given the input sequence 401 and the partially decoded output sequence, as indicated by Figure 4A which can equal <start>or statement begin <bos>tokens or labels. Similarly, the generation score and may equal the probability that the respective token 420 "best" or 422 "flight" is the next token in the output sequence given the input sequence 401 and the partially decoded output sequence, as Figure 4A shown, which can equal <start>or statement begin <bos>token or tokenization. If a suffix exists based on the predicted token 416, the decoder 406 will generate a generation score for such a suffix.

[0055] According to examples of the present disclosure, the decoder 406 can generate a modified generation score for a predicted token based on the generation score of the predicted token and the maximum generation score of future tokens . For example, the modified generation score of the token 414 can be generated according to: As shown by reference number 424, where the modified generation score is equal to the generation score plus the maximum generation score of the tokens 418, 420, and 422 multiplied by where the residual weight λ can be set to control the weight of the generation score of the predicted token and its lookahead score. The modified generation score may be associated with the token 414. The modified generation score of the token 416 can be generated in a similar manner. Thus, selecting the next predicted token as the next token in the output sequence can be based on the modified generation score . As Figure 4A shown, between the tokens 414 and 416, the token 414 can have the highest modified generation score .

[0056] Continuing Figure 4A with the example, Figure 4B depicts a subsequent time step where the decoder 406 can generate a generation score for both the token to be predicted [tbp] 426 and the future token to be predicted [tbp] 430. The generation score of the token to be predicted [tbp] 426 can be referred to as and the generation score of the future token to be predicted [tbp] 430 can be referred to as . The previously decoded sequence can be referred to as seq, as Figure 4B shown, which can include the sequence <start>the" or <bos>the”.

[0057] Thus, as Figure 4B illustrated, the decoder 406 can generate a generation score for the next predicted token 420 for the next predicted token 418 for the next predicted token 422 Figure 2E where the predicted tokens 420, 418, and 422 are constrained to a Trie, such as the Trie of . That is, the generation score Figure 4B may be equal to the probability of the next predicted token 420 given the input sequence 401 and the partially decoded output sequence (as illustrated, which can include the <start>In the case of the word token 420 "best" the probability of the word token being the next word token in the output sequence. The generation score may be equal to the probability of the given input sequence 401 and the partially decoded output sequence (as Figure 4B shown, which can include <start>In the case of the word token 418 "texas", the probability of the word token 418 "texas" being the next word token in the output sequence is generated may equal the probability of the next word token in the output sequence given the input sequence 401 and the partially decoded output sequence (as Figure 4B As shown, this can include <start>the”) is the probability of the next token in the output sequence.

[0058] Further, the decoder 406 can generate a generation score for the future token 434 for the future token 436 for the future token 438 and for the future token 440, which can be an end-of-sentence identifier where the tokens 432 and 434 are suffixes of the predicted token 420, the tokens 436 and 438 are suffixes of the predicted token 418, and the token 440 is a suffix of the predicted token 422, as indicated by the Trie. The generation scores , , , and may equal the probability of the input sequence 401 and the partially decoded output sequence (as Figure 4B indicated, which can include <start>the” case, the corresponding token 432, 434, 436, 438, or 440 is the probability that the next token in the output sequence is the predicted token.

[0059] As previously described with respect to Figure 4A the decoder 406 can generate a modified generation score for a predicted token based on the generation score of the predicted token and the maximum generation score of a future token. For example, the modified generation score of token 420 can be generated according to where the modified generation score is equal to the generation score plus the maximum generation score in tokens 432 and 434 multiplied by where the residual weight λ can be set to control the weight of the generation score of the predicted token and its lookahead score. The modified generation score may be associated with token 420. Similarly, the modified generation scores of tokens 418 and 422 can be generated in a similar manner. Thus, selecting the next predicted token as the next token in the output sequence can be based on the modified generation scores Figure 4B . As shown, between tokens 418, 420, and 422, token 420 can have the highest modified generation score .

[0060] Continuing Figure 4A and Figure 4B the example, Figure 4C depicts a subsequent time step in which the decoder 406 can simultaneously generate generation scores for a to-be-predicted token [tbp] 442 and a future to-be-predicted token [tbp] 444. The generation score of the to-be-predicted token [tbp] 442 can be referred to as and the generation score of the future to-be-predicted token [tbp] 444 can be referred to as The previously decoded sequence can be referred to as seq, as Figure 4C shown, which can include the sequence <start>"the best”.

[0061] Thus, as Figure 4C illustrated, the decoder 406 can generate a generation score for the next predicted token 432 , where the predicted tokens 432 and 434 are constrained to the Trie as the next suffix along the Trie path.

[0062] Further, the decoder 406 can generate a generation score for the lookahead token 432 and a generation score for the lookahead token 434. As previously described with respect to Figure 4A and Figure 4B , the decoder 406 can generate a modified generation score for a predicted token based on the generation score for the predicted token and the maximum generation score for the future token. For example, the modified generation score for the token 432 can be generated according to:

[0063] where the modified generation score is equal to the generation score plus the maximum of the generation score and multiplied by , where the residual weight λ can be set to control the weight of the generation score for the predicted token and its lookahead score. The modified generation score may be associated with the token 432. Similarly, the modified generation score for the token 434 can be generated in a similar manner. Thus, selecting the next predicted token as the next token in the output sequence can be based on the modified generation score . As Figure 4C illustrated, between the tokens 432 and 434, the token 432 can have the highest modified generation score .

[0064] Continuing Figures 4A-4C with the example, Figure 4D depicts a subsequent time step in which the decoder 406 can generate a generation score for both the token to be predicted [tbp] 452 and the future token to be predicted [tbp] 454. The generation score for the token to be predicted [tbp] 452 can be referred to as and the generation score for the future token to be predicted [tbp] 454 can be referred to as . The previously decoded sequence can be referred to as seq, as Figure 4D illustrated, which can include the sequence " <start>"the best hotel”.

[0065] Thus, as Figure 4D shown, the decoder 406 can generate a generation score for the next predicted token 446 , where the predicted tokens 446 and 448 are constrained to the Trie as the next suffix along the Trie path.

[0066] Further, the decoder 406 can generate a generation score for a lookahead token of the token 446 and a generation score for a lookahead token 448. As previously described with respect to Figures 4A-4C , the decoder 406 can generate a modified generation score for a predicted token based on the generation score for the predicted token and the maximum generation score for future tokens. For example, the modified generation score for the token 446 can be generated according to:

[0067] where the modified generation score is equal to the generation score plus the maximum of the generation score and multiplied by , where the residual weight λ can be set to control the weight of the generation score for the predicted token and its lookahead score. The modified generation score may be associated with the token 446.

[0068] Similarly, the modified generation score for the token 448 can be generated in a similar manner. Thus, selecting the next predicted token as the next token in the output sequence can be based on the modified generation score . As Figure 4D shown, between the tokens 446 and 448, the token 446 can have the highest modified generation score . When generating the next token for "the best hotel”, "in” and "of” are suffix tokens from the Trie. Although "in” and "of” are both generally good suffixes for generating the keyword, "of” does not have a future token with a high generation score, while the future token for "in” covers the desired token "texas”, which provides a high generation score for "texas” and a higher modified generation score for "in”. Thus "in” is generated.

[0069] Continuing Figures 4A-4D the example, Figure 4E A subsequent time step is depicted, in which the decoder 406 can simultaneously generate generation scores for the to-be-predicted token [tbp] 460 and the future to-be-predicted token [tbp] 462. The generation score for the to-be-predicted token [tbp] 460 can be referred to as , and the generation score for the future to-be-predicted token [tbp] 462 can be referred to as . The previously decoded sequence can be referred to as seq, as shown in Figure 4E , which can include the sequence <start>"the best hotel in”.

[0070] Thus, as Figure 4D illustrated, the decoder 406 can generate a generation score for the next predicted token 458. where the predicted tokens 458 and 456 are constrained to the Trie as the next suffix along the Trie path.

[0071] Further, the decoder 406 can generate a generation score for the respective look-ahead token of the token. As previously described with respect to Figures 4A-4D , the decoder 406 can generate a modified generation score for the predicted token based on the generation score of the predicted token and the maximum generation score of the future token. For example, the modified generation score of the token 458 can be generated according to:

[0072] .

[0073] Similarly, the modified generation score of the token 456 can be generated in a similar manner. Thus, selecting the next predicted token as the next token in the output sequence can be based on the modified generation score . As Figure 4E illustrated, between the tokens 458 and 456, the token 458 can have the highest modified generation score . Further, this token 458 covers the desired token "texas" provided in the input sequence 401. Since the next suffix in the Trie based on the path including "the best hotel in texas" is the end-of-sentence identifier, the decoder 406 can output the decoded sequence "the best hotel in texas" as the possible candidate sequence.

[0074] In some examples, the size of the bundle search (e.g., the bundle width) can determine how many results with the highest modified generated score can be passed to the parent node and / or how many Trie paths can be traversed at any given time. For example, a bundle search with a bundle width of 2 could allow both lexical 418 and lexical 420 to be selected as possible additions to the decoding sequence. Alternatively or additionally, only lexical terms associated with the highest modified generated score can be selected as additions to the decoding sequence. In some examples, high-confidence suffixes might be considered before explicitly looking ahead. For example, if no high-confidence lookahead lexical term exists, a low future score can be passed backward. Conversely, if any noisy lexical term in the suffix has a high-confidence lookahead lexical term, additional high-confidence scores will be passed over the "noise" (e.g., lexical terms with low generated scores) to provide an enhanced modified generated score for the lexical term being predicted.

[0075] The modified generated scores can be used to sort and select the best sequence. (The score is then used to...) When using beam search techniques to compute decoded sequence scores, the results may differ from those of generative models that do not employ a look-ahead strategy. In other words, the look-ahead strategy modifies the output sequence score, which can lead to error accumulation. Therefore, the modified generated score... It can be used to sort and select the best sequence with the highest score, but the generated score... (For example, the raw generated score without a look-ahead strategy) can be stored and used to select the next predictive term.

[0076] Figure 5 Additional details of the decoding process 500 associated with a Seq2Seq model 504 according to an example of this disclosure are described. In the example, Figure 5 and Figure 3 The difference lies in that the decoder of the Seq2Seq model 502 can provide generation scores for the predicted lexical unit 510 and two lookahead lexical units 512 and 514, where the first lookahead lexical unit 512 is a suffix of the predicted lexical unit 510, and the second lookahead lexical unit 514 is a suffix of the first lookahead lexical unit 512. As previously mentioned, the predicted next lexical unit and the predicted future lexical unit can be constrained to words in the Trie, and more specifically, to words along the path of the Trie.

[0077] Figure 6 Examples of this disclosure are described for use with Figure 5 Additional details regarding the decoding process associated with the Seq2Seq model. Encoder 602 may be the same as or similar to encoder 308 discussed previously. Encoder 602 may receive input sequence 601 and encode the input sequence into source sequence representation h 604. Decoder 606 may be the same as or similar to decoder 312 discussed previously. According to examples of this disclosure, the generation score of the next predicted word can be modified based on the scores of one or more future words anticipated from previous predicted segments. Figure 6 As shown, decoder 606 can simultaneously generate scores for the unpredicted word [tbp]608, the future unpredicted word [tbp]610, and another future unpredicted word 612, where the future unpredicted word 612 is a suffix of the unpredicted word [tbp]610. The generated score for the unpredicted word [tbp]608 can be called... The generation score of the future predictable word [tbp]610 can be called Furthermore, the generation score of 612 for the future predicted word can be called... Therefore, the modified generation score for example 614 could be:

[0078] .

[0079] because Figures 2A-4E An example is provided where the selected words to be added to the output sequence are based on predicted lexical units and their suffixes, while Figure 5 and Figure 6 An example is provided where the selected word to be added to the output sequence is based on predicted lexical units, the suffixes of the predicted lexical units, and the suffixes of the predicted lexical units. It is conceivable that any number of future look-ahead lexical units can be used to select the next word in the output sequence. Table 1 provides an example recursive algorithm for generating and ranking lexical units based on the input sequence to produce a text segment.

[0080] Table 1

[0081]

[0082] Figure 7 Additional details of the search engine service 116, the sponsored content service 130, and the Trie server 748 are depicted in accordance with examples of the present disclosure. As previously described, the search engine service 116 can provide content to a computing device based on a received query, such as the query 702. For example, the search engine service 116 can receive the query 702 at the search service 728 and use the search service 728 to identify content that matches or otherwise is similar to the query 702. In some examples, a search index that indexes content that can be located in the content repository 732 can be utilized to determine content that matches or otherwise is similar to the query 702. The search engine service 116 can then assemble results and provide the assembled results to the computing device. The content can also include one or more advertisements that are rendered at a display of the computing device in response to the query 702.

[0083] To identify sponsored content to be provided as part of the content that is rendered to the display, the search engine service 116 can include a query keyword expander 118 that receives the query 702 and expands the query 702 into a plurality of contextually aware text segments. The query keyword expander 118 can utilize a Seq2Seq model that employs an encoder 308 that encodes the query into a source sequence representation and a decoder 312 that provides one or more text segments as a conditioned likelihood based on the source sequence representation and a previously predicted segment, such as a word, as input. As previously described, the encoder 308 can be a multi-layer encoder and the decoder can be a multi-layer decoder. The plurality of expanded text segments can be provided to the sponsored content service 130 that receives the plurality of expanded text segments and uses a query matching service 132 to identify sponsored content associated with the plurality of expanded text segments. For example, the query matching service can determine sponsored content to display based on the plurality of expanded text segments and a plurality of keywords 136 associated with sponsored content 138 in a sponsored content repository 140. In some examples, the sponsored content can be an advertisement. The plurality of keywords 136 can correspond to a plurality of keyword bids that have been placed by or otherwise are associated with a sponsor; thus, sponsored content associated with the sponsor that has the highest bid for one or more keywords that match one or more of the plurality of expanded text segments can be provided in response to the query 702. Accordingly, the sponsored content service 130 can provide the sponsored content to the search engine service 116 such that the search engine service 116 provides the sponsored content as part of a response to the user input query 702. Thus, the sponsored content can be provided in addition to other content and can be rendered to a display of the computing device.

[0084] In some examples, the query keyword expander 118 can provide or generate multiple expanded text segments based on a vocabulary, where the vocabulary may include multiple text segments. These multiple text segments may correspond to keywords with keyword bids and can be arranged as a Trie as described above. The Trie may reside in a Trie repository 736 and can be generated, modified, or altered using a Trie generator 740. The Trie can also be traversed using a Trie traversal engine 744. A Trie server 748 can provide access to the generation and traversal of the Trie data structure.

[0085] Figure 8 An overview of example method 800 for receiving key fields associated with a query bid and adding the received key fields to a Trie is shown. The general order of the steps in method 800 is as follows: Figure 8 As shown. Typically, method 800 begins at 802 and ends at 824. Method 800 may include more or fewer steps, or may be combined with... Figure 8 The steps shown are arranged in a different order. Method 800 can be executed as a set of computer-executable instructions that are executed by a computer system and encoded or stored on a computer-readable medium. Furthermore, method 800 can be executed by gates or circuits associated with a processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SoC), graphics processing unit (GPU), or other hardware device. References will be incorporated herein by reference. Figures 1-7 Method 800 is explained by describing the system, components, modules, software, data structures, user interface, etc.

[0086] Method 800 begins at operation 804, where an instruction to add a key field to a Trie can be received. For example, a sponsor can bid on one or more keywords. Once a bid is established, one or more keywords can be added to the Trie. Therefore, method 800 can proceed to 808, where the Trie to receive the keywords or key fields can be identified. In some examples, the Trie may include a vocabulary of all key fields from the sponsored search provider. For example, a sponsored search provider such as a search engine may store all keywords and / or key fields as a vocabulary in a single Trie. Alternatively or additionally, the sponsored search provider may segment the Trie based on search categories or groupings. In an example, the Trie may reside in a Trie repository 376. Method 800 can proceed to 812, where keywords(s) associated with the query bid can be received. More specifically, for example, in step 816, Trie server 364 may receive the keywords(s) and add them(s) to an existing Trie using Trie generator 368. Therefore, in step 820, the Trie with the received keywords(s) can be stored in a Trie repository such as Trie repository 736. Once stored, method 800 can end at 824.

[0087] Figure 9 An overview of an example method 900 for identifying sponsored content to be displayed on a computing device based on a query is shown. The general sequence of steps in method 900 is as follows: Figure 9 As shown. Typically, method 900 begins at 904 and ends at 932. Method 900 may include more or fewer steps, or may be combined with... Figure 9 The steps shown are arranged in a different order. Method 900 can be executed as a set of computer-executable instructions that are executed by a computer system and encoded or stored on a computer-readable medium. Furthermore, method 900 can be executed by gates or circuits associated with a processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SoC), graphics processing unit (GPU), or other hardware device. References will be incorporated herein by reference. Figures 1-8 Method 900 is explained by describing the system, components, modules, software, data structures, user interface, etc.

[0088] Method 900 begins at operation 904 and proceeds to operation 908, where it receives a text segment associated with the query. The method may proceed to 912 to identify a Trie containing a vocabulary used to expand or otherwise generate multiple text segments associated with the query. In the example, multiple text segments can be considered as an expanded list of text segments associated with the query. The method may proceed to 916, where additional text segments are generated according to the examples described herein. For example, query keyword expander 118 may encode the received text segments using encoder 308 and provide the encoded text segments to decoder 312. Decoder 312 may generate scores for one or more nodes in the identified Trie. Scores may be based on and / or related to the previously described... Figure 10 The described prospective prediction method. Once multiple text segments are generated, method 900 can proceed to operation 920, where one or more of the generated text segments can be sent to a sponsored content service, such as sponsored content service 130. For example, sponsored content service 130 can match one or more of the generated text segments with sponsored content and / or keywords associated with the sponsored content, and send the sponsored content to search engine service 116. Method 900 can proceed to 928, where search engine service 116 can provide an add-on to a computing device, thereby rendering the sponsored content onto a display associated with the computing device. Method 900 can then terminate at 932.

[0089] Figure 10 An overview of an example method 1000 for generating an output sequence based on an input sequence is shown. The general order of the steps in method 1000 is as follows: Figure 10 As shown. Typically, method 1000 begins at 1004 and ends at 1036. Method 1000 may include more or fewer steps, or may be combined with... Figure 10 The steps shown are arranged in a different order. Method 1000 can be executed as a set of computer-executable instructions that are executed by a computer system and encoded or stored on a computer-readable medium. Furthermore, method 1000 can be executed by gates or circuits associated with a processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), system-on-a-chip (SoC), graphics processing unit (GPU), or other hardware device. References will be incorporated herein by reference. Figures 1-9 The system, components, modules, software, data structures, user interface, etc., are described to explain method 1000.

[0090] The method 1000 begins at operation 1004 and proceeds to operation 1008, where a predicted word token can be identified based on a received decoded partial hypothesis. More specifically, the decoded partial hypothesis can correspond to an intermediate output sequence that has been generated by a decoder (such as the decoder 312) based on an input sequence that was encoded by an encoder (such as the encoder 308). Thus, the decoder can provide a prediction of a number of next predicted word tokens. At 1012, for each predicted word token that resides in the Trie, the decoder can generate a generation score for the predicted word token. In some cases, the decoder can generate the generation score when initially providing a prediction of the next predicted word token that is constrained by the Trie. In other examples, the decoder can first look at the Trie to determine the next predicted word token and generate the generation score based on the identified predicted word token.

[0091] The method 1000 can proceed to 1016, where a lookahead suffix of the predicted word token is identified, such that a generation score can be generated for the identified lookahead suffix at 1020. In some cases, the decoder can generate the generation score when initially providing a prediction of the next predicted word token and the lookahead suffix that is constrained by the Trie. In other examples, the decoder can first look at the Trie to determine the next predicted word token and the lookahead token and generate the generation score based on the identified predicted word token and the lookahead token. The method 1000 can proceed to 1024, where a modified generation score can be generated for the predicted word token. As previously discussed with respect to Figures 3-6 the decoder can generate the modified generation score for the predicted word token based on the generation score of the predicted word token and the maximum generation score of the future word tokens The method 1000 can then proceed to 1028, where one or more predicted word tokens having a maximum modified generation score can be selected. In some examples that utilize a greedy search, only the predicted word token having the maximum modified generation score is selected. In examples that can utilize a beam search, the top x predicted word tokens having the maximum modified generation score can be selected, where x is equal to the beam width of the beam search. Thus, the method 1000 can proceed to 1032, where the selected predicted word token(s) are added to the partially decoded sequence. The method 1000 can repeat at 1008, depending on, for example, the path length of the Trie or the maximum text segment length. For example, in the case of additional word tokens in the Trie path, the method 1000 can proceed to 1008 to generate the next segment or word in the partially decoded sequence. In the case of encountering an end of sentence as a predicted word token or as a lookahead suffix, the method 1000 can end at 1036.

[0092] Figures 11-13 The accompanying description provides a discussion of various operating environments in which aspects of the disclosure can be practiced. However, the Figures 11-13 The devices and systems illustrated and discussed in this specification are for the purpose of example and illustration and are not limiting of the numerous configurations of computing device that can be utilized for practicing aspects of the disclosure described herein.

[0093] Figure 11 is a block diagram illustrating physical components (e.g., hardware) of a computing device 1100 with which aspects of the disclosure can be practiced. The computing device components described below can be suitable for the computing and / or processing devices described above. In a basic configuration, the computing device 1100 can include at least one processing unit 1102 and a system memory 1104. Depending on the configuration and type of computing device, the system memory 1104 can comprise, but is not limited to, volatile storage (e.g., random access memory (RAM)), non-volatile storage (e.g., read-only memory (ROM)), flash memory, or any combination

[0094] The system memory 1104 can include an operating system 1105 and one or more program modules 1106, suitable for running software applications 1120 such as one or more components supported by the systems described herein. For example, the system memory 1104 can include a query keyword expander 1123, an encoder 1125, a decoder 1127, a sponsored content service 1135, a query matching service 1133, a Trie generator 1129, and a Trie traversal engine 1131. The query keyword expander 1123 can be the same as or similar to the query keyword expander 118. The encoder 1125 can be the same as or similar to the previously described encoder 308. The decoder 1127 can be the same as or similar to the previously described decoder 312. The sponsored content service 1135 can be the same as or similar to the previously described sponsored content service 130. The query matching service 1133 can be the same as or similar to the previously described query matching service 132. The Trie generator 1129 can be the same as or similar to the previously described Trie generator 740. The Trie traversal engine can be the same as or similar to the previously described Trie traversal engine 1131. The operating system 1105 can be suitable for controlling the operation of the computing device 1100, for example.

[0095] Furthermore, embodiments of the disclosure can be practiced in conjunction with a graphics library, other operating systems, or any other application program and are not limited to any particular Figure 11 application or system. This basic configuration is illustrated in FIG. 11 by those components within dashed line 1108. The computing device 1100 can have additional features or functionality. For example, the computing device 1100 can also include additional data storage devices (removable and / or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 11 by removable storage device 1109 and non-removable storage device 1110. Figure 11 The storage device 1108, which can be implemented via a memory card, a Blu-ray® disc, or other non-removable, non-removable storage device 1110 is also shown.

[0096] As stated above, a number of program modules and data files can be stored in the system memory 1104. While executing on the processing unit 1102, the program modules 1106 can perform processes including, for example, one or more of the aspects described herein. Other program modules that can be used with the aspects of the present disclosure can include electronic mail and contacts applications, word processing applications, spreadsheet applications, database applications, slide presentation

[0097] Furthermore, embodiments of the disclosure can be practiced in an electrical circuit comprising discrete electronic elements, packaged or integrated electronic chips containing logic gates, a circuit utilizing a microprocessor, or by using Figure 11 Each or more of the illustrated components can be integrated on a single integrated circuit. Such a SOC device can include one or more processing units, graphics units, communications units, system virtualization units, and various application functionality all of which are integrated (or "burned") onto the chip substrate as a single integrated circuit. When operating via a SOC, the functionality described herein with regard to the capabilities of the client switching protocol can be operated via application-specific logic integrated with other components of the computing device 1100 on the single integrated circuit (chip). Embodiments of the disclosure can also be practiced using other technologies

[0098] The computing device 1100 can also have one or more input device(s) 1112 such as a keyboard, a mouse, a pen, a microphone, or voice or speech recognition device, a touch or swipe input device, etc. One or more output device(s) 1114 such as a display, speakers, a printer, etc. can also be included. The aforementioned devices are examples and others can be used. The computing device 1100 can include one or more communication connections 1116 allowing communications with other computing devices 1150. Examples of suitable communication connections 916 include, but are not limited to, radio frequency (RF) transmitter, receiver, and / or transceiver circuitry; universal serial bus (USB), parallel, and / or serial ports.

[0099] The term computer readable media as used herein can include computer storage media. Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, or program modules. The system memory 1104, the removable storage device 1109, and the non-removable storage device 1110 are all computer storage media examples (e.g., memory storage). Computer storage media can include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other article of manufacture which can be used to store information and which can be accessed by the computing device 1100. Any such computer storage media can be part of the computing device 1100. Computer storage media does not include a modulated data signal or other propagated or modulated data signals.

[0100] Communication media can be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" can describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0101] Figures 12A-12B A mobile computing device 1200, such as a mobile telephone, smart phone, wearable computer (such as a smart watch), a tablet computer, a laptop computer, and the like, is shown and can be used to practice embodiments of the present disclosure. In certain aspects, a client can be a mobile computing device. Referring to Figure 12A One aspect of the mobile computing device 1200 for implementing the aspects is shown in FIG. 12B. In a basic configuration, the mobile computing device 1200 is a handheld computer having both input elements and output elements. The mobile computing device 1200 typically includes a display 1205 and one or more input buttons 1210 that allow the user to enter information into the mobile computing device 1200. The display 1205 of the mobile computing device 1200 also functions as an output element for the mobile computing device 1200.

[0102] If included, optional side input element 1215 allows further user input. Side input element 1215 can be a rotate

[0103] In yet another alternative embodiment, mobile computing device 1200 is a portable phone system, such as a cellular phone. Mobile computing device 1200 can also include an optional keypad 1235. Optional keypad 1235 can be a physical keypad or a "soft” keypad generated on the touch screen display.

[0104] In various embodiments, output element comprises display 1205 for showing a graphical user interface (GUI), visual indicator 1220 (e.g., a light emitting diode), and / or audio transducer 1225 (e.g., a speaker). In some aspects, mobile computing device 1200 incorporates a vibration transducer for

[0105] Figure 12B is a block diagram illustrating the architecture of one aspect of a mobile computing device. That is, mobile computing device 1200 can incorporate a system (e.g., an architecture) 1202 to implement some aspects. In one embodiment, system 1202 is implemented as a "smart phone” capable of running one or more applications (e.g., browser, e-mail, calendaring, contact managers, messaging clients, games, and media clients / players). In some aspects, system 1202 is integrated as a computing device, such as an integrated personal digital assistant (PDA) and wireless phone.

[0106] One or more application programs 1266 can be loaded into the memory 1262 and run on or in association with the operating system 1264. Examples of the application programs include phone dialer programs, e-mail programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, Internet browser programs, messaging programs, and so forth. The system 1202 also includes a non-volatile storage area 1268 within the memory 1262. The non-volatile storage area 1268 can be used for storage of persistent information that should not be lost if the system 1202 is powered down. The application programs 1266 can use and store information in the non-volatile storage area 1268, such as e-mail or other messages used by an e-mail application, and the like. A synchronization application (not shown) also resides on the system 1202 and is programmed to interact with a corresponding synchronization application resident on a host computer to keep the information stored in the non-volatile storage area 1268 synchronized with the corresponding information stored at the host computer. As should be appreciated, other applications can be loaded into the memory 1262 and run on the mobile computing device 1200 described herein (e.g., search engine, extractor module, relevance ranking module, answer scoring module, etc.).

[0107] The system 1202 has a power supply 1270, which can be implemented as one or more batteries. The power supply 1270 might further include the ability to be

[0108] The system 1202 can also include a radio interface layer 1272 that works with the radio interface 1274 to facilitate wireless communication between the system 1202 and the "outside world.” The radio interface layer 1272 facilitates wireless communication between the system 1202 and devices other than the system 1202. This is in contrast to the radio interface 1274, which is specifically tailored to facilitate communication with one or more particular apparatuses, such as a base station 102 or a mobile device 104. Transmissions to and from the radio interface layer 1272 are conducted under the control of the operating system 1264. In other words, communications received by the radio interface layer 1272 can be disseminated to the application programs 1266 via the operating system 1264, and vice versa.

[0109] The visual indicator 1220 can be used to provide visual notifications, and / or the audio interface 1274 can be used for audible notifications via the audio transducer 1225. In the illustrated embodiment, the visual indicator 1220 is a light emitting diode (LED) and the audio transducer 1225 is a speaker. These devices can be directly coupled to the power supply 1270 so that when activated, they remain on for a duration dictated by the notification mechanism even though the processor 1260 and other components might shut down for conserving battery power. The LED can be programmed to remain on indefinitely until the user takes action to indicate that the device is powered on. The audio interface 1274 is used to provide audible signals to and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 1225, the audio interface 1274 can also be coupled to a microphone to receive audible input, such as to facilitate voice-activated

[0110] The mobile computing device 1200 implementing the system 1202 can have additional features or functionality. For example, the mobile computing device 1200 can also include additional data storage devices (removable and / or non-removable) such as, magnetic disks, optical disks, or tape. Such additional storage is illustrated in Figure 12B by the non-volatile storage area 1268.

[0111] Data / information generated or captured by the mobile computing device 1200 and stored via the system 1202 can be stored locally on the mobile computing device 1200, as described above. Alternatively, such data / information can be stored on any number of storage media that is accessible by the device via the radio interface layer 1272 or via a wired connection between the mobile computing device 1200 and a separate computing device associated with the mobile computing device 1200, for example, a server computer in a distributed computing network, such as the Internet. As

[0112] Figure 13 One aspect of the architecture of a system for processing data received at a computing system from a remote source, such as the personal computer 1304, the tablet computing device 1306, or the mobile computing device 1308 described above, is shown. Content at the server device 1310 can be stored in different communication channels or other storage types. For example, various documents can be stored using a directory service 1322, a portal 1324, a mailbox service 1326, an instant messaging store 1328, or a social networking site 1330.

[0113] As described above, one or more of the aforementioned program modules 1106 or software applications 1107 can be employed by the server device 1310 and / or the personal computer 1304, the tablet computing device 1306, or the mobile computing device 1308. For example, the server device 1310 can include a query keyword expander 1323, an encoder 1325, a decoder 1327, a sponsored content service 1335, a query match service 1333, a Trie generator 1329, and a Trie traversal engine 1331. The query keyword expander 1323 can be the same as or similar to the query keyword expander 118. The encoder 1325 can be the same as or similar to the previously described encoder 308. The decoder 1327 can be the same as or similar to the previously described decoder 312. The sponsored content service 1335 can be the same as or similar to the previously described sponsored content service 130. The query match service 1333 can be the same as or similar to the previously described query match service 132. The Trie generator 1329 can be the same as or similar to the previously described Trie generator 740. The Trie traversal engine can be the same as or similar to the previously described Trie traversal engine 1331.

[0114] The server device 1310 can provide data to and from client computing devices, such as the personal computer 1304, the tablet computing device 1306, and / or the mobile computing device 1308 (e.g., a smart phone), over the network 1315. For example, the computer system described above can be embodied in the personal computer 1304, the tablet computing device 1306, and / or the mobile computing device 1308 (e.g., a smart phone). In addition to receiving graphical data that can be pre-processed at a graphics originating system or post-processed at a receiving computing system, any of these embodiments of the computing device can also acquire content from the repository 1316.

[0115] The present disclosure relates to systems and methods for generating a sequence of keywords from an input query according to the examples provided in at least the following sections:

[0116] (A1) In one aspect, some embodiments include a method (e.g., 1000) for generating a sequence of keywords from an input query (e.g., 108); receiving a first sequence of text (e.g., 908) corresponding to the input query (e.g., 108), and encoding the first sequence of text into a source sequence representation using an encoder (e.g., 308) of a machine learning model (e.g., 302); generating a sequence of keywords (e.g., 128) from the source sequence representation using a decoder (e.g., 312) of the machine learning model (e.g., 302); generating a generation score (e.g., ) for a plurality of predicted word pieces (e.g., 418, 420, 422), where the plurality of predicted word pieces (e.g., 418, 420, 422) are based on the source sequence representation (e.g., 318) and a previously decoded portion hypothesis provided by the decoder (e.g., 312); generating a modified generation score (e.g., ) for the plurality of predicted word pieces (e.g., 418, 420, 422), where the modified generation score (e.g., ) is based on a respective predicted word piece generation score (e.g., ) and a maximum generation score of a suffix (e.g., 432, 434; 436, 438, 440) of each predicted word piece (e.g., 420; 418; 422) of the plurality of predicted word pieces (e.g., 418, 420, 422); selecting a predicted word piece (e.g., 420) of the plurality of predicted word pieces (e.g., 418, 420, 422) based on the modified generation score (e.g., ); adding text represented by the selected predicted word piece (e.g., 420) to the previously decoded portion hypothesis provided by the decoder (e.g., 312); and selecting content (e.g., 142) to be provided to a computing device (e.g., 104) in response to the input query (e.g., 108) based on the selected predicted word piece (e.g., 420), where the selected predicted word piece (e.g., 420) matches at least one keyword (e.g., 136) associated with the content (e.g., 142) to be provided to the computing device (e.g., 104).

[0117] (A2) In some embodiments of Al, the content (e.g., 142) to be provided to the computing device (e.g., 104) is an advertisement.

[0118] (A3) In some embodiments of Al-A2, the generation score (e.g., ) based on the generation score (e.g., 446) of the word piece (e.g., 432) that is a suffix of the one or more predicted word pieces (e.g., 420). .

[0119] (A4) In some embodiments of A1-A3, the machine-learned model (e.g., 302) is a sequence-to-sequence machine-learned model trained on data corresponding to pairs of keyword sequences and output sequences.

[0120] (A5) In some embodiments of A1-A4, the modified generation scores (e.g., 418, 420, 422) of the plurality of predicted word pieces (e.g., 418, 420, 422) are based on the respective predicted word piece generation scores (e.g., ) and a maximum generation score (e.g., ) of a suffix (e.g., 432, 434; 436, 438, 440) of each predicted word piece (e.g., 420; 418; 422) of the plurality of predicted word pieces (e.g., 420, 418, 422). .

[0121] (A6) In some embodiments of A1-A5, the plurality of predicted word pieces (e.g., 420, 418, 422) and the suffix (e.g., 432, 434; 436, 438, 440) of each predicted word piece (e.g., 420; 418; 422) are accessed from a Trie (e.g., Figure 2E ).

[0122] (A7) In some embodiments of A1-A6, the method further comprises receiving a keyword (e.g., 812) to be added to the Trie (e.g., Figure 2B ); and adding (e.g., 816) the keyword to the Trie (e.g., Figure 2B ), wherein the keyword (e.g., 240) is added as a suffix to an existing word piece (e.g., 212) in the Trie (e.g., Figure 2B ).

[0123] (A8) In some embodiments of A1-A7, the plurality of predicted word pieces (e.g., 420, 418, 422) and the suffix (e.g., 432, 434; 436, 438, 440) of each predicted word piece (e.g., 420; 418; 422) are constrained to the Trie (e.g., Figure 2E ).

[0124] (A9) In some embodiments of A1-A8, the method further comprises selecting content (e.g., 142) to be provided to the computing device (e.g., 104) in response to the input query (e.g., 108) based on the previously decoded partial hypothesis and the selected predicted word piece (e.g., 420), wherein the previously decoded partial hypothesis and the selected predicted word piece (e.g., 420) match at least one keyword (e.g., 236) associated with the content (e.g., 142) to be provided to the computing device (e.g., 104).

[0125] (A10) In some embodiments of A1-A9, the method further comprises generating a plurality of hypotheses (e.g., 420) based on the selected predicted word piece.

[0126] (A11) In some embodiments of A1-A10, the method further comprises identifying a Trie (e.g., 808) associated with the first text sequence.

[0127] (A12) In some embodiments of A1-A11, the method further comprises identifying a suffix (e.g., 432, 434) of the predicted word piece (e.g., 420, 418) from the Trie (e.g., 808) associated with the first text sequence. Figure 2E

[0128] (A13) In some embodiments of A1-A12, the method further comprises generating a modified generation score (e.g., 430, 438) for a suffix (e.g., 432, 434; 436, 438, 440) of each predicted word piece (e.g., 420; 418; 422) of the plurality of predicted word pieces (e.g., 420, 418, 422).

[0129] In yet another aspect, some embodiments include a computing system comprising one or more processors and a memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs comprising instructions for performing any of the methods described herein (e.g., A1-A13 described above).

[0130] In yet another aspect, some embodiments include a non-transitory computer- readable storage medium storing one or more programs for execution by one or more processors of a storage device, the one or more programs comprising instructions for performing any of the methods described herein (e.g., A1-A13 described above).

[0131] ​​Furthermore, aspects and functionalities described herein can operate over distributed systems (e.g., cloud-based computing systems), where various devices render user interfaces and functions, processing functions, and data storage and retrieval conforming to a central theme. Various types of user interfaces and information are presented to users through board computing device displays or remote display units associated with one or more computing devices. For instance, various types of user interfaces and information are presented to users through a wall surface and interacted with using a stylus, a finger, or other user input device. Interactions with the multiple computing systems that can practice embodiments of the application include keyboard input, touchscreen input, voice or other audio input, gesture inputs, where associated computing devices are equipped with detection (e.g., camera) functionality for capturing and interpreting user gestures for controlling the functionality of the computing device, and the like.

[0132] For example, various aspects of the disclosure are described with reference to block diagrams and / or operational illustrations of method, system, and computer program product according to aspects of the disclosure. The functions / acts noted in the blocks can occur out of the order as shown in any flowchart. For example, two blocks shown in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality / acts involved.

[0133] The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure as claimed in any way. Such aspects, examples, and specifics provided in this application are intended to be illustrative rather than restrictive. The scope of the claimed disclosure is not to be limited to any aspect, example, or detail provided in this application. Whether separately recited or described in conjunction with illustrative aspects, various features (structural and methodological) are intended to be selective for specific embodiments. The description and illustration of the application provided in this application are intended to be illustrative rather than restrictive. Changes, modifications, and alternative aspects that fall within the spirit and scope of the general inventive concept provided in this application can be made by those skilled in the art without departing from the scope of the claimed disclosure.< / start> < / start> < / start> < / start> < / start> < / start> < / start> < / bos> < / start> < / bos> < / start> < / bos> < / start> < / bos> < / start> < / bos> < / start> < / bos> < / start> < / eos> < / eos> < / eos>

Claims

1. A method for generating a keyword sequence from an input query, the method comprising: receiving a first sequence of text corresponding to an input query; encoding the first sequence of text into a source sequence representation using an encoder of a machine learning model executed by a server computer system, the machine learning model comprising the encoder and a decoder, the source sequence representation comprising a first set of tokens corresponding to the input query; generating a keyword sequence from the source sequence representation using the decoder of the machine learning model, the keyword sequence comprising a second set of tokens related to the input query; generating, by the decoder, a generation score for a first predicted token of a plurality of predicted tokens, wherein the plurality of predicted tokens are based on the source sequence representation and a previously decoded partial hypothesis provided by the decoder, wherein the previously decoded partial hypothesis comprises a first subset of tokens of the second set of tokens stored in a trie; generating, by the decoder, a modified generation score for the first predicted token, wherein the modified generation score for the first predicted token is based on the generation score and a maximum generation score of a suffix of the first predicted token, wherein the suffix comprises a second subset of tokens of the second set of tokens corresponding to the first predicted token stored within a path of the trie; selecting a second predicted token of the second subset of tokens based on the modified generation score; adding the second predicted token to the previously decoded partial hypothesis generated by the decoder; and selecting content to be provided to a computing device in response to the input query based on the second subset of tokens of the second set of tokens, wherein at least one token of the second subset of tokens of the second set of tokens matches at least one keyword associated with the content to be provided to the computing device.

2. The method of claim 1, wherein the content to be provided to the computing device is an advertisement.

3. The method of claim 1, wherein the generation score is based on a second generation score of a token that is a suffix of the suffix of the one or more predicted tokens.

4. The method of claim 1, wherein the machine learning model is a sequence-to-sequence machine learning model trained on data corresponding to keyword sequences and output sequences.

5. The method of claim 1, further comprising: receiving a keyword to be added to the trie; and adding the keyword to the trie, wherein the keyword is a suffix of an existing keyword in the trie.

6. The method of claim 1, wherein the plurality of predicted tokens and the suffix of each predicted token are constrained to the trie. selecting content to be provided to the computing device in response to the input query based on the previously decoded partial hypothesis and the second predicted token, wherein the previously decoded partial hypothesis and the second predicted token match at least one keyword field associated with the content to be provided to the computing device. ​ 7. The method of claim 1, further comprising: ​ 8. The method of claim 1, further comprising: generating a plurality of hypotheses based on the second predicted token.

9. The method of claim 1, further comprising: identifying the trie associated with the first text sequence.

10. The method of claim 9, further comprising: identifying a suffix of a token stored in the trie associated with the first text sequence.

11. The method of claim 1, further comprising: generating a set of modified generation scores for a set of suffixes of the plurality of predicted tokens.

12. A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1-11.

13. A system comprising a processor and a memory, the memory comprising instructions that, when executed by the processor, cause the processor to perform the method of any one of claims 1-11.

14. A method for generating a sequence of keywords from an input query, the method comprising: encoding a text sequence into a source sequence representation using an encoder of a machine learning model executed by a server computer system, the machine learning model comprising the encoder and a decoder, the source sequence representation comprising a first set of tokens corresponding to an input query; generating a sequence of keywords from the source sequence representation using the decoder of the machine learning model, the sequence of keywords comprising a second set of tokens related to the input query; generating, by the decoder, a generation score for a first predicted token of a plurality of predicted tokens, wherein the plurality of predicted tokens are based on the source sequence representation and a previously decoded partial hypothesis provided by the decoder, wherein the previously decoded partial hypothesis comprises a first subset of tokens of the second set of tokens stored in a trie; generating, by the decoder, a modified generation score for the first predicted token; selecting a second predicted token of the second set of tokens based on the modified generation score; adding the second predicted token to the previously decoded partial hypothesis provided by the decoder; and selecting content to be provided to a computing device based on the second predicted token, wherein the second predicted token matches at least one keyword associated with the content to be provided to the computing device.

15. The method of claim 14, wherein the modified generation score is based on the generation score and a maximum generation score of a suffix of the first predicted token.

16. The method of claim 15, wherein the plurality of predicted tokens and the suffix are accessed from a trie, and wherein the method further comprises: receiving a keyword to be added to the trie; and adding the keyword to the trie, wherein the keyword is a suffix of an existing keyword in the trie.

17. A method for generating a sequence of keywords from an input query, the method comprising: receiving a text sequence corresponding to an input query; encoding the text sequence into a source sequence representation using an encoder of a machine learning model, the machine learning model comprising an encoder and a decoder executed by a computer system, the source sequence representation comprising a first set of tokens corresponding to the input query; generating a sequence of keywords from the source sequence representation using the decoder of the machine learning model, the sequence of keywords comprising a second set of tokens related to the input query; generating, by the decoder, a generation score for a first predicted token of a plurality of predicted tokens, wherein the plurality of predicted tokens are based on the source sequence representation and a previously decoded partial hypothesis provided by the decoder of the machine learning model, wherein the previously decoded partial hypothesis includes a first subset of tokens of a second set of tokens stored in a trie; generating, by the decoder, a modified generation score for the first predicted token of the plurality of predicted tokens, wherein the modified generation score for the first predicted token of the plurality of predicted tokens is based on a respective predicted token generation score for the first predicted token and a maximum generation score for a suffix of the first predicted token of the plurality of predicted tokens, wherein the suffix includes a second subset of tokens of the second set of tokens stored within a path of a trie corresponding to the first predicted token; selecting a second predicted token of the second subset of tokens based on the modified generation score; adding the second predicted token to the previously decoded partial hypothesis provided by the decoder; and selecting content to be provided to a computing device in response to the input query based on the second predicted token.

18. The method of claim 17, wherein the modified generation score is based on the generation score and a maximum generation score for a suffix of the first predicted token.

19. The method of claim 17, wherein the suffix is obtained from a trie generated based on the input query.

20. The method of claim 17, further comprising: receiving a keyword to be added to the trie; and adding the keyword to the trie, wherein the keyword is a second suffix of a second keyword stored in the trie.

Images