building an answer to a query by using a deep model

Abstract

Various techniques related to constructing answers to queries in which the answer is in a list form are described herein. The answer includes a header and list elements. A deep model receives content of web pages that are deemed relevant to a query by a search engine and constructs an answer to the query from the web pages upon receiving the query.

Description

Background Technology

[0001] Computer-implemented search engines are configured to receive queries and identify web pages that the search engine considers to contain content relevant to the received query. In operation, a traditional search engine receives a query from a client computing device, searches its webpage index based on the query to identify multiple web pages relevant to the query, and then ranks these web pages based on the characteristics of the query and the characteristics of the multiple web pages. The search engine then constructs a Search Engine Results Page (SERP) and transmits the SERP to the client computing device from which it received the query. The SERP includes a ranked list of search results, where each search result represents a web page.

[0002] In recent times, search engines have been configured to directly provide answers to fact-based questions; for example, when a query for "George Washington's birthday" is received from a client computing device, the search engine can return the answer (February 22, 1732) as part of a SERP, which also includes a ranking list of search results. Therefore, the user submitting the query does not need to select a hyperlink on the SERP to obtain the answer.

[0003] Furthermore, for certain queries, traditional search engines have been configured to return answers in list format. For example, when a traditional search engine receives a query for "the most populous country in the world," it can return a list containing several list elements, where each list element identifies the most populous country among all countries in the world. Currently, this type of functionality is achieved by building lists offline, generating searchable list indexes for these lists, and storing the lists on computer-readable storage devices. Therefore, when a search engine receives a query from a client computing device, it searches the list index to identify one or more lists relevant to the query, calculates a score for each list (where the score indicates how relevant the list is to the query), compares the highest score to a predefined threshold, and if the highest score is higher than the predefined threshold, includes the list with the highest score in the SERP (Search Engine Resource Plan), and returns the SERP (with the included lists) to the client computing device.

[0004] Traditional methods for providing list answers have several problems. For example, building and approving list answers currently requires a significant amount of manual labor, thus limiting their applicability to a relatively small number of queries. Additionally, list answers can become outdated relatively quickly. For instance, the answer to the query "the player with the most home runs this year" might change frequently. However, using the traditional methods described above, search engines generate these lists offline, meaning they either cannot answer such queries or may provide outdated answers. Summary of the Invention

[0005] The following is a brief overview of the topics described in more detail herein. This overview is not intended to limit the scope of the claims.

[0006] This paper describes various techniques related to computing systems configured to, in response to a received query, construct an answer to the query at runtime from documents (e.g., web pages) searchable by the computing system. More specifically, and in an example, a search engine receives a query from a client computing device and identifies multiple web pages containing content closely related to the query. The search engine ranks these web pages to generate a ranking list and retrieves the top N (e.g., top five) web pages from the ranking list. The search engine can retrieve the top N web pages from a cache maintained by the search engine and / or from web servers that separately host the web pages.

[0007] To reduce the latency in constructing query answers, for each retrieved webpage, the search engine generates several webpage representations using multiple webpage partitioning rules. For example, a first webpage partitioning rule might require a first webpage representation to consist of all the webpage's headers and the first sentence following each header; a second rule might require a second webpage representation to consist of multiple lists (defined by list tags in the webpage's HTML code) and sentences immediately preceding and following the lists; a third rule might require a webpage representation to consist of multiple tables, multiple headings of those tables, and multiple sentences immediately following those tables. Therefore, the search engine can construct several distinct representations of a webpage, each containing less content than the entire webpage content.

[0008] In response to constructing a representation of a webpage, the search engine processes these representations in parallel to determine whether one or more representations of the webpage contain the answer to the query. For each webpage representation, the search engine delineates the boundaries of segments within that representation, where exemplary segments include sentences, phrases (such as headers that are independent but not complete sentences), headings, etc. For example, the search engine may leverage natural language processing (NLP) techniques on these webpage representations to delineate segment boundaries, and rule-based methods may be used to delineate segment boundaries (e.g., identifying paragraph breaks as annotation segment boundaries, identifying periods as segment boundaries, etc.).

[0009] The search engine includes a deep learning model, and fragments of webpage representations are provided to the deep learning model (e.g., sequentially). The deep learning model is configured to determine whether a webpage representation includes the answer to a query, and output that answer if the webpage representation does. In summary, the deep learning model is configured to scan each webpage representation of a webpage and determine whether that webpage representation includes the answer to the query. When a webpage representation includes the answer to the query (in list form), the deep learning model outputs a total score for that answer, indicating the confidence that the answer correctly answers the query. When an answer's score is below a predefined threshold, the search engine discards that answer. It can be determined that the deep learning model can output multiple answers to a query, each with a score equal to or higher than the predefined threshold. In this case, the search engine selects the answer with the highest score as the "best" answer and includes such answers in the SERP. When the deep learning model does not output an answer with a score equal to or higher than the predefined threshold, the search engine cannot include the answer in the SERP.

[0010] As previously indicated, the answers can be in the form of a list, which includes a header (i.e., a description of the list's contents) and several list elements. In an exemplary embodiment, the deep model constructs the list by selecting webpage representation fragments to include as adjacent elements in the list, where the fragments are not adjacent to each other on the webpage.

[0011] The search engine described in this paper offers various improvements over traditional search engines. In contrast to the traditional method of constructing answers offline in list form, the search engine described in this paper is configured to construct answers in list form at runtime in response to a received query, thereby ensuring that the answers are not outdated. Furthermore, the search engine is also configured to construct answers in list form, where the search engine can select adjacent elements from non-adjacent areas of the webpage.

[0012] The above overview presents a simplified summary to provide a basic understanding of some aspects of the systems and / or methods discussed herein. This overview is not a comprehensive overview of the systems and / or methods discussed herein. It is not intended to identify key / critical elements or define the scope of such systems and / or methods. Its sole purpose is to present some concepts in a simplified form as an introduction to the more detailed descriptions that follow. Attached Figure Description

[0013] Figure 1 This is a functional block diagram of an exemplary system configured to construct answers to queries based on the content of a webpage.

[0014] Figure 2 It is a diagram illustrating several different representations of dividing a webpage into webpages.

[0015] Figure 3This is a functional block diagram of an exemplary builder module that is configured to construct answers to queries based on the content of a webpage.

[0016] Figure 4 This is a functional block diagram of an exemplary sentence embedder system configured to perform sentence embedding based on encoded word sequences.

[0017] Figure 5 An exemplary search engine results page (SERP) is depicted.

[0018] Figure 6 The illustration shows how to extract a fragment from a table on a webpage.

[0019] Figure 7 The flowchart illustrates an exemplary method that supports the generation of SERPs that include answers to queries.

[0020] Figure 8 This is an exemplary computing system. Detailed Implementation

[0021] The various techniques relating to constructing answers to queries at runtime will now be described with reference to the accompanying drawings, wherein the same reference numerals are used throughout to refer to the same elements. In the following description, numerous specific details are set forth for illustrative purposes to provide a thorough understanding of one or more aspects. However, it will be apparent that these aspects(s) can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to facilitate the description of one or more aspects. Furthermore, it should be understood that functionality described as being performed by certain system components can be performed by multiple components. Similarly, for example, components can be configured to perform functionality described as being performed by multiple components.

[0022] Furthermore, the term "or" is intended to mean inclusive "or" rather than exclusive "or". That is, unless otherwise stated or clearly apparent from the context, the phrase "X adopts A or B" is intended to mean any naturally inclusive permutation. That is, the phrase "X adopts A or B" is satisfied in any of the following cases: X adopts A; X adopts B; or X adopts both A and B. Additionally, unless otherwise stated or clearly apparent from the context that it refers to the singular form, the articles "a" and "an" used in this application and the appended claims should generally be interpreted as meaning "one or more".

[0023] Furthermore, as used herein, the terms “component,” “system,” and “module” are intended to cover a computer-readable data storage device configured with computer-executable instructions that, when executed by a processor, cause certain functionalities to be performed. Computer-executable instructions may include routines, functions, etc. It should also be understood that a component, module, or system may reside on a single device or be distributed across several devices. Furthermore, as used herein, the term “exemplary” is intended to mean as an illustration or example of something and is not intended to indicate preference.

[0024] This article describes various techniques related to constructing answers to queries based on web page content, where answers are constructed at runtime by processing web page content identified as relevant to the query. The constructed answer can be in the form of a list or a table, where a list includes a header and one or more list elements, and a table includes at least one table row. Furthermore, the techniques described herein allow answers to be constructed in the form of a list or table, where the answer includes a first list or table element and a second list or table element, which are adjacent to each other in the answer but not adjacent to each other in the web page used to construct the answer. While the examples described herein involve constructing answers based on web page content, it should be understood that the techniques described herein are also well-suited for constructing answers based on the content of other types of documents such as emails, word processing documents, etc.

[0025] refer to Figure 1 The diagram illustrates a functional block diagram of an exemplary system 100 configured to construct answers to queries. System 100 includes a computing system 102 and a client computing device 104, wherein the computing system 102 communicates with the client computing device 104 via a network (e.g., the Internet). System 100 further includes a plurality of sources 106-108, wherein sources 106-108 may store web pages or other suitable documents. Thus, sources 106-108 may, for example, be web servers hosting web pages.

[0026] The computing system 102 includes a processor 110 and a memory 112 therein storing instructions executed by the processor 110. The computing system 102 also includes a data repository 114, wherein the data repository 114 may include an index 116, wherein the index 116 indexes web pages available via the World Wide Web (e.g., wherein the web pages are hosted by sources 106-108).

[0027] Memory 112 has a search engine 118 loaded therein, configured to perform a search on index 116 in response to a received query. Search engine 118 includes an answer generator system 120 configured to generate answers to queries received by search engine 118. As will be described in detail below, answer generator system 120 is configured to, at runtime and in response to a received query, process web pages and construct answers to the query based on the content of the web pages. This contrasts with conventional methods where answers are constructed offline and the search engine returns the answer to the query by performing a search on an index that indexes previously constructed answers.

[0028] The answer generator system 120 includes a document partitioner module 122, which is configured to receive a webpage and generate multiple non-equivalent representations of the webpage. For example, the document partitioner module 122 can generate a first representation of the webpage consisting of the headers in the webpage, and can also generate a second representation of the webpage consisting of a list included in the webpage (excluding the webpage headers).

[0029] The answer generator system 120 further includes a builder module 124, which is configured to construct answers to queries based on the content of webpage representations generated by the document segmenter module 122. As will be described in more detail below, the builder module 124 includes a computer-implemented deep model comprising multiple recurrent neural networks (RNNs), which is configured to process webpage representations and construct answers to queries based on the content of the webpage representations.

[0030] As previously described, builder module 124 can construct the answer in the form of a list or a table. A list may include a header and several list elements, where the header describes the overall content of the list, and the list elements are the elements of the list. A table may include at least one table row. Therefore, for example, when search engine 118 receives the query "most home runs in baseball this year," builder module 124 can construct an answer to the query, where the answer is in the form of a list or table. When the answer is in list form, the header of the list could be "2019 home runs," the first element of the list could be the name of the first baseball player and the number of home runs hit by that first baseball player, the second element of the list could be the name of the second baseball player and the number of home runs hit by that second baseball player, and so on. However, in the webpage from which builder module 124 constructs the answer, the names of the first and second baseball players may not be adjacent to each other. For example, biographical text may separate the names of the first and second players. However, builder module 124 is still able to construct an answer that includes the first two list elements mentioned above, even though the names of the first and second players are separated by text in the webpage from which the answer is constructed.

[0031] An exemplary operation of system 100 will now be described. A user 126 submits a query to client computing device 104, and client computing device 104 transmits the query (via a network) to computing system 102. Search engine 118 receives the query and searches index 116 for web pages closely related to the received query. Thus, search engine 118 can identify multiple web pages related to the query. Search engine 118 then calculates relevance scores for the multiple web pages based on the characteristics of the web pages and / or the characteristics of the received query. Search engine 118 ranks the web pages based on the relevance scores, where a higher relevance score indicates that the search engine considers the web page more relevant to the query.

[0032] The answer generator system 120 can select a threshold number of web pages with the highest relevance score assigned to it by the search engine 118. For example, the answer generator system 120 can select the highest-ranking web page (the web page assigned the highest relevance score). In another example, the answer generator system 120 can select the top five-ranking web pages (five web pages assigned the five highest relevance scores). The answer generator system 120 can then retrieve the selected web pages(s) from the appropriate sources 106-108. As previously indicated, sources 106-108 can be web servers hosting the web pages. Additionally or alternatively, the answer generator system 120 can retrieve one or more selected web pages from the cache of the search engine 118.

[0033] Each retrieved webpage is provided to document segmenter module 122. For each retrieved webpage, document segmenter module 122 generates several webpage representations using multiple webpage segmentation rules. For example, document segmenter module 122 can construct a first representation of the webpage consisting of the first sentence of a paragraph. Document segmenter module 122 can construct a second representation of the webpage consisting of the last sentence of a paragraph. Additionally, document segmenter module 122 can construct a third representation of the webpage, consisting of the webpage header and the sentence immediately following those headers. In yet another example, document segmenter module 122 can construct a fourth representation of the webpage, consisting of a list in the webpage (e.g., identified by list tags in the webpage's HTML code) and the sentence immediately following the list. The above examples should not be construed as limiting—it can be seen that document segmenter module 122 can construct several different representations for each webpage provided to document segmenter module 122. Furthermore, document segmenter module 122 can be optional in computing system 102, as webpage representations are constructed to reduce latency associated with generating answers from webpage content. More specifically, as will be described in more detail below, builder module 124 is configured to process document content provided to the builder module, and may require a relatively large number of processing cycles (and thus time) to process web pages containing a large amount of content. Web page representations generated by document partitioner module 122 have less content than a full web page and can be processed in parallel by builder module 124.

[0034] The builder module 124 receives each webpage representation generated by the document segmenter module 122 and determines whether the webpage representation includes the answer to the query. More specifically, and in this example, the builder module 124 processes the content of the webpage representation that includes the answer to the query, where the answer is in list form. The answer generator system 120 calculates a confidence score for the answer, where the confidence score indicates the confidence that the answer is a suitable answer to the query. The builder module 124 can compare the confidence score to a predefined threshold, and when the confidence score is below the threshold, it can indicate that the webpage representation does not include the answer to the query. When the confidence score of an answer reaches or exceeds the predefined threshold, the answer generator system 120 compares the confidence score with other confidence scores assigned by the builder module 124 to other answers, where other answers are generated based on other webpage representations and assigned confidence scores that meet or exceed the predefined threshold. The answer generator system 120 selects the answer assigned the highest confidence score and causes the search engine 118 to include that answer in the search engine results page (SERP) returned to the client computing device 104. When no answer has a confidence level that reaches or exceeds a threshold, the answer generator system 120 can indicate that no answer is to be returned as part of the search engine results page (SERP) to be provided to the client computing device 104. As described in more detail herein, the builder module 124 constructs the answer for the webpage representation by first word encoding the words in the fragments of the webpage representation, then performing sentence embedding, and then performing semantic analysis on the embedded words with respect to the received query.

[0035] As previously described, while System 100 has been described as including a search engine configured to retrieve web pages, the aspects described herein are also well-suited for constructing answers from the content of other types of documents. For example, Answer Generator System 120 could be used in an email search system to construct answers to queries, where the answers are constructed based on the content of email documents. In another example, Answer Generator System 120 could be employed in a local search system configured to construct answers to queries, where the answers are constructed based on the content of documents stored in a file system (e.g., word processing documents, spreadsheets, etc.).

[0036] Now for reference Figure 2 A schematic diagram is shown illustrating the operation of the document partitioner module 122. The document partitioner module 122 receives a webpage 202. The webpage 202 includes a header and X headers, each of which contains multiple sentences. The document partitioner module 122 constructs multiple representations 204-208 of the webpage 202 based on document partitioning rules. Figure 2As illustrated, the first webpage representation 204 includes the title of webpage 202, each of the X headers of webpage 202, and a sentence immediately following each header. The second webpage representation 206 differs from the first webpage representation 204. The second webpage representation 206 includes the title of webpage 202, each header of webpage 202, and the last sentence following each header of webpage 202. The third webpage representation 208 differs from both the first and second webpage representations 204 and 206. The third webpage representation 208 includes the title of webpage 202, and the first and last sentences following each of the X headers in webpage 202.

[0037] Web page representations 204-208 are presented for illustrative purposes and are not intended to be limiting; other web page representations are also envisioned. For example, document partitioning module 122 can scan web page 202 from top to bottom and place the scanned content into web page representations until a certain number of characters are reached, at which point document partitioning module 122 can create a new representation of web page 202. Additionally, document partitioning module 122 can be configured to generate document representations so that lists or tables are not split into different document representations. For example, a list in web page 202 can be generated using the list tag in the HTML of web page 202 (…). … … The document partitioner module 122 can generate web page representations that include the entire list. Each of the web page representations 204-208 can be provided to a corresponding instance of the builder module 124 so that the web page representations 204-208 can be processed in parallel.

[0038] Now for reference Figure 3 The diagram illustrates a functional block diagram of the builder module 124. The builder module 124 is configured to receive a web page representation 302 from the client computing device 104 and a query received by the computing system 102. The web page representation 302 comprises multiple fragments (fragment 1-fragment Z). A fragment can be a sentence, header text (which may or may not be a complete sentence), title text, a row of a table, or other suitable fragments. Typically, a fragment includes multiple words, although a fragment may include a single word.

[0039] Builder module 124 includes query word encoder module 304, which is configured to receive a query and encode each word in the query into a fixed-size vector, where each vector represents the corresponding word in the query. Any suitable technique can be used when encoding words into fixed-size vectors. An exemplary technique is described in “Glove: Global Vectors for Word Representation”, Pennington et al., 2014, Empirical Approaches in Natural Language Processing (EMNLP), pp. 1532-1543. Builder module 124 also includes multiple word encoder modules 306-308, where a first word encoder module 306 receives a first fragment of webpage representation 302, and a Z-th word encoder module 308 receives a Z-th fragment of webpage representation 302. Word encoder modules 306-308 are configured to encode words in the fragments of webpage representation 302 in parallel, with each word in each fragment encoded into a fixed-size vector. Therefore, for example, when the first segment includes five words, the first word encoder module 306 receives these five words sequentially and encodes each of the five words into five fixed-size vectors. Thus, the output of the first word encoder module 306 is a sequence of five fixed-size vectors.

[0040] The builder module 124 also includes multiple sentence embedding systems 310-312, which are configured to perform sentence embedding with respect to the vector sequences output by the word encoder modules 306-308, respectively. Continuing the example described above, when the first segment of the webpage representation 302 includes five words, the word encoder module 306 outputs a sequence of five vectors, and the first sentence embedding system 310 receives these five vectors sequentially. The first sentence embedding system 310 outputs fixed-size vectors, which are the encodings of the first segment of the webpage representation 302, as will be described in more detail below.

[0041] The builder module 124 also includes a query RNN 314, which receives a fixed-size sequence of vectors output by the query word encoder module 304. The output of the query RNN 314 is another sequence of fixed-size vectors, where each fixed-size vector represents a word in the query context (e.g., the meaning of a word typically depends on what other words appear in the segment alongside it). Sentence embedding systems 310-312 receive the output of the query RNN 314 and perform sentence embedding based on the output of the query RNN 314. For example, sentence embedding systems 310-312 may include an RNN with an attention mechanism, where attention is provided to the encoding of the output of the query RNN 314.

[0042] As noted above, the first sentence embedder system 310 outputs a first fixed-length vector, which is the encoding of a first segment of the webpage representation 302. Additionally, the Zth sentence embedder system 312 outputs a Zth fixed-length vector, which is the encoding of the Zth segment of the webpage representation. The builder module 124 optionally includes a segment RNN 316, which sequentially receives the fixed-length vectors output by the sentence embedder systems 310-312. The segment RNN 316 is configured to output a sequence of fixed-length vectors encoding segments of the webpage representation 302, within the context of the segments in the webpage representation. Similar to a word having a meaning that depends on what other words appear in the segment along with it, the meaning of a segment depends on other segments that appear in the webpage representation along with it.

[0043] Builder module 124 may also optionally include a query-focused segment RNN 318, which receives a fixed-length vector sequence output by segment RNN 316 and outputs another fixed-length vector sequence, both of which encode the web page representation segment. Alternatively, the query-focused segment RNN 318 may directly receive the output of sentence embedder systems 310-312. The query-focused segment RNN 318 has a focus mechanism and focuses on the encoding output by query RNN 314. The query-focused segment RNN 318 outputs yet another fixed-length vector sequence, where this fixed-length vector is the corresponding encoding of the segment of web page representation 302. Builder module 124 may also optionally include a self-focused segment RNN 320, which receives the vector sequence output by the query-focused segment RNN 318 and outputs yet another vector sequence with additional encodings of the segment of web page representation 302. Therefore, the self-focused segment RNN 320 has a focus mechanism that focuses on itself.

[0044] Although builder module 124 is illustrated to include fragment RNN 316, query-focused fragment RNN 318, and self-focused fragment RNN 320, it should be understood that builder module 124 may include only one of these RNNs or a combination of these RNNs, and the arrangement of the RNNs may be modified. For example, builder module 124 may not include fragment RNN 316, and self-focused fragment RNN 320 may receive the output of sentence embedder systems 310-312 and feed it into query-focused fragment RNN 318.

[0045] The builder module 124 also includes a query connector 322, which performs maximum mean pooling on the vector sequence output by the query RNN 314, such that the query connector 322 outputs a vector of length that matches the length of the vectors output by the segment RNNs 316-320. The vector output by the query connector 322 is an encoding of the query received from the client computing device 104.

[0046] The builder module 124 also includes multiple elements RNN 324-328, where elements RNN 324-328 correspond to portions of the potential answers in list form. For example, the first element RNN 324 may correspond to the header of the list, the second element RNN 326 may correspond to the first list element, and the second element RNN 328 may correspond to the second list element. Similarly, the first element RNN 324 may correspond to the first row of the answer table, the second element RNN 326 may correspond to the second row, and the third element RNN 328 may correspond to the third row. It should be understood that the builder module 124 may include more than Figure 3 The diagram illustrates a three-element RNN 324-328 with more elements. The first element RNN 324 receives a sequence of vectors output by segment RNN 316, either a query-focused segment RNN 318 or a self-focused segment RNN 320. The first element RNN 324 also has a focus mechanism, allowing it to focus on vectors output by the query connector 322. The first element RNN 324 outputs scores corresponding to vectors in the sequence of vectors received from one of the RNNs 316-318, where a score indicates the probability that the segment corresponding to that score will be included as the header of the answer (or as the first row of the answer table). The second element RNN 326 operates similarly to the first element RNN 324, where it receives vectors output from one of the RNNs 316-320 and outputs scores indicating the probability that the segment represented by the vector will become the first list element in the answer (and be placed immediately below the header) or the second row of the answer table. The focus weights are shared among the first element RNN 324, the second element RNN 326, and the third element RNN 328. The operation of the third element RNN 328 is similar to that of the first element RNN 324 and the second element RNN 326, but the output score indicates the probability that the segment represented by the vector will become the second list item or the third row in the answer.

[0047] Therefore, the first element RNN 324 can output a first probability distribution on the Z sentence fragments representing webpage 302, the second element RNN 326 can output a second probability distribution on the Z sentence fragments representing webpage 302, and the third element RNN 328 can output a third probability distribution on the Z sentence fragments representing webpage 302. The builder module 124 identifies candidate answers based on the probability distributions.

[0048] The builder module 124 also includes a validator module 330, which sequentially receives the outputs (probability distributions) of elements RNN 324-328 and generates adjusted probabilities, where the adjusted probabilities account for the probability that no fragment is suitable as the answer part corresponding to elements RNN 324-328. More specifically, the validator module 330 receives the output of the first element RNN 324 and also receives the encoded query output by the query connector 322. The validator module 330 calculates the adjusted probabilities, where the adjusted probabilities include the probability that such a fragment is the header of a list answer for each of the Z fragments, and where the adjusted probabilities also include the "no answer" probability. The "no answer" probability that none of the Z fragments is the header of a list answer is true. When the "no answer" probability calculated by the validator module 330 is higher than a predefined threshold, the builder module 124 outputs webpage representation 302 that does not contain a suitable answer. Otherwise, the validator module 330 selects the fragment with the highest probability and receives the output of the second element RNN 326 and the encoded query output by the query connector 322. Thus, the builder module 124 effectively determines, if any, how many list elements (or table rows) are included in the answer based on the webpage representation 302.

[0049] As previously indicated, the answer generator system 120 may include multiple instances of a builder module 124, wherein the instances of the builder module 124 process the webpage representation in parallel. Therefore, the multiple instances of the builder module 124 can output multiple candidate answers. Based on the probabilities output by the validator module 330, the answer generator system 120 selects the final answer to be included in the SERP generated by the search engine 118. The search engine 118 includes the answer in the SERP and transmits the SERP to the client computing device 104. When there are no candidate answers (determined by the validator module 330), the answer generator system 120 outputs an indication that no answer will be included in the SERP, and the search engine 118 transmits the SERP (with no answer) to the client computing device 104.

[0050] In summary, each of the word encoder modules 306-308 is then configured to receive words from a fragment (e.g., a sentence) of the webpage representation 302 and generate a sequence of encoded words (a fixed-length vector sequence). Each of the sentence embedder systems 310-312 is configured to receive the sequence of encoded words and generate encoded fragments (with attention to words in the query). RNNs 316-320 are configured to further encode fragments, where the further encoded fragments represent fragments within the context of the webpage representation 302. Element RNNs 324-328 are used to compare these further encoded fragments with the query to determine the probability distribution on the fragments of the webpage representation 302 regarding which fragment would serve as the answer corresponding to the part of each element RNN. Finally, the validator module 330 calculates an adjusted probability, taking into account that "no answer" is likely the best outcome.

[0051] Now for reference Figure 4 The illustration shows a functional block diagram of an exemplary sentence embedding system 400, wherein the sentence embedding system 400 may be... Figure 3 Any of the sentence embedder systems 310-312 illustrated herein. System embedder system 400 includes at least one of word RNN 402 or query-focused word RNN 404. In an exemplary embodiment, word RNN 402 receives an encoded word sequence output by a word encoder module (e.g., one of word encoder modules 306-308), wherein the encoded word sequence is encoded word 1-B. Word RNN 402 outputs a fixed-length vector sequence representing encoded word 1-B, wherein each vector represents a word in a segment whose context is that the word appears in the vector. Query-focused word RNN 404 receives such a vector sequence and outputs another fixed-length vector sequence, wherein query-focused word RNN 404 may include a focus mechanism that focuses on the output of query RNN 314. Thus, the other sequence vector is a further encoding of the words in the segment, focusing on the query words encoded by query RNN 314.

[0052] As noted above, the sentence embedder system 400 may include one of a word RNN 402 or a query-interested word RNN 404, or both. Furthermore, when the sentence embedder system 400 includes two RNNs, the query-interested word RNN 404 may receive the output of the word RNN 402, or the word RNN 402 may receive the output of the query-interested word RNN 404. The sentence embedder system 310 also includes a connector module 406 that receives a sequence of vectors output from either the word RNN 402 or the query-interested word RNN 404 and forms a fixed-length vector that includes the encoding of segments of words encoded by I-B encoding. For example, the connector module 406 may use maximum mean pooling on the received vector sequence to construct the fixed-length vector, which is then output to one of the RNNs 316-320. The RNN of builder module 124 can be or include any suitable type of RNN, including Long Short-Term Memory (LSTM) RNN, Gated Recurrent Unit (GRN), etc.

[0053] The exemplary training of builder module 124 is now briefly described. Initially, an example set is obtained, where each example includes a query, a webpage, and indications of whether each fragment of the document is an appropriate header, first list element, second list element, etc. To obtain this example set, queries are extracted from the query logs of search engine 318 and provided to search engine 318. For one or more queries, search engine 318 can output answers in list form using the conventional methods described above (where search engine 318 searches the answer index and returns offline generated answers). For each query, when search engine 318 returns an answer after receiving a query, the highest-ranking webpages by search engine 318 based on a threshold number of query identifiers are retrieved. List answers are mapped back to each retrieved webpage, and labels are assigned to each fragment of the retrieved webpage depending on whether the fragment is included as part of the answer (e.g., as a header or a list element in the answer). For example, labels can be assigned to fragments indicating that the sentence is a header of the answer, a first list element of the answer, unknown, or not part of the answer. Additionally, manually labeled training data may also be obtained.

[0054] The objective function used to train Builder Module 124 is now described. Builder Module 124 calculates the probability of each list element in the list answer. For illustrative purposes, the header of the list answer is illustrated as an example. Whether a fragment in a webpage is suitable as a header can be viewed as a classification problem. If we assume there are N fragments in the webpage representation, then there are a total of N+1 labels for the header of the answer (one label for each sentence fragment, plus a "no answer" label): {0, 1, ..., N-1, ∞}. If the label is i, 0 ≤ i < N, this label indicates that the header is the i-th fragment of the webpage representation. If the label is ∞, this label indicates that none of the N fragments in the webpage representation is suitable as a header. If we assume that only one sentence can be used as the header of the answer, then these N+1 labels are exclusive and exhausted. For example, i can be labeled with this label, and f(i) can be labeled with the log odds generated by Builder Module 124. In this case, the objective function has three cases.

[0055] In the first case, the known header is mapped back to the webpage representation. In this case, multiple segments can be matched, but it's unknown which segment the search engine 318 actually selected. Therefore, the target of this header can be written as:

[0056]

[0057] Where L contains the indexes of all segments that match the returned header.

[0058] In the second case, no answer is returned for the webpage representation (i.e., there is no header segment in the webpage representation that serves as the answer). The tag that can be used for the header is the equation ∞. The purpose of this case is as follows:

[0059]

[0060] In the third case, the human tagger will assign fragment i b The segment is marked as unsuitable for use as a header. In this case, it is known that the marked segment is not a suitable answer header, but it is unknown whether other segments in the webpage representation can serve as answer headers. Therefore, in the third case, besides i... b All other tags are missing, and the goal is:

[0061]

[0062] The fragment is excluding fragment i b The probability of headers other than those in the training example increased. A learning rate of 0.0005 and a dropout rate of 8.9 were used in the training example. Variations in the dropout rate were used for the attention mechanism.

[0063] Now for reference Figure 5The illustration shows an exemplary SERP 500 including answers generated by the answer generator system 120. The SERP includes a query field 502 that describes a query submitted by a user 126 of a client computing device 104 and from which new queries can be received. The SERP 500 includes a search results field 504 that includes a list of web pages identified by a search engine 118 as relevant to the query described in the query field 502. Optionally, the SERP 500 includes images 506 identified by the search engine 118 as relevant to the query. The SERP 500 may also include a query suggestion list 508. Finally, the SERP 500 may include an answer field 510 that describes the answer generated by the answer generator system 120 as described above. An exemplary answer field 510 includes a list header, a first list element, and a second list element. For example, when the query field 502 is “most home runs in 2019”, the header is the first fragment extracted from the web page representation (e.g., “2019 home runs”), the first list element is the second fragment extracted from the web page representation (e.g., “John Smith 34 home runs”), and the second list element is the second fragment extracted from the web page representation (e.g., “John Doe 32 home runs”).

[0064] refer to Figure 6 A diagram illustrating fragments from a table on a webpage is presented. In this example, webpage representation 600 includes a table with five rows and six columns. The answer generator system 120 can treat each row of the table as an independent fragment. Thus, the sequence of words "athlete," "team," "position," "game," "home run," and "average" in the first row of webpage representation 600 is considered the first fragment, the words / numbers "athlete A," "team 1," "1B," "45," "5," and ".289" are considered the second fragment, and so on. Regarding the exemplary query "most home runs in 2019," the builder module 124 in this example can output the first row of the table as the first row of the table answer, the fourth row as the second row of the table answer, and the third row as the third row of the table answer. Therefore, without leaving the SERP, the user issuing the query can easily find out which athletes C and B had the most home runs in 2019.

[0065] Figure 7The illustration depicts an exemplary method related to constructing answers based on web page content. While the method is shown and described as a sequence of actions performed sequentially, it should be understood and appreciated that the method is not limited to this sequence. For example, some actions may occur in a different order than described herein. Additionally, one action may occur simultaneously with another. Furthermore, in some cases, not all actions may be required to implement the method described herein.

[0066] Furthermore, the actions described herein can be computer-executable instructions, which can be implemented by one or more processors and / or stored on or within a computer-readable medium. Computer-executable instructions can include routines, subroutines, programs, threads of execution, etc. Moreover, the results of the method actions can be stored in a computer-readable medium, displayed on a display device, and so on.

[0067] Figure 7 An exemplary method 700 is illustrated, which is used to generate a SERP that includes answers in list form. Method 700 begins at 702 and receives a query from a client computing device at 704. At 706, the search engine identifies web pages based on the query, for example, where the web page could be the web page that the search engine has assigned the highest ranking to in relation to the received query.

[0068] At 708, the webpage identified at 706 is retrieved. For example, the webpage can be retrieved from a web server hosting the webpage or from a search engine cache. At 710, a representation of the webpage is generated, which is based on at least one webpage segmentation rule. As mentioned earlier, the representation of the webpage may include less than the entire webpage's content. At 712, fragment boundaries in the webpage representation are identified. For example, fragment boundaries are identified based on table rows, punctuation marks, paragraph breaks, line breaks, tags in the webpage's HTML code, section breaks, etc.

[0069] At 714, word encoding is performed on each word in each segment of the webpage representation. Thus, at 714, each segment is represented by a sequence of encoded words. At 716, sentence embedding is performed based on the words encoded at 714. In other words, at 716, each encoded word is further encoded based on the context in which the word appears in the segment, and these further encoded words are concatenated to form a fixed-length vector representing the semantics of the segment (when considered individually). At 718, semantic analysis of the segment is performed based on the sentence embedding performed at 716. In other words, the encoded segment is further encoded to take into account the context in which the segment appears in the webpage representation. These further encoded segments are compared with the received query to determine whether each encoded segment is suitable as the header of the answer, the first list element of the answer, the second list element of the answer, and so on.

[0070] At 720, it is determined whether the representation of the webpage includes a suitable answer to the query. As previously described, the builder module 124 can dynamically determine the "length" (e.g., the number of list elements or table rows) included in the answer. More specifically, for the header portion of the answer, the builder module 124 can determine the probability that there is no suitable segment in the webpage representation as a header portion of the answer. When this probability is higher than a threshold, the builder module 124 can output an indication that the webpage representation does not include the answer. When the probability is lower than the threshold, for the first list element of the answer, the builder module 124 can determine the probability that there is no suitable segment in the webpage representation as the first list element of the answer. If the representation of the webpage does not include a suitable answer, a SERP without the answer is generated at 722, and the SERP is transmitted to the client computing device from which the query is received. If it is determined at 720 that the webpage representation includes a suitable answer, a SERP including such an answer is generated at 724, and the SERP is transmitted to the client computing device. Method 700 is completed at 726.

[0071] It should be noted again that while the examples described in this article concern search engines that identify web pages relevant to queries, the techniques described herein are not limited to this type of web-based search engine. The techniques described herein are equally well-suited for building answers to queries performed on other types of documents, such as files stored in a file system, email documents, social media messages, etc. Furthermore, while the examples illustrated herein have indicated displaying the answer on a display, other methods for presenting answers to users can be envisioned. For example, a user could make a voice query to a smart speaker, and the smart speaker could return an answer to the user via voice, where the answer is constructed by builder module 124.

[0072] Now for reference Figure 8 The illustration shows a high-level diagram of an exemplary computing device 800 that can be used according to the systems and methods disclosed herein. For example, the computing device 800 can be used in a system for constructing query answers based on document content, where the answers are in list form. As another example, the computing device 800 can be used in a system supporting the training of a deep model, using the deep model in conjunction with the query answers constructed in list form. The computing device 800 includes at least one processor 802 that executes instructions stored in memory 804. For example, these instructions may be instructions for implementing functionality described as being performed by one or more of the components described above, or instructions for implementing one or more of the methods described above. The processor 802 can access memory 804 via system bus 806. In addition to storing executable instructions, memory 804 may also store web pages, representations of web pages, encodings, etc.

[0073] The computing device 800 also includes a data repository 808, which is accessible by the processor 802 via the system bus 806. The data repository 808 may include executable instructions, web pages, emails, web page representations, etc. The computing device 800 also includes an input interface 810 that allows external devices to communicate with it. For example, the input interface 810 can be used to receive instructions from external computer devices, users, etc. The computing device 800 also includes an output interface 812 that connects the computing device 800 to one or more external devices. For example, the computing device 800 can display text, images, etc., through the output interface 812.

[0074] It is conceivable that external devices communicating with computing device 800 via input interface 810 and output interface 812 can be included in an environment that provides virtually any type of user interface to which the user can interact. Examples of user interface types include graphical user interfaces, natural user interfaces, and the like. For example, a graphical user interface can accept input from a user employing multiple input devices such as a keyboard, mouse, remote control, etc., and provide output on an output device such as a display. Furthermore, a natural user interface allows the user to interact with computing device 800 in a manner unconstrained by input devices such as keyboards, mice, remote controls, etc. Instead, a natural user interface can rely on speech recognition, touch and stylus recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, machine intelligence, and so on.

[0075] Furthermore, although illustrated as a single system, it should be understood that computing device 800 can be a distributed system. Therefore, for example, several devices can communicate via a network connection and collaboratively perform tasks described as being performed by computing device 800.

[0076] The various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, these functions can be stored on or transmitted via a computer-readable medium as one or more instructions or code. Computer-readable media include computer-readable storage media. A computer-readable storage medium can be any available storage medium accessible to a computer. By way of example and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and is accessible to a computer. Disks and optical discs as used herein include compact optical discs (CDs), laser discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs (BDs), where disks typically copy data magnetically, while optical discs typically copy data optically using lasers. Furthermore, transmitted signals are not included within the scope of computer-readable storage media. Computer-readable media also include communication media, including any medium that facilitates the transfer of computer programs from one place to another. For example, a connection can be a communication medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, optical fiber, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, optical fiber, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are all included in the definition of communication media. Combinations of the above media should also be included within the scope of computer-readable media.

[0077] Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, illustrative types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0078] The above description includes examples of one or more embodiments. Of course, for the purposes of describing the foregoing aspects, it is impossible to describe every conceivable modification and variation of the described apparatus or method; however, those skilled in the art will recognize that many further modifications and arrangements of the aspects are possible. Therefore, the described aspects are intended to encompass all such changes, modifications, and variations falling within the spirit and scope of the appended claims. Furthermore, with regard to the use of the term "comprising" in the detailed description or claims, the term is intended to be inclusive, in a manner similar to how the term "comprising" is interpreted as a transitional word in the claims.

Claims

1. A method executed by said processor when at least one processor executes a search engine, the method comprising: Web pages are identified as relevant to a query based on a query received from a client computing device, which communicates with the at least one processor network. In response to identifying the webpage, retrieve the webpage; Generate a representation of the webpage that includes multiple elements of the webpage; The representation of the webpage is provided to a computer-implemented deep model, which is configured to: construct an answer to a query based on the plurality of elements of the representation by determining a first score indicating the probability that the element is included in a first list entry and a second score indicating the probability that the element is included in a second list entry for each element. Construct answers based on the representation of the webpage, wherein the answers are in list form and include: The first element, at the first list entry, is selected from the plurality of elements represented based on its first score, and the first element includes the first text of the webpage; as well as The second element, at the second list entry, is selected from the plurality of elements represented based on its second score. The second element includes the second text of the webpage, wherein the first and second texts are adjacent in the answer but separated by a third text on the webpage; and The answer is transmitted to the client computing device, wherein the client computing device is configured to present the answer to the user of the client computing device through an output interface associated with the client computing device.

2. The method of claim 1, wherein the webpage is identified by the search engine as the highest-ranking webpage among a plurality of webpages related to the query, and further wherein the webpage is retrieved because the webpage is from the highest-ranking webpage among the plurality of webpages.

3. The method according to claim 1, further comprising: In response to retrieving the webpage and before constructing the answer, a second representation of the webpage is generated, wherein the representation of the webpage is different from the second representation of the webpage; as well as The second representation of the webpage is provided to a computer-implemented second depth model, which is configured to construct a second answer to a query based on the text of the second representation of the webpage provided to the computer-implemented second depth model, wherein the second representation includes multiple elements of the webpage that are different from the multiple elements of the representation.

4. The method of claim 3, wherein generating the representation of the webpage and the second representation comprises: One or more boundaries are defined between multiple segments in the webpage, wherein a first segment of the multiple segments is included in the representation, and a second segment of the multiple segments is included in the second representation.

5. The method of claim 3, wherein generating the second representation of the webpage comprises: The list tag is identified in the HTML code of the webpage, wherein the second representation is generated based on the list tag identified in the HTML code.

6. The method of claim 3, wherein the computer-implemented second depth model constructs the second answer based on the second representation of the webpage, wherein the second answer is different from the answer, the method further comprising: Calculate the first confidence score of the answer; Calculate the second confidence score for the second answer; as well as The answer or the answer with higher confidence among the first and second answers is selected by comparing the first confidence score and the second confidence score.

7. The method of claim 3, wherein the first text has a first plurality of words, and the second text has a second plurality of words, and further wherein constructing the answer from the representation of the webpage comprises: The first text is encoded into a first vector, wherein the first vector represents the semantic meaning of the first element; as well as The second text is encoded into a second vector, wherein the second vector represents the semantic meaning of the second element, wherein the first vector and the second vector have the same length, and further wherein the answer is constructed based on the first vector and the second vector.

8. The method of claim 7, wherein constructing the answer from the representation of the webpage further comprises: The first vector and the second vector are provided as sequential inputs to the recurrent neural network (RNN); as well as The RNN generates a sequential output including a third vector and a fourth vector, wherein the third vector represents the first element and the fourth vector represents the second element, and further wherein the answer is constructed based on the third vector and the fourth vector.

9. The method of claim 8, wherein constructing the answer from the representation of the webpage further comprises: The third and fourth vectors are provided as sequential inputs to a second RNN, wherein the second RNN focuses on the encoding of the query; as well as The second RNN generates a sequential output including a fifth vector and a sixth vector, wherein the fifth vector represents the first element and the sixth vector represents the second element, and further wherein the answer is constructed based on the fifth vector and the sixth vector.

10. The method of claim 9, wherein constructing the answer from the representation of the webpage further comprises: The fifth and sixth vectors are provided as sequential inputs to the third RNN, wherein self-attention is employed in the third RNN; as well as The third RNN generates sequential outputs including a seventh vector and an eighth vector, wherein the seventh vector represents the first element and the eighth vector represents the second element, and further wherein the answer is constructed based on the seventh vector and the eighth vector.

11. A computing system configured to construct an answer to a query from the text of a webpage, wherein the computing system receives the query from a client computing device, the computing system comprising: At least one processor; as well as A memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform actions, the actions including: Receive a list of web page rankings from a search engine, wherein the search engine generates the list of web page rankings in response to receiving the query; The webpage is selected from the webpage ranking list based on the webpage ranking list being the highest-ranking webpage within a predefined threshold number of webpages, wherein the webpage includes text. In response to selecting the webpage from the webpage ranking list, the webpage is retrieved from the web server hosting the webpage; Generate a representation of the webpage that includes multiple elements of the webpage; The representation of the webpage is provided to a computer-implemented deep model, which is configured to: construct an answer to a query based on the plurality of elements of the representation by determining a first score indicating the probability that the element is included in a first list entry and a second score indicating the probability that the element is included in a second list entry for each element. The answer to the query is constructed based on the representation of the webpage, wherein the answer comprises a list, the list comprising a first element, the first element at a first list entry being selected from the plurality of elements of the representation based on its first score, the first element comprising first text of the webpage, and the list further comprising a second element, the second element at a second list entry being selected from the plurality of elements of the representation based on its second score, the second element comprising second text of the webpage, and wherein the second element is adjacent to the first element in the list, and the first element and the second element are separated by intermediate text in the webpage; and Generate a search engine results page, wherein the search engine results page includes the list of web page rankings, and further wherein the search engine results page includes the answer to the query.

12. The computing system of claim 11, wherein the webpage comprises a table, wherein the table comprises the first list entry and the second list entry.

13. The computing system of claim 11, wherein there are five web pages with the highest ranking among the predefined threshold numbers.

14. The computing system of claim 11, wherein the representation of the webpage is generated based on a first partitioning rule, and the action further includes: A second representation of the webpage is generated based on a second partitioning rule, wherein the representation of the webpage is different from the second representation of the webpage. The second representation of the webpage is provided to a computer-implemented second depth model, which is configured to construct a second answer to a query based on the text of the second representation of the webpage provided to the computer-implemented second depth model, wherein the second representation includes multiple elements of the webpage that are different from the multiple elements of the representation.

15. The computing system of claim 14, wherein generating the second representation of the webpage comprises: The list tag is identified in the HTML code of the webpage, wherein the second representation is generated based on the list tag identified in the HTML code.

16. The computing system of claim 14, wherein, in parallel with constructing the answer to the query, the computer-implemented second deep model constructs the second answer to the query based on the second representation of the webpage, wherein the second answer differs from the answer, the action further comprising: Calculate the first confidence score of the answer; Calculate the second confidence score for the second answer; as well as The answer, or the answer with higher confidence among the first and second answers, is selected to be included in the search engine results page by comparing the first confidence score and the second confidence score.

17. The computing system of claim 11, wherein the answer further includes a header, wherein the first list entry and the second list entry in the answer are following the header.

18. The computing system of claim 11, wherein constructing the answer to the query comprises: Perform sentence embedding on elements of the webpage, wherein performing sentence embedding on elements includes constructing a vector that encodes the elements.

19. The computing system of claim 18, wherein constructing the answer to the query further comprises: A sequence of vectors encoding the elements of the webpage is provided to a recurrent neural network (RNN), wherein the answer is constructed based on the output of the RNN.

20. A computer-readable storage medium comprising instructions that, when executed by a processor of a computing system, cause the processor to perform an action, the action comprising: Based on a query received from a client computing device communicating with the processor network, a webpage is identified as relevant to the query; In response to identifying the webpage as relevant to the query, the webpage is retrieved; Generate a representation of the webpage that includes multiple elements of the webpage; The representation of the webpage is provided to a computer-implemented deep model, which is configured to: construct an answer to a query based on the plurality of elements of the representation by determining a first score indicating the probability that the element is included in a first list entry and a second score indicating the probability that the element is included in a second list entry for each element. Construct answers based on the representation of the webpage, wherein the answers are in list form and wherein the answers include: The first element, at the first list entry, is selected from the plurality of elements represented based on its first score, and the first element includes the first text of the webpage; as well as The second element, at the second list entry, is selected from the plurality of elements represented based on its second score. The second element includes the second text of the webpage, wherein the first and second texts are adjacent in the answer but separated by a third text on the webpage; and The answer is transmitted to the client computing device, wherein the client computing device is configured to present the answer to the user of the client computing device through an output interface associated with the client computing device.

Images