Multi-source extraction and scoring of short query responses

Abstract

A technique for generating short responses to a search engine query includes performing training operations on a corpus of training data to train a score prediction engine, the corpus of training data including candidate passages and respective remaining passages that provide short responses for display during a callout, from which top-scoring short responses are generated. In such an implementation, the corpus of training data further includes respective titles of the candidate passages and respective remaining passages.

Description

[Technical field]

[0001] This description relates to generating short responses for display in a browser running a search engine. [Background technology]

[0002] Some search engines support short responses displayed during a callout, which is displayed in a prominent location within the search browser window that displays the search engine results. The prominent display of such short responses provides users with very fast responses to factual queries without the user having to select a search result and click on it before getting a response to the query. Furthermore, search engines can directly provide responses to a diverse range of questions without a curated knowledge base. Summary of the Invention [Means for solving the problem]

[0003] Implementations described herein relate to generating short responses for display based on a number of passages resulting from a query provided to a search engine. A short response is a response to a factual query that is prominently displayed in a browser window running a search engine, and such responses can be derived from or are passages in the top-ranked search results. Short responses are simply the passages from which they are derived. Thus, using other passages for context when deriving a short response from a passage should improve the quality of the short response. The decision to display a particular short response depends on a probability score provided by a probability score prediction engine, and the short response is displayed or not displayed based on whether the probability score is greater than or less than a probability score threshold, respectively. The probability score can be determined by a probability score prediction engine that predicts a probability score for a passage from a top-ranked search result based on a match with other passages from other search results. The accuracy score prediction engine is trained using passages from search engine results that have been manually scored by raters based on matches with contextual passages from other search results.

[0004] In one general aspect, a computer-based method can include receiving query data representing a search query entered by a user into a search engine. The method can also include generating a plurality of search results based on the search query, each of the plurality of search results having a respective passage related to the search query. The method can further include selecting a set of the respective passages, where one of the respective passages in the set is a candidate passage belonging to a top-ranked search result of the plurality of search results, and each remaining respective passage in the set is a context passage. The method can further include scoring the candidate passage using the context passage to generate a confidence score for the candidate passage. The method can further include providing the candidate passage for display as a short response in a search result page rendered by the browser window on a display based on the confidence score.

[0005] In another general aspect, a computer program product includes a non-transitory storage medium, the computer program product including code that, when executed by a processing circuit of a computing device, causes the processing circuit to perform a method. The method can include receiving query data representing a search query entered by a user into a search engine. The method can also include generating a plurality of search results based on the search query, each of the plurality of search results having a respective passage related to the search query. The method can further include selecting a set of the respective passages, where one of the respective passages in the set is a candidate passage that belongs to a top-ranked search result of the plurality of search results, and each remaining respective passage in the set is a context passage. The method can further include scoring the candidate passage using the context passage to create a confidence score for the candidate passage. The method can further include providing the candidate passage for display as a short response in a search result page rendered by the browser window on the display based on the confidence score.

[0006] In another general aspect, an electronic device includes a memory and a control circuit coupled to the memory. The control circuit can be configured to receive query data representing a search query entered by a user into a search engine. The control circuit can also be configured to generate a plurality of search results based on the search query, each of the plurality of search results having a respective passage related to the search query. The control circuit can be further configured to select a set of the respective passages, one of the respective passages in the set being a candidate passage belonging to a top-ranked search result of the plurality of search results, and each remaining respective passage in the set being a context passage. The control circuit can be further configured to score the candidate passage using the context passage to create a confidence score for the candidate passage. The control circuit can be further configured to provide the candidate passage for display as a short response in a search results page rendered by the browser window on the display based on the confidence score.

[0007] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0008] [Figure 1A] FIG. 1 illustrates an exemplary search engine capable of implementing the improved techniques described herein. [Figure 1B] FIG. 1 illustrates an exemplary short response resulting from a search query and displayed in a browser window on a display. [Diagram 2] FIG. 1 illustrates an exemplary electronic environment in which the improved techniques described herein may be implemented. [Diagram 3] FIG. 1 illustrates an example flow of data from multiple top-scoring passages from a search result to a callout. [Figure 4]FIG. 1 illustrates an example input and output from a likelihood score prediction engine. [Diagram 5] 1 is a flowchart illustrating an example method for implementing match-based short response presentation in accordance with disclosed implementations. [Figure 6] 1 illustrates examples of computing devices and mobile computing devices that can be used to implement the techniques described. [Figure 7] FIG. 1 illustrates an example of a distributed computing device that can be used to implement the techniques described. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Some search engines support short answers that are displayed in a prominent location within the search browser window that displays the search engine results. The short answers include passages or extracts from passages that represent a response to a factual query. The prominent display of such short answers provides the user with a very fast response to the query without the user having to select a search result and click on it before getting a response to the query. Furthermore, search engines are capable of directly providing responses to a diverse range of questions without a curated knowledge base. However, selecting passages for short answers can be complex and error-prone.

[0010] For example, in response to a user query "What does NFC mean in sports," a search engine may display a short response in a prominent location on the user's display. In this case, the user would expect to see "National Football Conference" prominently displayed in the search engine's browser window on the user's display. However, the response "Near Field Communication" may be displayed instead.

[0011] Conventional search engines that support short responses, which are often displayed in callouts, obtain each short response separately from each search result. The search results selected for a short response for a given query may represent websites and may include links to the websites that are displayed in the search results from which the short response was obtained. Selection of a short response by a search engine includes obtaining passages from the websites for the top-ranked search results and selecting the passages (ranges of text) that are determined to be most likely to contain a response to the query.

[0012] A technical problem with the above-mentioned conventional approach to generating short answers is that the short answers in the above-mentioned conventional search engines may vary in quality depending on the search result selected. In such an approach, the logic used to select the passage from which the short answer is obtained does not take into account the quality of the short answer. Thus, in some cases, the short answer may be misleading or even incorrect.

[0013] According to the implementation described herein, the technical solution to the above technical problem includes an improved scoring engine (confidence score prediction engine) for determining whether to display a short response. The improved scoring engine uses multiple passages from multiple different respective search results. The decision to display a particular short response depends on the confidence score provided by the confidence score prediction engine, and the short response is displayed or not displayed based on whether the confidence score is greater than or less than a confidence score threshold, respectively. The confidence score can be determined by the confidence score prediction engine, which predicts confidence scores for passages from top-ranked search results based on matches with other passages from other search results. The confidence score prediction engine is trained using passages from search engine results that have been manually scored by raters based on matches with contextual passages from other search results.

[0014] A technical advantage of the disclosed implementations is that, in contrast to conventional search engines, the short responses provided by the improved search engine described above are of higher quality (e.g., are more likely to be correct) because the search engine determines the degree of match between multiple passages from different sources. Furthermore, the improved quality of the short responses may result in fewer user queries and therefore less network data.

[0015] FIG. 1A depicts an example environment 100 in which a user can interact with one or more computer-implemented search services. Exemplary computer-implemented search services include search services for email services, chat services, document sharing services, calendar sharing services, photo sharing services, video sharing services, shopping services, blog services, microblog services, social network services, location (location-aware) services, check-in services, and / or ratings and reviews services. In the example of FIG. 1A, an Internet search service is depicted and will be discussed in further detail herein. However, it should be understood that implementations of the present disclosure may include one or more computer-implemented services, such as the examples discussed herein.

[0016] 1A , the search system 120 provides search services. The exemplary environment 100 includes a network 102, such as, for example, a local area network (LAN), a wide area network (WAN), the Internet, or a combination thereof, connecting websites 104, user devices 106, and the search system 120. In some examples, the network 102 can be accessed via wired and / or wireless communication links. For example, mobile computing devices, such as smartphones, can utilize cellular networks to access the network. The environment 100 can include millions of websites 104 and user devices 106.

[0017] In some examples, a website 104 is associated with a domain name and provided as one or more resources 105 hosted by one or more servers. An example website is a collection of web pages formatted in a suitable machine-readable language, such as, for example, HyperText Markup Language (HTML), which may include text, images, multimedia content, and programming elements such as scripts. Each website 104 is maintained by a publisher, for example, an entity that manages and / or owns the website. Website resources 105 may be static or dynamic.

[0018] In some examples, resources 105 are data provided over network 102 and associated with a resource address, e.g., a Uniform Resource Locator (URL). In some examples, resources 105 that website 104 can provide include web pages, word processing documents, and portable document format (PDF) documents, images, videos, and feed sources, among other suitable digital content. Resources 105 can include content, e.g., words, phrases, images, and sounds, and can include embedded information, e.g., meta information and hyperlinks, and / or embedded instructions, e.g., scripts.

[0019] In some examples, user devices 106 are electronic devices under the control of a user that can request and receive resources 105 over the network 102. Exemplary user devices 106 include mobile computing devices such as personal computers, e.g., smartphones, wearable devices, and / or tablet computing devices that can send and receive data over the network 102. As used throughout this document, the term mobile computing device (mobile device) refers to a user device that is configured to communicate over a mobile communication network. For example, a smartphone, which is a phone capable of communicating over the Internet, is an example of a mobile device, as are wearable devices and other smart devices, such as smart speakers. User devices 106 typically include user applications, e.g., a web browser, that facilitate sending and receiving data over the network 102.

[0020] In some examples, to facilitate searching for resources 105, the search system 120 identifies the resources 105 by crawling and indexing the resources 105 provided on the websites 104. Data about the resources 105 may be indexed based on the resources to which the data corresponds. The indexed and optionally cached copies of the resources 105 are stored in the search index 122, for example as indexed resources 126.

[0021] A user device 106 submits a search query 109 to the search system 120. In some examples, the user device 106 can include one or more input modalities. Exemplary modalities include a keyboard, a touch screen, and / or a microphone. For example, a user can use a keyboard and / or a touch screen to type the search query. As another example, a user can speak the search query and the user's voice can be captured through a microphone and processed through speech recognition to provide the search query.

[0022] In response to receiving the search query 109, the search system 120 accesses the search index 122 and identifies resources 105 that are related to the search query 109, e.g., have at least a minimum specified relevance score for the search query 109. The search system 120 identifies the resources 105, creates a search result display 111 including search results 112 and short responses 113 that identify the resources 105, and returns the search result display 111 to the user device 106. In the illustrated context, the search result display may include one or more web pages, e.g., one or more search result pages. In some examples, the web pages may be provided based on web documents that may be written in any suitable machine-readable language. However, it is contemplated that implementations of the present disclosure may include other suitable display types. For example, the search results may be provided in a display generated by an application executed on the computing device and / or a display generated by an operating system, e.g., a mobile operating system. In some examples, the search results may be provided based on any suitable format, e.g., JavaScript-html, plain text, etc.

[0023] Search results 112 are data generated by the search system 120 that identify resources 105 responsive to a particular search query and include links to the resources 105. An example search result 112 may include a webpage title, a snippet of text or a portion of an image extracted from the webpage, and a URL for the webpage. In some examples, the data provided in the search results 112 may be retrieved from a resource data store. For example, the search system 120 may provide a search result display 111 that displays the search results 112. In some examples, the search result display 111 may be populated with information provided from the resource data store, such as the webpage title, a snippet of text or a portion of an image extracted from the webpage.

[0024] According to implementations of the present disclosure, the example environment 100 also includes a short response system 130 that is communicatively coupled to the search system 120, e.g., directly coupled or coupled via a network, such as the network 102. In some examples, in the case of multiple computer-implemented services, the short response system 130 is communicatively coupled to each system that provides the functionality of each computer-implemented service. In some examples, as discussed in more detail herein, a system, such as the search system 120, can interact with the short response system 130 to provide the correct short response 113 in a prominent location, often displayed in a callout, in the browser window running the search system 120.

[0025] 1B is a diagram illustrating a user interface 150 in which an example short response 190 appears. In the example of FIG. 1B, the short response 190 includes an extracted response 170. The extracted response 170 is extracted from the short response 190. The short response 190 is displayed in a browser window on a display resulting from the search query 160. In this case, the short response 190 is based on a passage from the top-ranked search results 180. Note that the short response 190 (and, more specifically, the extracted response 170) correctly answers the question. In some implementations, the short response 190 may be displayed without the extracted response 170.

[0026] A conventional search engine displays a short answer 190 based on the search results 180. Often, this may result in a short answer that is not correct as a passage, such that the short answer is taken from a top-ranked search result and has not been checked for accuracy. The above technical solution in this case would include generating a confidence score based not only on the passage, but also on other passages taken from other search results. Thus, if a potential short answer passage does not match well with the information in the other passages, the short answer 190 will not be displayed.

[0027] 2 is a diagram illustrating an exemplary electronic environment 200 in which the above-mentioned technical solutions can be implemented. A computer 220 is configured to generate short responses using a likelihood score prediction engine and to train the likelihood score prediction engine using a corpus of training data.

[0028] The computer 220 includes a network interface 222, one or more processing units 224, a memory 226, and a display interface 228. The network interface 222 includes, for example, an Ethernet adapter, a token ring adapter, etc., to convert electrical and / or optical signals received from a network into an electrical format used by the computer 220. The set of processing units 224 includes one or more processing chips and / or assemblies. The memory 226 includes both volatile memory (e.g., RAM) and non-volatile memory such as one or more ROMs, disk drives, solid state drives, etc. The set of processing units 224 and the memory 226 together form a control circuit, which is configured and arranged to perform various methods and functionality as described herein. The display interface 228 is configured to provide data to a display device for rendering and displaying to a user.

[0029] In some implementations, one or more of the components of the computer 220 may be or may include a processor (e.g., processing unit 224) configured to process instructions stored in memory 226. Examples of such instructions as depicted in Figure 2 include a query manager 230, a search engine manager 232, an accuracy score manager 240, and a prediction engine manager 250. Additionally, as illustrated in Figure 2, the memory 226 is configured to store various data, which will be described with respect to each manager that uses such data.

[0030] The query manager 230 is configured to receive the query data 231. In some implementations, the query manager 230 receives the query data 231 via the network interface 222, i.e., over a network. In some implementations, the query manager 230 receives the query data 231 directly as user input.

[0031] The query data 231 represents a query input to a search engine (e.g., the search system 120 in FIG. 1A). In some implementations, the query data 231 takes the form of a string of characters. In some implementations, the query data 231 takes an alternative form, such as, for example, a graphic, an image, an audio clip, a video, etc.

[0032] Search engine manager 232 is configured to process query data 231 for input into a search engine (e.g., search system 120 in FIG. 1A ) to generate search engine data 233. Search engine data 233 includes a set of search engine results 234(1),...,234(N). Each of search engine results 234(1),...,234(N) includes a link to a website that includes a respective passage 235(1),...,235(N) and a respective ranking 236(1),...,236(N).

[0033] Each passage, e.g., passage 235(1), is at least a portion of the content on the website to which search result 234(1) is linked. In the examples described herein, each of passages 235(1),..., 235(N) includes text. In some implementations, the text forms a paragraph. In some implementations, there is a specified amount of text, e.g., a specified number of words, sentences, or other measures of the amount of information, in a passage, e.g., passage 235(1). However, in some implementations, at least one of passages 235(1),..., 235(N), e.g., passage 235(1), includes a graphic, such as an emoji, a portion of an image, an audio clip, a video clip, or the like.

[0034] The respective rankings 236(1),..., 236(N) of search results 234(1),..., 234(N) represent the arrangement of search results, such as being displayed in a browser window on a display via, for example, a display interface 228. In some implementations, each ranking 236(1),..., 236(N) takes the form of a sequence of numbers. In this way, the top-ranked search result is 234(1) and the lowest-ranked search result is 234(N).

[0035] The confidence score manager 240 is configured to receive confidence score data 241 from a human rater as training data for a prediction engine configured to predict the confidence score of the message of the top-ranked search result 234(1). As shown in FIG. 2, the confidence score data 241 received from the human rater includes confidence scores 242(1),..., 242(N) for each message 235(1),..., 235(N) respectively. In some implementations, the human rater provides confidence scores for only M top-ranked messages 235(1),..., 235(M), where M < N. In some implementations, only the top-ranked message 235(1) is provided with a confidence score.

[0036] In some implementations, the confidence score of the top-ranked message 235(1) is based on the confidence of the message 235(1) as understood by a human rater in the context of a set of other messages such as 235(2),..., 235(K). In some implementations, K = 1 or 2. In some implementations, there are multiple top-ranked messages each having a confidence score.

[0037] For example, in response to query data 231 of "how long does fish last in the freezer," the search engine returns as the top-rated passage 235(1) "When vacuum sealed and properly stored in a freezer, fish can last for two years. When frozen fish is properly thawed, there should be little difference in texture when compared to fresh fish." In some implementations, a short response based on passage 235(1) may now include "two years." prominently displayed. In some implementations, the short response may be passage 235(1). The confidence score 242(1) provided by a human rater without further context depends solely on passage 235(1) and the rater's personal subject matter knowledge, which may be lacking.

[0038] However, in an improved search engine, confidence score manager 240 provides the rater with the opportunity to examine other passages for context in order to provide a more accurate confidence score 242(1) for passage 235(1) that is less dependent on the rater's personal knowledge. For example, passage 235(2) might state, "Frozen fish or shellfish will be safe indefinitely, but flavor and texture will deteriorate after extended storage. For best quality, freeze cooked fish (below 0F / -17.8C) for a maximum period of three months. Frozen raw fish is best used within three to eight months, and shellfish within three to twelve months." When passage 235(2) is used in context, a rater may assign a lower confidence score 242(1) to the short response of "two years" because passage 235(2) has a clear distinction from passage 235(1). If the short response was derived from passage 235(2), the rater might have provided “3 to 8 months” rather than “2 years.” In this case, a prediction engine manager trained using the confidence scores might conclude that the short response should not be displayed.

[0039] In some implementations, the confidence score manager 240 provides the rater with more than one passage for context. For example, in addition to passages 235(1) and 235(2), passage 235(3) might state, "Fatty fish such as salmon can be frozen for 2-3 months, while lean fish such as cod or flounder will last up to 6 months. Cooked leftovers 3 months. Ground meat (beef, lamb, pork) will last 2-3 months. Roasts, steaks, and chops can last at least 6 months in the freezer." If passage 235(3) is provided for context in addition to passage 235(2), the rater may give the short response of "2 years" an even lower confidence score 242(1) because passage 235(3) also has a clear difference from passage 235(1). However, in this case, if either passage 235(2) or passage 235(3) is selected as the top-rated passage from which a short response is to be derived, the confidence score 242(1) for that passage may be increased over that for passage 235(1) because there is some match between passage 235(2) and passage 235(3).

[0040] In some implementations, when there is more than one top-ranked passage, the confidence score manager 240 selects the top-ranked passage with the highest confidence score.

[0041] In some implementations, the decision as to whether or not a short response is displayed is based on whether the confidence score 242(1) is greater than or less than a specified confidence score threshold 244. In some implementations, the confidence scores provided by the rater are between −1.0 and 3.0 in 0.5 steps, with −1.0 indicating a short response that is clearly bad in the context of other passages (or within the rater's personal knowledge) and 3.0 indicating a short response that is a perfect match in the context of other passages. In this case, the prediction engine manager 250 can be trained to output scores between −1.0 and 3.0, and the search system can use a confidence score threshold 244 of 0.5. That is, short responses with ratings of −1.0, −0.5, and 0.0 may not be displayed, while short responses with confidence scores of 0.5, 1.0, 1.5, 2.0, 2.5, or 2.0 may be displayed.

[0042] In some implementations, there are multiple accuracy score thresholds that correspond to different levels of short response accuracy. For example, the 0.5 accuracy score threshold 244 mentioned above can be considered a lower threshold ("good response accuracy (GAP)"). Other thresholds can be defined, for example, at 1.5 ("very good response accuracy (VGAP)") or 2.0 ("excellent response accuracy (XGAP)").

[0043] In some implementations, to generate training data for the accuracy score prediction engine, a human rater is given a dataset of examples (e.g., primary passages) to score based on the level of accuracy derived from the rater's knowledge as well as additional contextual passages. Primary passages are selected in response to a query and include short responses that the rater is asked to score as bad, good, very good, etc. In some implementations, examples are selected based on specified criteria, e.g., "difficult to rate without additional investigation," "examples of random traffic," etc. In some implementations, trivial examples such as "What number does 1+1 equal?" are filtered out. In some implementations, among such examples, a larger percentage of the examples, e.g., 95%, may satisfy the GAP threshold, while a smaller percentage, e.g., 85%, may satisfy the VGAP threshold.

[0044] To provide these examples and scoring results to an accuracy score prediction engine, in some implementations, the templates for which the rater provides an accuracy score may themselves be evaluated. In some implementations, the evaluation takes the form of VGAP precision and VGAP recall. (Precision is the ratio of the number of true positive results to the total number of true and false positives, and recall is the ratio of the number of true positive results to the total number of true positives and false negatives.) For example, a VGAP precision of about 50% for some datasets implies a significant number of false positives, while a VGAP of about 95% implies a smaller number of false negatives in such examples.

[0045] In some implementations, the templates for which raters provide confidence scores can be adjusted to improve VGAP accuracy. For example, a baseline template may not use the titles or URLs of search results 234(1),...,234(N). In some implementations, some or all of the following changes can be made to the template to improve VGAP accuracy: - Add two further passages that are top passages from different search results, the other passages themselves providing a response at least at the GAP threshold. - Raters are given more time to arrive at a confidence score. - The rater is requested / required to provide a comment and the title of the result is given. - Randomize the order of the passages (for example, so that the primary passage is not necessarily first in the list). - Add a bias term (e.g., -0.5) to the accuracy score. In some implementations, the actions taken to improve VGAP precision result in a decrease in VGAP recall. Empirically, adding a bias term appears to have the largest negative effect on recall and the largest positive effect on precision.

[0046] The prediction engine manager 250 is configured to make likelihood score predictions based on the prediction engine data 253 provided by the raters and to generate predicted likelihood score data 265 and, in some implementations, short response data 270. As shown in FIG. 2, the prediction engine manager 250 includes an encoder 251 and a decoder 252.

[0047] Encoder 251 is configured to encode inputs into the prediction engine into a format suitable for input into a machine learning engine such as a neural network. For example, in some implementations, inputs into the likelihood score prediction engine include a query (e.g., query data 231), a top-rated passage (e.g., passage 235(1)), a title of the top-rated passage, and an additional passage. In some implementations, the input may include an additional title (e.g., corresponding to the additional passage). The additional passage is referred to as a context passage. Encoder 251 is configured to convert these inputs into input tokens with embedded vectors. Decoder 252 is configured to decode the output layer of the likelihood score prediction engine to create a predicted likelihood score. The operations of encoder 251 and decoder 252 are illustrated in FIG. 4.

[0048] 4 is a diagram 400 illustrating an example input 405 and output 442 from a confidence score prediction engine. As shown in FIG. 4, the input takes the form of a query 410, a top-ranked passage 420, a corresponding title 422, further passages 430 and 434, and their respective titles 432 and 436. The encoder 251 (FIG. 2) encodes the input to create input token embeddings 408, which are input into the input layer of the neural network from which the confidence score prediction engine 438 is generated. Once the training cycle is complete, the confidence score prediction engine 438 decodes the encoded output 442 to generate a predicted confidence score 444. However, during training, the neural network creates a loss function output 440 based only on the top-ranked passages 420.

[0049] Returning to FIG. 2, prediction engine data 253 represents data used to train the likelihood score prediction engine. As shown in FIG. 2, prediction engine data 253 includes input embedding data 254 and intermediate data 255. In some implementations, the likelihood score prediction engine as shown in FIG. 2 adopts a teacher-student model from knowledge distillation (KD). The main idea of ​​KD includes two network structures named teacher and student. The student can be a simple model while the teacher is a model with strong ability. The teacher model is used to teach the student model by transferring considerable knowledge to the student.

[0050] The intermediate data 255 represents data that the accuracy score prediction engine uses during its training. As shown in FIG. 2, the intermediate data 255 includes any of the output of the loss function, hidden layer values, and intermediate score predictions that are fed between the teacher model and the student model. In some implementations, the teacher model and the student model each have multiple stages, such that the output of one stage is input into the next stage. In some implementations, the teacher model has four stages and the student model has two stages.

[0051] The predicted likelihood score data 265 represents the output of the likelihood score prediction engine. In some implementations, the predicted likelihood score data 265 is used to determine whether to display a short response.

[0052] Short response data 270 represents a short response that may or may not be displayed.

[0053] The components of the user device 106 (e.g., modules, processing unit 224) can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms), which can include one or more types of hardware, software, firmware, operating systems, runtime libraries, etc. In some implementations, the components of the computer 220 can be configured to operate within a cluster of devices (e.g., a server farm). In such implementations, the functionality and processing of the components of the computer 220 can be distributed across several devices of the cluster of devices.

[0054] The components of computer 220 may be or include any type of hardware and / or software configured to process attributes. In some implementations, one or more portions of the components depicted in the components of computer 220 in FIG. 2 may be or include hardware-based modules (e.g., digital signal processor (DSP), field programmable gate array (FPGA), memory), firmware modules, and / or software-based modules (e.g., modules of computer code, sets of computer-readable instructions that can be executed by a computer). For example, in some implementations, one or more portions of the components of computer 220 may be or include software modules configured to be executed by at least one processor (not shown). In some implementations, the functionality of a component may be included in different modules and / or different components than those depicted in FIG. 2, combining the functionality depicted as two components into a single component.

[0055] Although not shown, in some implementations, the components of computer 220 (or portions thereof) can be configured to operate, for example, in a data center (e.g., a cloud computing environment), a computer system, one or more server / host devices, etc. In some implementations, the components of computer 220 (or portions thereof) can be configured to operate in a network. Thus, the components of computer 220 (or portions thereof) can be configured to function within various types of network environments that can include one or more devices and / or one or more server devices. For example, the network can be and include a local area network (LAN), a wide area network (WAN), etc. The network can be and include a wireless network, and / or a wireless network implemented using, for example, gateway devices, bridges, switches, etc. The network can include one or more segments and / or have portions based on various protocols, such as the Internet Protocol (IP) and / or proprietary protocols. The network can include at least a portion of the Internet.

[0056] In some implementations, one or more of the components of the search system 120 may be or may include a processor configured to process instructions stored in a memory, such as the query manager 230 (and / or portions thereof), the search engine manager 232 (and / or portions thereof), the accuracy score manager 240 (and / or portions thereof), and the prediction engine manager 250 (and / or portions thereof).

[0057] In some implementations, the memory 226 may be any type of memory, such as random access memory, disk drive memory, flash memory, etc. In some implementations, the memory 226 may be implemented as more than one memory component (e.g., more than one RAM component or disk drive memory) associated with the components of the computer 220. In some implementations, the memory 226 may be a database memory. In some implementations, the memory 226 may be or include a non-local memory. For example, the memory 226 may be or include a memory shared by multiple devices (not shown). In some implementations, the memory 226 may be associated with a server device (not shown) in a network and configured to serve the components of the computer 220. As illustrated in FIG. 2, the memory 226 is configured to store various data including query data 231, search engine data 233, confidence score data 241, prediction engine data 253, predicted confidence score data 265, and short response data 270.

[0058] Figure 3 is a diagram illustrating an example flow of data 300 from multiple top-scoring passages from a search result to a callout. As shown in Figure 3, a query 310 entered into a search engine produces search results in the form of documents 320(1-4). As shown in Figure 3, each of documents 320(1-4) has M (although some may have fewer) passages 330(1-4)(1-M). Passages 330(1)(1), 330(1)(2), and 330(1,3) are entered into a probability score prediction engine to generate a callout 340.

[0059] 5 is a flow chart illustrating an exemplary method 500 for generating short responses for display based on multiple passages resulting from a query submitted to a search engine according to the improved techniques described above. Method 500 may be implemented by software instructions, described with respect to FIG. 2, resident in memory 226 of computer 220 and executed by set of processing units 224.

[0060] At 502, the query manager 230 receives query data (eg, query data 231) representing a search query entered by a user into a search engine.

[0061] At 504, search engine manager 232 generates a plurality of search results (e.g., search results 234(1-N)) based on the search query. Each of the plurality of search results has a respective passage (e.g., passages 235(1-N)) related to the search query. The respective passages may be selected using conventional methods.

[0062] At 506, the prediction engine manager 250 selects a set of respective passages for scoring the short response. One of the respective passages in the set is a candidate passage that belongs to a top-ranked search result of the plurality of search results, and each remaining passage in the set is a context passage. In some implementations, the set includes a predefined number of passages, e.g., from a predefined number of the top-ranked search results. In some implementations, the set includes three passages, including passages from the top-ranked search results.

[0063] At 508, the prediction engine manager 250 scores the candidate passage using the context passage to create a confidence score for the candidate passage. In addition to the context passage, the prediction engine manager 250 may also include as input the search query (or a portion thereof) and a title corresponding to the candidate passage. In some implementations, the prediction engine manager 250 may also include as input the title corresponding to the context passage.

[0064] At 510, the prediction engine manager 250 provides candidate passages for display as short responses in a search results page rendered by the browser window on the display, based on the confidence scores satisfying a threshold.

[0065] 6 illustrates an example of a generic computing device 600 and a generic mobile computing device 650 that can be used with the techniques described herein. The computing device 600 is one exemplary configuration of the search system 120 and / or short response system 130 of FIG. 1A and the computer 220 of FIG.

[0066] 6, computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Computing device 650 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, and other similar computing devices. The components, their connections and relationships, and their functions depicted herein are intended to be merely examples and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0067] The computing device 600 includes a processor 602, a memory 604, a storage device 606, a high-speed interface 608 that connects to the memory 604 and a high-speed expansion port 610, and a low-speed interface 612 that connects to a low-speed bus 614 and the storage device 606. Each of the components 602, 604, 606, 608, 610, and 612 are interconnected using various buses and may be mounted on a common motherboard or otherwise as appropriate. The processor 602 may process instructions for execution within the computing device 600, including instructions stored in the memory 604 or on the storage device 606, to display graphical information for a GUI on an external input / output device, such as a display 616, which is coupled to the high-speed interface 608. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and multiple types of memories. Also, multiple computing devices 600 may be connected with each device providing a portion of the required operations (e.g., a bank of servers, a group of blade servers, or a multiple processor system).

[0068] The memory 604 stores information within the computing device 600. In one implementation, the memory 604 is one or more volatile memory units. In another implementation, the memory 604 is one or more non-volatile memory units. The memory 604 may be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0069] The storage device 606 can provide mass storage for the computing device 600. In one implementation, the storage device 606 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar semiconductor memory device, or an array of devices, including a device in a storage area network or other configuration. The computer program product can be tangibly embodied in an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as the memory 604, the storage device 606, or a memory on the processor 602.

[0070] While the high-speed controller 608 manages bandwidth-intensive operations for the computing device 600, the low-speed controller 612 manages lower bandwidth-intensive operations. Such allocation of functions is merely an example. In one implementation, the high-speed controller 608 is coupled to the memory 604, the display 616 (e.g., through a graphics processor or accelerator), and is coupled to a high-speed expansion port 610 that can accept various expansion cards (not shown). In this implementation, the low-speed controller 612 is coupled to the storage device 506 and the low-speed expansion port 614. The low-speed expansion port can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), but can also be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, etc., or a network device such as a switch or router, etc., through a network adapter.

[0071] The computing device 600 can be implemented in a number of different forms, as shown in the figure. For example, the computing device 600 can be implemented as a standard server 620 or multiple times in a group of such servers. The computing device 600 can also be implemented as part of a rack server system 624. In addition, the computing device 600 can be implemented in a personal computer, such as a laptop computer 622. Alternatively, components from the computing device 600 can be combined with other components in a mobile device (not shown), such as device 650. Each such device may include one or more of the computing devices 600, 650, and the entire system may consist of multiple computing devices 600, 650 in communication with each other.

[0072] Computing device 650 includes, among other components, a processor 652, a memory 664, input / output devices such as a display 654, a communication interface 666, and a transceiver 668. Device 650 may also include a storage device, such as a microdrive or other device, to provide additional storage. Each of components 650, 652, 664, 654, 666, and 668 are interconnected using various buses, and some of the components may be mounted on a common motherboard or in other suitable manner.

[0073] The processor 652 can execute instructions within the computing device 650, including instructions stored in the memory 664. The processor can be implemented as a chipset of chips including separate analog and digital processors. The processor can provide, for example, control of the user interface, applications run by the device 650, and coordination of other components of the device 650, such as wireless communication by the device 650, etc.

[0074] The processor 652 can communicate with a user through a control interface 658 and a display interface 656 coupled to a display 654. The display 654 can be, for example, a TFT LCD (thin film transistor liquid crystal display) or an OLED (organic light emitting diode) display, or other suitable display technology. The display interface 656 can comprise suitable circuitry for driving the display 654 to present graphical and other information to the user. The control interface 658 can receive and translate commands from the user for submission to the processor 652. Additionally, an external interface 660 can be provided in communication with the processor 652 to enable short-range communication of the device 650 with other devices. The external interface 660 can be provided, for example, for wired communication in some implementations or for wireless communication in other implementations, and multiple interfaces can also be used.

[0075] The memory 664 stores information within the computing device 650. The memory 664 may be implemented as one or more of a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. An expansion memory 674 may also be provided and connected to the device 650 through an expansion interface 672, which may include, for example, a SIMM (single in-line memory module) card interface. Such expansion memory 674 may provide extra storage space for the device 650 or may store applications or other information for the device 650. In particular, the expansion memory 674 may include instructions that perform or supplement the above-mentioned processes and may also include security information. Thus, for example, the expansion memory 674 may be provided as a security module for the device 650 and may be programmed with instructions that enable secure use of the device 650. In addition, security applications may be realized via a SIMM card with additional information, such as placing identification information on the SIMM card in an unhackable manner.

[0076] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory, expansion memory 674, or memory on the processor 652, which may be received, for example, via the transceiver 668 or the external interface 660.

[0077] The device 650 can communicate wirelessly through a communication interface 666, which can include digital signal processing circuitry, if necessary. The communication interface 666 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication can occur, for example, through a radio frequency transceiver 668. In addition, short-range communication can occur, such as through the use of Bluetooth, WiFi, or other such transceivers (not shown). In addition, a GPS (Global Positioning System) receiver module 670 can provide additional navigation-related and positioning-related wireless data to the device 650, which can be used as appropriate by applications running on the device 650.

[0078] Device 650 can also communicate audibly using a voice codec 660, which can receive spoken information from a user and convert it into usable digital information. Voice codec 660 can also generate sounds audible to the user, such as through a speaker in the handset of device 650. Such sounds can include sounds from a voice telephone call, can include recorded sounds (e.g., voice messages, music files, etc.), and can also include sounds generated by applications running on device 650.

[0079] The computing device 650 can be implemented in a number of different forms, as shown in the figure. For example, the computing device 650 can be implemented as a cellular telephone 680. The computing device 650 can also be implemented as part of a smartphone 682, a personal digital assistant, or other similar mobile device.

[0080] 7 illustrates an example of a generic computing device 700 that may be the search system 120 of FIG. 2 and may be used with the techniques described herein. The computing device 700 is intended to represent various exemplary forms of large-scale data processing devices, such as servers, blade servers, data centers, mainframes, and other large-scale computing devices. The computing device 700 may be a distributed system having multiple processors, which may include networked storage nodes interconnected by one or more communication networks. The components, their connections and relationships, and their functions illustrated herein are intended to be merely examples and are not intended to limit the implementation of the invention described and / or claimed in this document.

[0081] The distributed computing system 70 may include any number of computing devices 780. The computing devices 780 may include servers or rack servers, mainframes, etc., and communicate via local or wide area networks, dedicated optical links, modems, bridges, routers, switches, wired or wireless networks, etc.

[0082] In some implementations, each computing device may include multiple racks. For example, computing device 780a includes multiple racks 758a-n. Each rack may include one or more processors, such as processors 752a-n and 762a-n. The processors may include data processors, networked storage devices, and other computer-controlled devices. In some implementations, one processor may act as a master processor to control scheduling and data distribution tasks. The processors may be interconnected through one or more rack switches 762a-n, and one or more racks may be connected through a switch 778. The switch 778 may handle communications between the multiple connected computing devices 700.

[0083] Each rack may include memory, such as memory 754 and memory 764, and storage devices, such as 756 and 766. Storage devices 756 and 766 may provide mass storage and may include volatile or non-volatile storage devices, such as networked disks, floppy disks, hard disks, optical disks, tapes, flash memory or other similar semiconductor memory devices or arrays of devices, and may include devices in a storage area network or other configuration. Storage devices 756 and 766 may be shared among multiple processors, multiple racks, or multiple computing devices and may include computer-readable media that store instructions executable by one or more of the processors. Memory 754 and 764 may include volatile memory units, non-volatile memory units, and / or other forms of computer-readable media, such as, for example, magnetic or optical disks, flash memory, cache, random access memory (RAM), read-only memory (ROM), and combinations thereof. Memory, such as memory 754, may also be shared among processors 752a-752n. Data structures such as an index may be stored, for example, across storage 756 and memory 754. Computing device 700 may include other components not shown, such as controllers, buses, input / output devices, communication modules, and the like.

[0084] An overall system, such as environment 100 of Figure 1, may be comprised of multiple computing devices 700 in communication with each other. For example, device 780a may communicate with devices 780b, 780c, and 780d, which may collectively be known as environment 100. As another example, computer 220 of Figure 2 may include one or more computing devices 700. Some of the computing devices may be located geographically close to each other and others may be located geographically far away. The layout of system 700 is merely an example, and the system may have other layouts or configurations.

[0085] In some implementations, providing a candidate passage for display based on the certainty score includes comparing the certainty score to a certainty score threshold, displaying the candidate passage on the display in response to the certainty score being greater than the certainty score threshold, and not displaying the candidate passage on the display in response to the certainty score being less than the certainty score threshold.

[0086] In some implementations, scoring the candidate passage using the context passages includes determining a level of match between the candidate passage and the set of context passages, and the confidence score is based on the level of match.

[0087] In some implementations, scoring the candidate passage using the context passage includes inputting the candidate passage and each of the remaining passages to a score prediction engine configured to predict a score based on the candidate passage and each of the remaining passages.

[0088] In some implementations, the step of scoring the candidate passage using the context passage further includes, in addition to inputting the candidate passage and each of the remaining passages into a score prediction engine, inputting respective titles of the candidate passage and each of the remaining passages into the score prediction engine.

[0089] In some implementations, the method further includes performing a training operation on a corpus of training data to train the score prediction engine, the corpus of training data including the candidate passage and each of the remaining passages providing short responses for display during the callout, from which a top-scoring short response is generated. In such implementations, the corpus of training data further includes each of the remaining passages and respective titles of the candidate passage and each of the remaining passages. In such implementations, performing the training operation includes applying a loss function based on a set of confidence score thresholds applied to the candidate passages, the loss function including, for each of the set of confidence score thresholds for the primary passages, a sigmoid cross-entropy loss for that threshold score. In such implementations, the loss function generates as an output an average of the sigmoid cross-entropy loss over the set of confidence score thresholds. In such implementations, the training operation includes multiple training stages, and the set of confidence score thresholds varies between the multiple training stages of the training operation. In such implementations, the training operations include a teacher training operation to create a teacher model and a student training operation to create a student model, where the student training takes as input a set of teacher accuracy scores created by the teacher model. In such implementations, the teacher training operation is based on a loss function that includes the mean squared error of the teacher model and the student model.

[0090] In some implementations, the corpus of training data also includes a third set of previously scored passages, a confidence score for each of the previously scored passages that is not based on a match between the candidate passage and each of the remaining passages.

[0091] In some implementations, providing candidate passages for display as the short response includes selecting a top-ranked passage having a highest confidence score from a predefined number of the top-ranked passages.

[0092] Various implementations of the systems and techniques described herein may be embodied in digital electronic circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may be special purpose or general purpose and may include implementations in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, at least one input device, and at least one output device that can be coupled to receive data and instructions from and transmit data and instructions to the storage system.

[0093] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0094] To achieve interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, as well as a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Similarly, other types of devices can be used to achieve interaction with a user. For example, feedback provided to the user can be in the form of any sensory feedback (e.g., visual feedback, audio feedback, or haptic feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0095] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or includes middleware components (e.g., as an application server), or includes front-end components (e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or includes any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0096] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0097] Although several implementations have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the present specification.

[0098] When an element is referred to as being on, connected to, electrically connected, coupled, or electrically coupled to another element, it will be understood that an element may be directly on, connected to, or coupled to another element, or one or more intermediate elements may be present. In contrast, when an element is referred to as being directly on, directly connected to, or directly coupled to another element, there are no intermediate elements present. Although the terms directly on, directly connected, or directly coupled may not be used throughout the detailed description, elements shown as being directly on, directly connected, or directly coupled may be referred to as such. The claims of this application may be modified to account for the example relationships described herein or shown in the figures.

[0099] Although certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It should therefore be understood that the appended claims are intended to cover all such modifications and changes as fall within the scope of the present implementations. It should be understood that the implementations are presented merely as examples and not as limitations, and that various changes in form and details can be made. Any part of the apparatus and / or method described herein can be combined in any combination other than mutually exclusive combinations. The implementations described herein can include various combinations and / or subcombinations of the functions, components, and / or features of the different implementations described.

[0100] Additionally, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desired results. Additionally, other steps may be provided or steps may be removed from the described flows, and other components may be added or removed from the described systems. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0101] 70 Distributed Computing Systems 100 Example Environments 102 Network 104 Websites 105 website resources 106 User Devices 109 Search Queries 111 Search results display 112 Results 113 Short Responses 120 Search System 122 Search Index 126 Indexing Resources 130 Short Response System 150 User Interface 160 Search Queries 170 Responses 180 Top Ranked Results 190 Short Responses 200 Electronic environment 220 Computer 222 Network Interface 224 Processing Unit 226 Memory 228 Display Interface 230 Query Manager 231 Query Data 232 Search Engine Manager 233 Search Engine Data 234(1) Search Engine Results 234(N) Search Engine Results 235(1) Passage 235(N) Passage 236(1) Ranking 236(N) Ranking 240 Accuracy Score Manager 241 Accuracy score data 242(1) Accuracy Score 242(N) Accuracy Score 244 Probability Score Threshold 250 Prediction Engine Manager 251 Encoder 252 Decoder 253 Prediction Engine Data 254 Input padding data 255 intermediate data 265 Predicted Probability Score Data 270 Short Response Data 310 Query 320(1) Document 1 330(1)(1) Passage 1 330(2)(1) Passage 2 330(M)(1) Passage M 320(2) Document 2 330(1)(2) Passage 1 330(2)(2) Passage 2 330(M)(2) Passage M 320(3) Document 3 330(1)(3) Passage 1 330(2)(3) Passage 2 330(M)(3) Passage M 320(4) Document 4 330(1)(4) Passage 1 330(2)(4) Passage 2 330(M)(4) Passage M 340 Callout 405 Input 408 Input Token Padding 410 Query 420 Passage 422 Titles 430 Further Passage 1 432 More Titles 1 434 Further Passage 2 436 More Titles 2 438 Accuracy Score Prediction Engine 440 Loss function output 442 encoded output 444 Predicted Probability Score 506 Storage Devices 600 General computer devices, computing devices 602 Processor 604 Memory 606 Storage Devices 608 High Speed ​​Interface 610 High Speed ​​Expansion Port 612 Low Speed ​​Interface 614 Low Speed ​​Expansion Port 616 Display 620 Standard Server 622 Laptop Computer 624 Rack Server System 650 General Mobile Computing Devices 652 processor 654 Display 656 Display Interface 658 Control Interface 660 external interface, audio codec 664 Memory 666 Communication Interface 668 Transceiver 670 GPS (Global Positioning System) Receiver Module 672 Expansion Interface 674 Expandable Memory 680 Cellular Phone 682 Smartphones 700 General Computer Devices 755 Memory 752a Processor 752n Processor 756 Storage device 758a Rack 758n Rack 762a Processor, Rack Switch 762n processor, rack switch 764 Memory 766 Storage device 778 Switch 780 Computing Devices 780a Computing Device 780b device 780c device 780d Device

Claims

1. receiving a search query; generating a plurality of search results based on the search query, each of the plurality of search results having a respective passage related to the search query; selecting a set of the respective passages, one of the respective passages in the set being a candidate passage belonging to a top-ranked search result of the plurality of search results, and the remaining respective passages in the set being context passages; scoring the candidate passage using the context passage to generate a confidence score for the candidate passage; providing the candidate passages for display as short responses in a search results page rendered by a browser window on a display based on the confidence scores; A computer-implemented method comprising:

2. providing the candidate passages for display based on the confidence scores, comparing the certainty score to a certainty score threshold; displaying the candidate passage on the display in response to the confidence score being greater than the confidence score threshold; not displaying the candidate passage on the display in response to the confidence score being less than the confidence score threshold; 2. The computer-implemented method of claim 1, comprising:

3. Scoring the candidate passage using the context passage, 2. The computer-implemented method of claim 1, comprising determining a level of match between the candidate passage and the context passage, wherein the confidence score is based on the level of match.

4. Scoring the candidate passage using the context passage, inputting the candidate passage, the search query, and the context passage into a score prediction engine configured to predict the accuracy score based on the candidate passage, the search query, and the context passage; 2. The computer-implemented method of claim 1, comprising:

5. Scoring the candidate passage using the context passage, inputting the candidate passage, the search query, and the context passage into the score prediction engine, and further inputting titles of each of the candidate passage and the context passage into the score prediction engine. The computer-implemented method of claim 4, further comprising:

6. 5. The computer-implemented method of claim 4, wherein the score prediction engine is trained on a corpus of training records, the corpus of training records including training queries, primary passages selected for the training queries, at least one context passage, and respective confidence scores for the primary passages.

7. 7. The computer-implemented method of claim 6, wherein the corpus of training recordings further comprises a title of each of the primary passages for the at least one context passage.

8. 7. The computer-implemented method of claim 6, wherein the score prediction engine is further trained by applying a loss function based on a set of likelihood score thresholds applied to the primary passage, the loss function comprising, for each of the set of likelihood score thresholds for the primary passage, a sigmoid cross-entropy loss for that threshold score.

9. 9. The computer-implemented method of claim 8, wherein the loss function produces as an output an average of the sigmoid cross-entropy loss over the set of likelihood score thresholds.

10. the score prediction engine is trained using multiple training stages; 9. The computer-implemented method of claim 8, wherein the set of confidence score thresholds varies among the multiple training stages.

11. 7. The computer-implemented method of claim 6, wherein the corpus of training records also includes a set of previously scored passages, confidence scores for each of the previously scored passages that are not based on matching with the primary passage, and at least one context passage.

12. 2. The computer-implemented method of claim 1, wherein the set of each passage includes a predefined number of top-ranked passages.

13. 13. The computer-implemented method of claim 12, wherein providing the candidate passages for display as the short response comprises selecting a top-ranked passage having a highest confidence score among the predefined number of top-ranked passages.

14. A computer program stored on a computer readable storage medium, the computer program comprising code that, when executed by a processing circuit of a computing device, causes the processing circuit to perform a method, the method comprising: receiving query data representing a search query; generating a plurality of search results based on the search query, each of the plurality of search results having a respective passage related to the search query; selecting a set of the respective passages, one of the respective passages in the set being a candidate passage belonging to a top-ranked search result of the plurality of search results, and the remaining respective passages in the set being context passages; scoring the candidate passage using the context passage to generate a confidence score for the candidate passage; providing the candidate passages for display as short responses in a search results page rendered by a browser window on a display based on the confidence scores; A computer program comprising:

15. providing the candidate passages for display based on the confidence scores, comparing the certainty score to a certainty score threshold; displaying the candidate passage on the display in response to the confidence score being greater than the confidence score threshold; not displaying the candidate passage on the display in response to the confidence score being less than the confidence score threshold; 15. The computer program of claim 14, comprising:

16. Scoring the candidate passage using the context passage, inputting the candidate passage and the context passage into a score prediction engine configured to predict the accuracy score based on the candidate passage and the context passage.

15. The computer program of claim 14, comprising:

17. Scoring the candidate passage using the context passage, In addition to inputting the candidate passage and the context passage into the score prediction engine, inputting titles of each of the candidate passage and the context passage into the score prediction engine.

17. The computer program of claim 16, further comprising:

18. The method further comprising:

17. The computer program product of claim 16, further comprising: performing a training operation on a corpus of training data to train the score prediction engine, the corpus of training data including candidate passages providing short responses for display and respective remaining passages from which top-scoring short responses are generated.

19. Memory, a control circuit coupled to the memory, receiving query data representing a search query; generating a plurality of search results based on the search query, each of the plurality of search results having a respective passage related to the search query; selecting a set of the respective passages, one of the respective passages in the set being a candidate passage belonging to a top-ranked search result among the plurality of search results, and each remaining respective passage in the set being a context passage; scoring the candidate passage using the context passage to generate a confidence score for the candidate passage; providing the candidate passages for display as short responses in a search results page rendered by a browser window on a display based on the confidence scores; A control circuit configured as follows: An apparatus comprising:

20. the control circuitry configured to provide the candidate passages for display based on the confidence scores, comparing said likelihood score to a likelihood score threshold; displaying the candidate passage on the display in response to the confidence score being greater than the confidence score threshold; not displaying the candidate passage on the display in response to the confidence score being less than the confidence score threshold.

20. The apparatus of claim 19, further configured to:

Images