Dynamically suppress query responses in search

By dynamically evaluating and suppressing redundant content in search results, using query suppressors and pattern rules, the redundant problem in search results is solved, user experience and search efficiency are improved, and efficient and real-time presentation of search results is achieved.

CN112912873BActive Publication Date: 2024-11-26MICROSOFT TECHNOLOGY LICENSING LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201980067915.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2018-10-15
Filing Date
2019-09-04
Publication Date
2024-11-26
Estimated Expiration
2039-09-04

AI Technical Summary

Technical Problem

The prior art is difficult to effectively suppress redundant query replies and entity descriptions in search results, resulting in a decline in content redundancy and user experience, and the offline filtering mechanism is inefficient and cannot respond to changes in user query in real time.

Method used

By instantiating query suppressors, pattern suppressors and site suppressors, dynamically evaluate and suppress candidate query responses in search results based on query blacklists, pattern rules and site rules, ensuring that the content is not redundant and search results in real time Relevant information is presented in .

Benefits of technology

It improves the relevance and user experience of search results, reduces the display of redundant content, reduces the cost of computing and network, and enhances the real-time and accuracy of search results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112912873B_ABST
    Figure CN112912873B_ABST
Patent Text Reader

Abstract

A method for determining whether to dynamically suppress a candidate query answer that is designated for inclusion in search results includes: instantiating multiple filtering rules for evaluating suppression of candidate query answers. The filtering rules include one or both of pattern rules and site rules. The method also includes: receiving a query, and after receiving the query, receiving one or more candidate query answers previously associated with the query. The method also includes: for each candidate query answer, if either or both of the pattern rules and the site rules match the query, dynamically suppressing the candidate query answer from a curated position within the search results that has enhanced prominence relative to multiple other result entries. The method also includes: returning search results that include up to one candidate query answer in the curated position in response to the candidate query answer not being dynamically suppressed.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Internet search engines and other search providers are typically designed to provide many different search results in response to a search query. Internet search engines may be configured to present individual results in the results with different levels of prominence, such as a collection of web links or articles that may directly answer the query, thereby making it easier for users to digest the search results. Summary of the invention

[0002] This summary is provided to introduce in a simplified form a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all of the disadvantages noted in any part of this disclosure.

[0003] A method for determining whether to dynamically suppress a candidate query answer that is designated for inclusion in search results includes: instantiating multiple filtering rules for evaluating suppression of candidate query answers. The filtering rules include one or both of pattern rules and site rules. The method also includes: receiving a query. Such a mapping of queries to query answers is maintained for each of a plurality of expected queries. After receiving the query, one or more candidate answers (e.g., query answers previously associated with the query) are obtained. The method also includes: for each candidate query answer, if either or both of the pattern rules and the site rules match the query, dynamically suppressing the candidate query answer from a curated position within the search results that has enhanced prominence relative to a plurality of other result entries. The method also includes: returning search results that include up to one candidate query answer in a curated position in response to the candidate query answer not being dynamically suppressed. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1A An exemplary system architecture for suppressing query answers from search results is shown.

[0005] Figure 1B A method for providing search results that may suppress the answering of a query is shown.

[0006] Figure 2 A method for determining whether to suppress a query answer from search results is shown.

[0007] FIG. 3A to FIG. 3C Search results that suppress query answers and / or entity descriptions are shown.

[0008] Figure 4A method of determining whether to suppress a query answer based on similarity to an entity description provided within search results is shown.

[0009] Figure 5 An exemplary computing system is shown. DETAILED DESCRIPTION

[0010] The search results may include a query reply in the form of a natural language response to the question posed in the query (e.g., presented as text or voice audio). In addition to the query reply, the search results may include any other suitable result items, such as web search results, keyword-based search results, advertisements, descriptions of entities / events / places in the search query, and the like. In some examples, the query reply or another result item (e.g., an entity description) may be presented in a curated position with enhanced prominence relative to other result items. However, in some cases, one or more result items may not be expected to be included in a curated position. Therefore, the present disclosure relates to a method for determining whether to suppress certain result items in the search results when providing search functionality, such as to suppress query replies and / or entity descriptions that may be redundant in view of other result items in the search results. The suppression of a result item that is to be shown in a curated position may include removing the result item from the curated position, such as by downgrading the result item to be shown in a different position without much prominence, or by removing the result item entirely.

[0011] Figure 1A A non-limiting example of an architecture for providing search results to a client is shown. The search functionality may be provided by Figure 1A The search architecture shown or any other suitable architecture for providing search results may be provided. For example, the search architecture may allow client computer 110 to perform a search on any suitable search domain (e.g., website(s), encyclopedia(s), database(s), financial / medical / scientific records, etc.) using any suitable query (e.g., natural language keywords, regular expression patterns, Boolean operators for composing multiple queries, etc.). Figure 1A The devices shown may be communicatively coupled in any suitable manner to formulate Figure 1A The data flows shown by the arrows in , for example, by being communicatively coupled to a suitable computer network, such as the Internet.

[0012] The client computer 110 may be any suitable computing device, such as a mobile phone, a personal computer, a smart assistant speaker device, etc. The client computer 110 may optionally include a display device configured to present a browser 111. The browser 111 may be configured to present one or more pages, such as a search page configured to allow a user to enter a search query and / or view search results, such as Figure 1AAlternatively or additionally, the client computer 110 may allow the user to input a search query and / or receive search results in any other suitable manner, for example, the client computer 110 may include a microphone and a speaker and may be configured to receive a query via a natural language speech expression received at the microphone and output search results via speech audio output at the speaker. Although the present disclosure may be used in Figure 1A The illustrated aspects of the graphical browser 111 are described, but the methods and processes disclosed herein may be implemented in the context of any other search interface, such as a speech-based natural language search interface.

[0013] The search interface may include a search input field configured to receive user input representing a user query, such as a search bar 112 configured to receive user text. The search interface may also include visual cues configured to submit a query to receive search results, such as a "SEARCH" button. After the user enters the query, the browser 111 sends a computer-readable representation of the query to the service endpoint 120 to receive search results from the service endpoint. For example, the user submits the query "Who is Elrond?" In addition to the main results 113, the search results may optionally include additional result items presented in a curated position (e.g., a privileged position) of the search results that are indicated to have enhanced prominence within the search results relative to other result items. For example, as Figure 1A As shown, the search results may optionally also include a query answer 114 in a first curated position and / or an entity description 115 in a second curated position. The curated position may be indicated and / or given enhanced prominence in any suitable manner. Non-limiting examples of curated positions of search results include: 1) as the first result item in a list of result items; 2) near the top of the search results page (e.g., directly below the search bar as with the query answer 114); 3) as the left or right column of the search results page (e.g., on the right as with the entity description 115); 4) in a separate page (e.g., a pop-up, dialog box, and / or a new tab); 5) in a separate frame of the search results page; 6) in an outlined, shaded, or otherwise marked box; 7) in an area marked by a label indicating the type of curated result item (e.g., a label indicating "query answer") and / or 8) in a section of the search results page that does not scroll when the scroll bar of the search results page is used to scroll other result items. In some examples, the search results may be presented in the form of audio and / or text conversational interaction. Thus, a curated location may be indicated and / or given enhanced prominence through an audio and / or text announcement demarcating the start of a result entry in the curated location, through ranking of different result entries, and the like.

[0014] The service endpoint 120 is configured to return one or more search results for presentation in the browser 111 in response to receiving a query. The service endpoint 120 may be configured to provide a "raw" query in the form of literal text entered by a user. Alternatively or additionally, the service endpoint 120 may be configured to provide a "normalized" query in the form of a computer-readable description of the query content, such as by processing a computer-readable description indicating the query intent (representing the user's question, purpose, and / or task indicated by the query), by processing one or more terms in the query, and / or by processing the semantic structure of the query (e.g., a parse tree of the query). Query normalization may be performed by any suitable computer device(s), such as by the client computer 110 and / or the service endpoint 120. The normalized query may include relevant information content of the query (e.g., relevant intent / entities) while limiting the amount of variability between queries (e.g., different raw queries that reparse the same question may be normalized to the same normalized query).

[0015] The search results include primary results 113 that include various websites that may be relevant to the user's query. Figure 1A An example of a web search is shown, but the service endpoint 120 can be configured to return results from any other suitable domain. The service endpoint 120 can return search results based on data received from one or more other computers. For example, the service endpoint 120 can cooperate with the search result provider 130 to send a search query to the search result provider 130, and receive the main results 113 of the user query expected by the search domain, which may include relevant websites, documents, etc. Therefore, the search result provider 130 can be configured to determine the relevant search results for the query in any suitable manner, such as by searching in a database, indexing / crawl websites or documents, etc.

[0016] Non-limiting examples of search result items include query answers, entity descriptions, and non-curated search results. Alternately or in addition to query answers, entity descriptions, and / or non-curated search results, the search results may include any other suitable curated and / or non-curated content that may be presented in a curated location (e.g., in a different curated location separate from the query answers and entity descriptions). Non-limiting examples of other types of result items that may be included in the search results include: news search results, image search results, video search results, shopping search results, recipe search results, etc. Although the present description is about the suppression of query answers and / or entity descriptions, the methods of the present disclosure may be used to determine whether to suppress and / or include any type of curated and / or non-curated results.

[0017] A query reply is a direct response to one or more questions posed within a query, for example in the form of a natural language response, which may include any suitable content, such as one or more text passages containing information responsive to the question (e.g., pertinent, relevant, accurate, concise, and / or clearly worded (multiple) responses to the question). In some examples, a query reply may be provided only when the query expresses a question (e.g., when the query is in the form of a natural language question), and may not be provided for general searches (e.g., when the search consists solely of keywords without any semantic structure or question-related terms, a query reply may not be provided). Query replies may be selected based on the intent expressed by the user's question, for example, to satisfactorily address every aspect of the user's question. Query replies may be particularly suitable for answering queries in the context of an audio and / or text conversation because they can help users answer questions without having to browse through multiple other result entries. Entity descriptions are a collection of descriptive information about entities mentioned in a query.

[0018] Entity descriptions provide potentially relevant information about people, places, things, events, etc. that are mentioned anywhere within a query. For example, if the query includes a list of keywords and one of the keywords is the name of a celebrity, the search results provided for the query may include entity descriptions about the celebrity. Entity descriptions are typically not in the form of a natural language response to a question, and may include a collection of any suitable information about the relevant entity. Non-limiting examples of information for entity descriptions may include biographical information, historical information, photos, videos, audio clips, etc.

[0019] Non-curated search results include results from keyword-based searches, web searches, database searches, etc. that are not configured to adhere to a curated format such as a query answer or entity description. For example, the browser 111 shows a main result 115 that includes multiple non-curated results. Non-curated results may be related to a query without necessarily satisfying any specific relationship to the query, such as a non-curated result may not be in the form of an answer to a question and may not describe any specific entity named in the query. Non-curated results may be presented in the form of a title, a link URL, and / or a short summary and / or a content fragment (e.g., a web page fragment) from a non-curated result. Compared to a query answer, a user may need to navigate to a different page to view relevant content from a non-curated result, requiring navigation and redirection of attention from the search results to reach relevant information.

[0020] Query replies 114 include direct replies to the user query “Who is Elrond?” Query replies may be any curated and / or specifically selected content that may be particularly relevant to the user's search query. For example, when the search query is in the form of a question, the query reply may be in the form of a direct response to the question, as shown in query reply 114. Presenting relevant query replies may save user time and / or improve the efficiency of client computer 110 and / or service endpoint 120 in providing search results. For example, providing relevant query replies may alleviate the need for a user to perform repeated web searches and / or scroll through a large number of search results in order to successfully answer a question. In some examples, the query reply is pre-associated with a query that may be received at service endpoint 120. Accordingly, service endpoint 120 is configured to receive query replies from query-reply data repository 170.

[0021] The query-answer data store 170 is configured to maintain a query-answer mapping that includes, for each of a plurality of expected queries, a corresponding answer to the expected query. Although the present disclosure is described in terms of a query-answer mapping that includes one answer for each expected query, alternately, the query-answer mapping may include more than one candidate answer for each expected query. In some examples, more than one answer may be obtained for a given query to return more than one query answer in the search results for inclusion in a curated position. In some examples, more than one answer may be obtained to rank and / or select from the candidate answers, for example, to return the best query answer for inclusion in a curated position. The query-answer data store 170 may maintain associations with respect to the original and / or normalized query, for example, looking up a query answer in the query-answer mapping may include looking up the original query string and / or looking up a computer-readable representation of the normalized query. The query-answer data store 170 may determine associations in the query-answer mapping in any suitable manner, such as based on a natural language model, data mining / web crawling, etc. Using pre-associated queries / answers in the query-answer mapping can improve the efficiency of obtaining query answers (e.g., because the query answers are essentially pre-computed for the query and stored in the query-answer mapping in a locally organized manner) and / or the robustness / quality of the answers (e.g., because the answers in the query-answer mapping can be audited / validated to assess and improve quality). Alternately or additionally to using pre-associated mappings between search queries and search results, the query-answer data repository 170 can provide answers to queries in any other suitable manner, such as in real time based on data crawled from a website, etc.

[0022] Entity description 115 includes descriptions of one or more entities mentioned in the user's query. For example, since the user's query mentions "Elrond", entity description 115 includes a description of "Elrond" and a link to an encyclopedia entry about "Elrond" (e.g., a uniform resource locator (URL)). Entity descriptions can be included in search results for any suitable entity or any other object, concept, or noun, such as a fictional character (as shown), a real person, a place, a historical event, a movie, a music album, etc. The entity description can include any suitable descriptive content and / or links to other content. The entity description can be determined in any suitable manner, such as based on natural language processing, machine learning, artificial intelligence, data mining, and / or based on a previously configured association between an entity name and an entity description. In some examples, the entity description is received from an entity description provider 160, which is configured to receive a query from a service endpoint 120 and provide entity descriptions of one or more entities mentioned in the query. For example, the entity description provider 160 may maintain associations between entities and corresponding entity descriptions, so that the entity description(s) of a query may be returned by looking up the entity name in the query.

[0023] In some examples, it may not be desirable to include one or both of the query answers 114 and / or the entity descriptions 115. In some examples, content in different sections of the search results (e.g., content in the primary results 113, the query answers 114, and / or the entity descriptions 115) may be substantially repetitive, resulting in content redundancy in the search results. Therefore, such redundant content may be suppressed from one or both of the query answers 114 and / or the entity descriptions 115. In some examples, the content in the query answers 114 and / or the entity descriptions 115 may not be desirable (e.g., relevant, appropriate, and / or suitable) for a given query. For example, relevant results for a query may include content that may be sensitive (e.g., obscene content) that may not be appropriate for presentation in a prominent location in the search results. Alternately or additionally, based on user feedback (e.g., feedback indicating that the entity descriptions 115 and / or query answers 114 are inappropriate and / or unhelpful for a given query and the search results shown for that query), content may be indicated as not desirable for inclusion in the query answers 114 and / or for inclusion in the entity descriptions 115.

[0024] Thus, the query answers 114 and / or entity descriptions 115 may be suppressed from the search results prior to presenting the search results in the browser 111. In some examples, the query-answer data database 170 may include an offline filtering mechanism 171 that is configured to suppress one or more possible query answers from the search results, such as by removing such query answers from a query-answer mapping. However, such offline filtering may be slow and / or computationally intensive because suppressing answers from a query-answer mapping may require processing multiple different candidate query-answer pairs (e.g., to find and modify each relevant query for which a answer should be suppressed). Offline filtering may be particularly inappropriate when the suppression of query answers is based on changing data about user queries, search results, and / or user satisfaction with the query answers, because it may not be feasible to perform offline filtering with sufficient frequency to ensure that the search results reflect the changing data. In addition, the query-answer data repository 170 may not be aware of potentially relevant content that may be generated based on content retrieved from a query by the service endpoint 120. Figure 1A 130 and / or entity descriptions 115 from entity description providers 160 are included in the search results. Therefore, query-answer data repository 170 may not have sufficient information to suppress certain query answers that should be suppressed, such as query answer 114 that is substantially duplicated with the content of entity description 115 (which is not substantially duplicated with the content of entity description 115). Figure 1A The query response 114 is shown as is the entity description 115).

[0025] Thus, the service endpoint 120 is configured to dynamically suppress content from the search results when preparing the search results for presentation at the client computer 110 based on the submitted query and the primary search results, query answers, and / or entity descriptions for the query. By suppressing content based on the search query and all content that would be displayed in the search results if not suppressed, the service endpoint 120 may be able to suppress content that is not suppressed by the offline filtering mechanism 171. For example, the service endpoint 120 may include a duplicate content suppression machine 124 that is configured to suppress duplicate content from one or more of the query answers 114 and / or entity descriptions 115 based on redundancy of such content within the search results. The service endpoint 120 may determine what content to suppress dynamically based on a query blacklist provided by the query blacklist repository 140. The service endpoint 120 may also determine what content to suppress dynamically based on one or more rules provided by the rule repository 150. The one or more rules may specify a particular query, site, and / or answer, and the results may be suppressed based on matching the specified query / site / answer. The suppression of undesirable content may improve the service endpoint 120 and / or client computer 110 and / or the ability to dynamically suppress content. Figure 1AFor example, service endpoint 120 may not need to migrate as much data to client computer 110 because suppressed results do not need to be migrated. In addition, the user can more efficiently obtain relevant results in fewer searches with less data migration than if the user had to perform multiple searches to find relevant content, thereby reducing computing and network costs and user effort.

[0026] therefore, Figure 1B A method 1000 is shown for providing search results when content (e.g., query answers and / or entity descriptions) may be suppressed from the search results. The method 1000 may be performed by any suitable computer system, such as from Figure 1A Any suitable combination of devices of the search provider architecture shown, such as service endpoint 120 and / or client computer 110. Although method 1000 is described with respect to suppressing query replies and / or entity description content, method 1000 or substantially similar methods may be applied to suppress any other kind of content from a search, such as to suppress one or more primary search results and / or to suppress different kinds of content from a curated location in the search results. Method 1000 may be at least partially performed by Figure 1A The service endpoint 120 shown may be instantiated by one or more machines. For example, the service endpoint 120 may instantiate a query suppression engine 121, a pattern suppression engine 122, a site suppression engine 123, and / or a duplicate content suppression engine 124. Figure 1A , but alternatively or additionally to service endpoint 120, any other suitable combination of machines (e.g., client computer 110) may instantiate query suppression engine 121, pattern suppression engine 122, site suppression engine 123, and / or duplicate content suppression engine 124 to provide any functionality described herein.

[0027] At 1100, method 1000 includes maintaining a query-reply mapping, e.g. Figure 1A A query-answer mapping is shown in the query-answer data store 170. The query-answer mapping is configured to include, for each of a plurality of expected queries, a corresponding answer to the expected query.

[0028] At 1200, method 1000 includes maintaining a query blacklist comprising a plurality of computer-readable representations of blacklist queries. For example, the query blacklist may include a list of text query strings. Service endpoint 120 is configured to instantiate a query suppression engine 121 configured to evaluate whether a received query matches any query in the blacklist. Matching a received query to a query in the blacklist may be based on an exact text comparison of the received query string to a query string in the blacklist. Alternately or additionally, the query may be matched via "fuzzy" matching (e.g., matching at least to a threshold similarity, such as a threshold proportion of matching terms in the query), probabilistic matching, and / or using a machine learning or natural language processing model. When the received query matches one of the strings in the blacklist query string, service endpoint 120 may suppress a query reply for the query from the search results.

[0029] At 1300, method 1000 includes instantiating a plurality of filter rules for evaluating whether to suppress any candidate query answer. In other words, the plurality of filter rules are instantiated without regard to any particular query answer, and can later be evaluated with respect to any given query answer to determine whether to suppress the given query answer. The plurality of filter rules may include one or both of a pattern rule and / or a site rule. Figure 1A As shown, site and / or mode rules may be maintained in a rules repository 150 .

[0030] The service endpoint 120 is configured to instantiate a pattern suppression engine 122 that is configured to evaluate pattern rules with respect to a query. Pattern rules are rules for suppressing queries based on the textual content of the query and / or the reply. As with query blacklist entries, pattern rules can be used to specify queries for which replies should not be provided in a curated location. Alternately or in addition to matching literal query text, pattern rules can be used to match a query to any suitable pattern specification, which can be used to parse the query to evaluate whether it matches a specified pattern. Non-limiting examples of pattern specifications include formal languages ​​(e.g., regular expressions), patterns for checking whether a query contains a specific string, starts with a specific string, and / or ends with a specific string, patterns for identifying dates, times, names, currencies, etc. in a language / region-independent manner, and the like. Pattern rules can be used to parse a query to find content in the query that can indicate that the query should not be considered a question for which query reply content should be provided in a curated location. Alternately or additionally, pattern rules can be used to parse replies to a query in order to determine whether the reply should be provided in a curated location. As a non-limiting example, a pattern rule may be configured to match one or more profane phrases; thus, replies to queries containing any one or more profane phrases and / or replies containing any one or more profane phrases may be suppressed so they are not displayed in a curated location in the search output. In some examples, pattern rules may be evaluated against queries and / or answers. Alternately or additionally, pattern rules may be specified to be evaluated only against queries or replies, e.g., to suppress replies that include profane content while not suppressing replies to queries when the query itself includes profane content.

[0031] The service endpoint 120 is configured to instantiate a site suppression engine 123 that is configured to evaluate site rules with respect to query replies based on a plurality of site rules loaded, for example, from a rule repository 150. Site rules are rules for suppressing query replies based on metadata associated with a website from which the replies were derived and / or the content of the website. The metadata may include, for example, a site URL, a publication date / time stamp, author information, or any other suitable metadata. Site rules may be specified in a manner similar to pattern rules, such as by a formal language specification, a text containment specification, and the like. Moreover, site rules may be specified to apply to any text data associated with a website (e.g., to apply to the site URL, date / time information, author information, and other metadata) and / or to apply to specific data fields (e.g., a site rule may be specified to apply only to the site URL) to flexibly detect different aspects of a reply that may indicate that the reply is not desirable to be included in search results in a curated / prominent position. As an example, some sites may be associated with low-quality replies and / or duplicate replies that have been provided by other sites. Thus, site rules may match such sites (e.g., by matching URLs) to suppress replies derived from such sites from the search output.

[0032] In some examples, the entries in the query blacklist and / or the filtering rules (e.g., site rules and mode rules) can be based at least in part on user feedback. For example, the search page can be configured to include additional user interface elements to receive user approval and / or disapproval signals about the appropriateness of including a particular response in the search output. As an example, the search output can include a "thumbs up" approval button and a "thumbs down" disapproval button to rate the query responses included in the curated location. As another example, the client computer 110 can be configured to receive verbal feedback from the user in the form of voice audio, and interpret the user's approval / disapproval based on the feedback. For example, the client computer 110 can present the query response in the form of voice audio, and prompt the user with a question (e.g., "Is this helpful?"). Therefore, the user can answer by saying "yes" or "no" (or any other suitable response indicating whether the query response is helpful). Therefore, the client computer 110 can capture voice audio (e.g., using a microphone) and process the captured voice audio to interpret "yes" as approval and "no" as disapproval. In some examples, if a user disapproval signal is received for a query, the query may be added to a query blacklist in response to receiving the user disapproval signal so that replies are not included for the query. In some examples, if a threshold number of user disapproval signals are received for replies generated from the same site, a new site rule may be added to the filter rule list to match the site from which the query replies are derived to suppress results from the site in the future. In some examples, multiple user disapproval signals may be received for one or more queries. Thus, a new pattern rule may be added to the filter rule list, wherein the new pattern rule is configured to match any subsequent received queries consistent with the one or more queries. As an example, the new pattern rule may be configured to match a regular expression for one or more queries. In some examples, the new pattern rule may be configured to match one or more queries while being constrained to match as few unrelated other queries as possible. For example, a pattern may be a regular expression consisting of an alternative text string for each of the one or more queries, so that the regular expression only matches the one or more queries and does not match any other different queries. In some examples, the new pattern rule may be a rule based on matching a string that starts with a string, contains a string, and / or ends with a string, or a sub-pattern common to one or more queries. Determining new pattern rules may be based on any suitable state-of-the-art and / or future string processing, parsing, natural language and / or machine learning techniques.New query blacklist entries, pattern rules, and / or site rules can be added based on any suitable processing of user signals, such as based on data mining user approval and / or disapproval signals using artificial intelligence, machine learning, and / or natural language processing techniques. Changing query blacklists and / or filtering rules based on user signals can be performed for any group of one or more users, such as to perform personalized suppression for a small group of users and / or improve suppression results for a general group of users. In examples where users can indicate disapproval and approval, determining whether to add new rules to suppress query reply content can be based on weighting disapproval versus approval in a group of users.

[0033] At 1400, method 1000 includes receiving a query. The query may be received in any suitable manner, such as as a raw and / or normalized query received at service endpoint 120 from client computer 110 over a computer network. The query may be based on user input, such as input in a search box of a graphical browser 111 presented by client computer 110 and / or input in a spoken dialogue with an intelligent assistant program running on client computer 110. In some examples, the query may be normalized before the query is received, such as at client computer 110. In some examples, the query may be received as a raw query and normalized after receipt, such as at service endpoint 120.

[0034] At 1500, after receiving a query, method 1000 includes suppressing content from the search results based on the query and the search results. At 1510, method 1000 includes operating a query-reply mapping to map the query to a candidate query reply. Although the present description relates to obtaining a single candidate query reply for a query and determining whether to suppress the candidate query reply, the methods described herein are not so limited and may be applied to evaluate suppression of multiple different candidate query replies to a query. For example, instead of operating a query-reply mapping to map a query to a single candidate query reply, obtaining search results may include operating a query-reply mapping to map the query to multiple different candidate replies, and for each candidate reply, evaluating whether to suppress the candidate reply in accordance with the present disclosure. Thus, the search results may include more than one candidate reply (if more than one candidate reply is not suppressed). Alternately or additionally, when more than one candidate reply is not suppressed, the unsuppressed candidate replies may be arranged, filtered, and / or otherwise processed to select a single candidate reply for inclusion in a curated position within the search results. In some examples, suppression of a result entry that is to be shown in a curated position may include removing the result entry from the curated position, such as by demoting the result entry to be shown in a different position without as much prominence. In other examples, suppression of a result entry may include removing the result entry altogether so that it is not shown in the search results at all.

[0035] At 1520, method 1000 includes suppressing the candidate query answer from the search results in the curated location in response to the candidate query answer matching any suppression rules. In an example, determining whether the query answer matches any suppression rules may be performed using the following description of the query answer. Figure 2 At 1530, method 1000 includes returning search results. The search results may include primary search results (e.g. Figure 1A 113). Optionally, in some examples, method 1000 includes identifying a computer-readable entity description based on a query that mentions an entity (e.g., an entity description received from entity description provider 160 and / or an entity description identified in any other suitable manner, such as based on processed data at service endpoint 110). The returned search results are configured to include the candidate query answer in a curated search position of the query answer in response to the candidate query answer not being dynamically suppressed.

[0036] Service endpoint 120 may be configured to dynamically suppress query replies based on matching one or more different suppression rules. Figure 2 A non-limiting example of a method 2000 for determining whether to suppress a query or reply based on a pipeline of suppression rules is shown. Figure 2 As shown, method 2000 includes checking a query and a reply against a plurality of different suppression rules and suppressing the query or reply if it matches any of the suppression rules. Figure 2 Method 2000 is shown with suppression rules 2100 , 2200 , 2300 , and 2400 being evaluated sequentially, but suppressions may be evaluated in any suitable order (eg, in parallel).

[0037] At 2100, method 2000 includes checking whether the computer-readable representation of the received query matches any computer-readable representation of any blacklist query from a query blacklist (e.g., as described above with respect to Figure 1A If the received query does not match the blacklist query, the method 2000 includes suppressing the query at 2540. Otherwise, the method 2000 may include evaluating other suppression rules (e.g., the suppression rules at 2200, 2300, and 2400).

[0038] At 2200, method 2000 includes checking whether the computer-readable representation of the query matches any pattern rule in a filter rule list (eg, received from rule repository 150 and generated by Figure 1AIf the received query does not match the pattern rule, method 2000 includes suppressing the query at 2540. Otherwise, method 2000 may include evaluating other suppression rules (e.g., the suppression rules in 2100, 2300, and 2400).

[0039] At 2300, method 2000 includes checking whether the computer-readable representation of the metadata of the candidate query answer matches a site rule from a list of filtering rules (eg, received from rule repository 150 and generated by Figure 1A If the received query does not match the pattern rule, then method 2000 includes suppressing the candidate query answer at 2540. Otherwise, method 2000 may include evaluating other suppression rules (e.g., the suppression rules in 2100, 2200, and 2400).

[0040] At 2400, method 2000 includes checking whether a candidate query answer is substantially similar to a computer-readable entity description (e.g., when the computer-readable entity description is scheduled for inclusion in search results). Determining substantial similarity can be based on Figure 1A 2400 . If so, method 2000 includes suppressing the candidate query answer at 2540 . Otherwise, method 2000 may include evaluating other suppression rules (e.g., suppression rules in 2100 , 2200 , and 2300 ). Alternately or additionally, at 2400 , method 2000 may include checking whether the candidate query answer is substantially similar to any other portion of the search results, such as whether the candidate query answer is substantially similar to a top result in the primary search results. Evaluating the substantial similarity between the candidate query answer and other result entries may be performed in any suitable manner, such as using text / string matching techniques (e.g., edit distance), natural language processors, and / or machine learning techniques and / or in accordance with method 400 for evaluating similarity between two search content. For example, evaluating the similarity between the candidate query answer and a computer-readable entity description received from entity description provider 160 may include operating a natural language processor to evaluate the natural language similarity of the candidate query answer to the computer-readable entity description. If the candidate query answer is similar to the entity description, method 2000 also includes suppressing the query answer content at 2540 . Otherwise, method 2000 may include evaluating other suppression rules (eg, suppression rules in 2100 , 2200 , and 2400 ).

[0041] If no suppression rules (e.g., in 2100, 2200, 2300 and / or 2400) result in suppression of the query answer content in 2540, then in 2520, the query answer content is not suppressed but is included in the search output in a curated position within the search results with enhanced prominence compared to other result items.

[0042] In some examples, if a candidate query answer is similar to an entity description, one or the other of the candidate query answer or the entity description may be suppressed from the search results, depending on how the search results will be used. Figure 3A An example is shown where no search replies are suppressed. FIG. 3B to FIG. 3C An example is shown where either a candidate query answer or an entity description is suppressed. FIG. 3A to FIG. 3C An example of suppressing results based on similarity between candidate replies and entity descriptions is shown, but results may be suppressed in a similar manner to present desired search result content using any other suppression rule(s), such as based on similarity between candidate replies and entity descriptions. Figure 1A The site rules evaluated by the site suppression engine 123, the pattern rules evaluated by the pattern suppression engine 122, the query matches in the query blacklist evaluated by the query suppression engine 121, and / or the similarity of the content shown.

[0043] exist Figure 3A , a web browser shows results for the search query "Why did Elrond leave Middle-Earth?" The curated content site shows query answers 114 to the question in the search query. Additionally, entity descriptions 115 show descriptions of the person "Elrond" mentioned in the question. Additionally, primary results 113 may indicate (e.g., link to and / or summarize) potentially related content. Figure 3A , the content in query answer 114 is not substantially similar to that in entity description 115. For example, the entity description is a general description of "Elrond", while the query answer is a specific answer to a question about "Elrond", and the corresponding text of the entity description and the query answer do not have very many common words / phrases (e.g., except for "Elrond" and "Middle-earth", the content is substantially different).

[0044] In some examples, a candidate query answer can be dynamically suppressed from search results in a curated location in response to the candidate query answer being substantially similar to a computer-readable entity description. Figure 3BAnother example search with a different query "Who is Elrond?" is shown. In this case, the answer to the question "Who is Elrond?" will include a description of "Elrond" and will therefore be substantially similar to the entity description of "Elrond." Therefore, the search results may be configured to suppress the query answer and present the entity description in a second, different, curated location (in addition to the curated location of the query answer) with enhanced prominence compared to other result entries. In the example, the user may be accustomed to checking the curated location of the entity description to find descriptions of entities (such as people and fictional characters). By presenting the entity description instead of the query answer, the user may find relevant information more easily. In some examples, when the query answer is suppressed due to similarity to the entity description, the entity description may be presented in the curated location of the query answer instead of a different curated location of the entity description. For example, the curated location of the query answer may be even more prominent than the curated location of the entity description (e.g., the curated location of the query answer may be given more space and / or more central placement compared to the curated location of the entity description). Therefore, when the query answer is suppressed, the entity description may be given additional prominence by presenting the entity description in a more prominent curated location (which would otherwise be used for the query answer).

[0045] In other examples, in response to a candidate query answer being substantially similar to the entity description, the entity description may be dynamically suppressed from the search results, and thus the candidate query answer may not be suppressed from the search results. Figure 3C Shown with Figure 3B The same search, but query answers 114 are presented instead of entity descriptions 115. In an example, the search interface may have a curated location for query answers while not having a curated location for entity descriptions. Figure 3B A search in a graphical web browser is depicted, but in some examples, the search results are configured to be output via a speaker as an audio response that includes a spoken narration of one or more search results. Thus, a query response may be more suitable for a spoken narration than other items in the search results (e.g., result items and / or entity descriptions in primary results 113), for example because the query response may be formulated as a natural language response that is directly responsive to the question posed in the query.

[0046] Figure 4A non-limiting example of method 4000 for evaluating the similarity of content of two search results is shown, such as for evaluating substantial similarity between a query reply and an entity description. For example, method 4000 can be used to identify a computer-readable entity description of a query, evaluate whether a candidate query reply is substantially similar to the computer-readable entity description, and based on the evaluation also dynamically suppress one of: 1) the computer-readable entity description or 2) a candidate query reply from a search output. The similarity is evaluated based on the length of the query calculated in 4110 (LQ, e.g., the length of text character tokens in the original query), the length of the query reply content calculated in 4120 (QCL, e.g., the length of text character tokens in the query reply), the length of the longest common subsequence of the query reply and the entity description calculated in 4130 (LCS, e.g., the number of tokens in a sequence, such as natural language word tokens represented as text character tokens and / or indicative of words in a subsequence), and the length of a common token list present in the query reply and the entity description calculated in 4140 (CTL, e.g., the number of common tokens in the common token list).

[0047] At 4210, method 4000 includes determining whether a ratio of a length of a longest common subsequence (LCS) to a length of preprocessed query answer content (QCL) (e.g., LCS / QCL) exceeds a predefined threshold. If LCS / QCL exceeds the predefined threshold, the query answer and the entity description are evaluated to be substantially similar.

[0048] At 4220, method 4000 includes determining whether a ratio of a length of a common token list (CTL) to a length of pre-processed query reply content (QCL) exceeds a predefined threshold (which may be the same or different than the threshold used at 4120). If the CTL / QCL exceeds the predefined threshold, then the query reply and entity description are evaluated to be substantially similar.

[0049] At 4230, method 4000 includes determining whether the length of a common token list (CTL) present in the entity description and the pre-processed query answer content exceeds the length of the query (LQ). If CTL>LQ, then the query answer and entity description are evaluated to be substantially similar.

[0050] The methods and processes described herein can be combined with a computing system of one or more computing devices. In particular, such methods and processes can be implemented as an executable computer application, a network accessible computing service, an application programming interface (API), a library, or a combination of the above and / or other computing resources.

[0051] Figure 5Schematically illustrated is a simplified representation of a computing system 500 configured to provide any and all computing functionality described herein. The computing system 500 may take the form of one or more personal computers, network accessible server computers, tablet computers, home entertainment computers, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phones), virtual / augmented / mixed reality computing devices, wearable computing devices, Internet of Things (IoT) devices, embedded computing devices, and / or other computing devices. For example, the computing system 500 may include any combination of logical subsystems, storage subsystems, and / or other subsystems of one or more of the client computers 110, service endpoints 120, search result providers 130, query blacklist repositories 140, rule repositories 150, entity description providers 160, and / or query-reply data repositories 170.

[0052] The computing system 500 includes a logic subsystem 502 and a storage subsystem 504. The computing system 500 may optionally include an input / output subsystem 506 (e.g., including one or more input devices or sensors and one or more output devices, such as a graphics display and / or an audio mapper), a communication subsystem 508, and / or Figure 5 Other subsystems not shown.

[0053] The logic subsystem 502 includes one or more physical devices configured to execute instructions. For example, the logic subsystem may be configured to execute instructions as part of one or more applications, services, or other logical structures. The logic subsystem may include one or more hardware processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware devices configured to execute hardware or firmware instructions. The processor of the logic subsystem may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. The various components of the logic subsystem may optionally be distributed in two or more separate devices, which may be remotely located and / or configured for coordinated processing. Various aspects of the logic subsystem may be virtualized and executed by a remotely accessible networked computing device configured in a cloud computing configuration.

[0054] The storage subsystem 504 includes one or more physical devices configured to temporarily and / or permanently retain computer information, such as data and instructions, executable by the logic subsystem. When the storage subsystem includes two or more devices, the devices may be collocated and / or remotely located. The storage subsystem 504 may include volatile, nonvolatile, dynamic, static, read / write, read-only, random access, sequential access, location addressable, file addressable, and / or content addressable devices. The storage subsystem 504 may include removable and / or built-in devices. When the logic subsystem executes instructions, the state of the storage subsystem 504 may be transformed, for example, to retain different data.

[0055] Various aspects of logic subsystem 502 and storage subsystem 504 may be integrated together into one or more hardware logic components. For example, such hardware logic components may include program and application specific integrated circuits (PASIC / ASIC), program and application specific standard products (PSSP / ASSP), systems on chips (SOC), and complex programmable logic devices (CPLD).

[0056] The logic subsystem and the storage subsystem can collaborate to instantiate one or more logic machines. As used herein, the term "machine" is used to collectively refer to hardware and any software, instructions, and / or other components that collaborate with such hardware to provide computer functionality. In other words, a "machine" is never an abstract concept and always has a tangible form. A machine can be instantiated by a single computing device, or a machine can include two or more subcomponents instantiated by two or more different computing devices. In some implementations, a machine includes a local component (e.g., a computer service) that collaborates with a remote component (e.g., a cloud computing service). The software and / or other instructions that give a particular machine its functionality can optionally be saved as an unexecuted module on a suitable storage device. Non-limiting examples of machines that can be instantiated by the computing system 500 according to the present disclosure include a browser 111, a query suppression machine 121, a pattern suppression machine 122, a site suppression machine 123, and / or a duplicate content suppression machine 124.

[0057] Machines according to the present disclosure may be implemented using any suitable combination of state-of-the-art and / or future machine learning (ML), artificial intelligence (AI), and / or natural language processing (NLP) techniques. Non-limiting examples of techniques that may be incorporated into an implementation of one or more machines include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or videos, temporal convolutional neural networks for processing audio signals and / or natural language sentences, and / or any other suitable convolutional neural network configured to convolve and pool features in one or more temporal and / or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines, and / or neural random access memories), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI knowledge bases), and / or neural language processing techniques (e.g., tokenization, stemming, constituency and / or dependency parsing and / or intent recognition, segmentation models, and / or super-segmentation models (e.g., hidden dynamic models)).

[0058] In some examples, the methods and processes described herein can be implemented using one or more differentiable functions, where the gradients of the differentiable functions can be calculated and / or estimated with respect to the inputs and / or outputs of the differentiable functions (e.g., with respect to training data and / or with respect to an objective function). Such methods and processes can be determined, at least in part, by a set of trainable parameters. Thus, the trainable parameters of a particular method or process can be adjusted by any suitable training procedure to continually improve the operation of the method or process. For example, machine learning training techniques can be used to mine user approval / disapproval signals, such as to determine whether to add new query blacklist entries, site rules, and / or pattern rules for suppressing query replies.

[0059] Non-limiting examples of training procedures for adjusting trainable parameters include supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on categories derived from unsupervised clustering methods), reinforcement learning (e.g., feedback-based deep Q-learning), and / or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consistency), contextual bandit methods, maximum likelihood methods, and / or expectation maximization. In some examples, multiple methods, processes, and / or components of the systems described herein can be trained simultaneously with respect to an objective function that measures the performance of the joint operation of multiple components (e.g., with respect to reinforced feedback and / or with respect to labeled training data). Training multiple methods, processes, and / or components simultaneously can improve such joint operation. In some examples, one or more methods, processes, and / or components can be trained independently of other components (e.g., offline training on historical data).

[0060] The methods and processes disclosed herein can be configured to give control of any privacy and / or potentially sensitive data to the user and / or any other person. Whenever data is stored, accessed and / or processed, the data can be disposed of according to privacy and / or security standards. When user data is collected, the user or other stakeholders can specify how the data is used and / or stored. Whenever user data is collected for any purpose, the user who owns the data should be notified, and the user data should only be collected when the user provides affirmative consent. If data is to be collected, it can and should be collected with maximum respect for user privacy. If the data is to be released to be accessed by anyone other than the user or used in any decision-making process, the user's consent can be collected before using and / or releasing the data. The user can choose to join and / or opt out of data collection at any time. After the data has been collected, the user can issue a command to delete the data and / or restrict access to the data. Optionally, all potentially sensitive data can be encrypted and / or further protect user privacy when anonymized. The user can specify part of the data, metadata, or statistical information / results of the processed data to be released to other parties, such as for further processing. Private and / or confidential data may be kept completely private, such as only temporarily decrypted for processing, or only decrypted for processing on the user device and otherwise stored in encrypted form. The user may retain and control the encryption keys for the encrypted data. Alternatively or additionally, the user may designate a trusted third party to retain and control the encryption keys for the encrypted data, such as to provide the user with access to the data in accordance with an appropriate authentication protocol.

[0061] When the methods and processes described herein incorporate ML and / or AI components, the ML and / or AI components may make decisions based at least in part on the training of the components on training data. Thus, the ML and / or AI components can and should be trained on different representative data sets that include sufficient relevant data for different users and / or user groups. In particular, the training data sets for different individuals and groups should be inclusive so that when the ML and / or AI components are trained, the performance of the user experience for the users and / or user groups is improved.

[0062] For example, a dialog system according to the present disclosure may be trained to interact with different groups of users using language models trained to work well for those groups based on any other characteristics of the groups' language, dialect, accent, and / or speaking style.

[0063] ML and / or AI components may additionally be trained to make decisions so as to minimize possible biases with individuals and / or groups. For example, when AI systems are used to evaluate any qualitative and / or quantitative information about individuals or groups, they may be trained to be invariant to differences between individuals or groups that are not intended to be measured by the qualitative and / or quantitative evaluations, e.g., so that any decision is not influenced in an unconscious manner by differences between individuals and groups.

[0064] ML and / or AI components can and should be designed to provide as much context as possible about how they operate, so that implementers of ML and / or AI systems can be held accountable for decisions / evaluations made by the system. For example, ML and / or AI systems should have replicable behavior, e.g., when they make pseudo-random decisions, random seeds should be used and recorded to be able to replicate the decisions later. As another example, data used to train and / or test ML and / or AI systems should be curated and maintained to support future investigations of the behavior of ML and / or AI systems with respect to the data. Furthermore, ML and / or AI systems can and should be constantly monitored to identify possible biases, errors, and / or unintended results.

[0065] When included, the input / output subsystem 506 can be used to present a visual representation of the data held by the storage subsystem 504. The visual representation can take the form of a graphical user interface (GUI). The input / output subsystem 506 can include one or more display devices that utilize virtually any type of technique. In some implementations, the input / output subsystem 506 can include one or more virtual reality, augmented reality, or mixed reality displays. The input / output subsystem 506 can be used to visually present content, such as a browser 111 and search results displayed in a page of the browser 111. The input / output subsystem 506 can include one or more microphones and / or speaker devices configured to receive and / or output audio. In some examples, a microphone device can be used to receive voice audio input, which can be processed (e.g., using natural language processing and / or machine learning techniques) to receive user queries, determine user intent, etc. For example, voice audio input can be processed to control the browser 111. For example, voice audio input can be processed to identify a user query of a search engine, such as in addition to or in place of the user input via the text in the search bar 112. In some examples, a speaker device may be used to output voice audio, such as to provide information to a user, to interact with a user in a spoken conversation, etc. In some examples, the browser 111 may be configured to present content in the form of voice audio. For example, the browser 111 may present search results by outputting voice audio indicating the result entry for each result entry in the search results. For example, when the browser 111 presents search results including a query reply and a plurality of other result entries, the browser 111 may output voice audio narrating the query reply, and output other voice audio listing the title and / or summary of each of the plurality of other result entries.

[0066] When included, the input / output subsystem may also include one or more input devices or interface with one or more input devices. The input device may include a sensor device or a user input device. Examples of user input devices include a keyboard, a mouse, a touch screen, or a game controller. In some embodiments, the input subsystem may include a selected natural user input (NUI) component portion or interface with it. Such component portions may be integrated or peripheral, and the transduction and / or processing of input actions may be on-board or off-board. Example NUI component portions may include microphones for speech and / or voice recognition; infrared, color, stereo and / or depth cameras for machine vision and / or gesture recognition; head trackers, eye trackers, accelerometers and / or gyroscopes for motion detection and / or intent recognition.

[0067] When included, the communication subsystem 508 can be configured to communicatively couple the computing system 500 with one or more other computing devices. The communication subsystem 508 can include wired and / or wireless communication devices compatible with one or more different communication protocols. The communication subsystem can be configured to communicate via a personal area network, a local area network, and / or a wide area network.

[0068] The language model can utilize vocabulary features to guide the sampling / search of words for speech recognition. For example, the language model can be defined at least in part by the statistical distribution of words or other vocabulary features. For example, the language model can be defined by the statistical distribution of n-grams, thereby defining the transition probability between candidate words based on vocabulary statistics. The language model can also be based on any other appropriate statistical features and / or the results of processing statistical features using one or more machine learning and / or statistical algorithms (e.g., confidence values ​​generated by such processing). In some examples, the statistical model can constrain which words can be recognized for an audio signal, such as based on the assumption that the words in the audio signal are from a particular vocabulary.

[0069] Alternatively or additionally, the language model can be based on one or more neural networks previously trained to represent audio input and words in a shared latent space, such as a vector space learned by one or more audio and / or word models (e.g., wav2letter and / or word2vec). Thus, finding candidate words can include searching the shared latent space based on vectors encoded by the audio model for the audio input to find candidate word vectors for decoding using the word model. The shared latent space can be used to evaluate, for one or more candidate words, a confidence that the candidate word is a feature in the speech audio.

[0070] The language model can be used together with an acoustic model, which is configured to evaluate the following confidence for candidate words and audio signals: the confidence is that the candidate word is included in the speech audio in the audio signal based on the acoustic features of the word (e.g., Mel-frequency cepstral coefficients, formants, etc.). Optionally, in some examples, the language model can be incorporated into the acoustic model (e.g., the evaluation and / or training of the language model can be based on the acoustic model). The acoustic model defines the mapping between the acoustic signal and the basic sound unit (such as phoneme) based on the labeled speech audio. The acoustic model can be based on any suitable combination of the most advanced or future machine learning (ML) and / or artificial intelligence (AI) models, such as: deep neural networks (e.g., long short-term memory, temporal convolutional neural networks, restricted Boltzmann machines, deep belief networks), hidden Markov models (HMMs), conditional random fields (CRFs) and / or Markov random fields, Gaussian mixture models, and / or other graphical models (e.g., deep Bayesian networks). The audio signal to be processed using the acoustic model can be preprocessed in any suitable manner, such as encoding at any suitable sampling rate, Fourier transform, bandpass filter, etc. The acoustic model can be trained to identify the mapping between the acoustic signal and the sound unit based on training with labeled audio data. For example, the acoustic model can be trained based on labeled audio data including speech audio and corrected text to learn the mapping between the speech audio signal and the sound unit represented by the corrected text. Therefore, the acoustic model can be continuously improved to improve its practicality for correctly identifying speech audio.

[0071] In some examples, the language model can incorporate any suitable graphical model, such as a hidden Markov model (HMM) or a conditional random field (CRF), in addition to a statistical model, a neural network, and / or an acoustic model. The graphical model can utilize statistical features (e.g., transition probabilities) and / or confidence values ​​to determine the probability of recognizing a word given the speech audio and / or other words recognized to date. Thus, the graphical model can utilize statistical features, previously trained machine learning models, and / or acoustic models to define transition probabilities between states represented in the graphical model.

[0072] In an example, a method for determining whether to dynamically suppress a candidate query answer that is indicated for inclusion in search results includes: maintaining a query-answer mapping that includes, for each of a plurality of expected queries, a corresponding answer to the expected query; instantiating a plurality of filtering rules for evaluating suppression of any candidate query answer, the plurality of filtering rules including one or both of pattern rules and site rules; maintaining a query blacklist including a plurality of computer-readable representations of blacklisted queries; receiving a query; identifying a computer-readable entity description of the query; upon receiving the query: operating the query-answer mapping to obtain a candidate query answer previously associated with the query; in response to any of the following, selecting from the search results a candidate query answer that is previously associated with the query; and selecting from the search results a candidate query answer that is previously associated with the query. Dynamically suppress a candidate query answer in a curated location with enhanced prominence relative to multiple other result entries within a search result: a computer-readable representation of the query matches a computer-readable representation of a blacklisted query from a query blacklist; 1) the computer-readable representation of the query matches a pattern rule from a filtering rule list; 2) a computer-readable representation of metadata of the candidate query answer matches a site rule from the filtering rule list; or 3) the candidate query answer is substantially similar to the computer-readable entity description; and return search results that include the computer-readable entity description and, in response to the candidate query answer not being dynamically suppressed, also include the candidate query answer in the curated location. In this example or any other example, the method further includes: receiving a user disapproval signal regarding the query, and adding the query to the query blacklist in response to receiving the user disapproval signal. In this example or any other example, the method further includes: receiving multiple user disapproval signals regarding one or more queries and adding a pattern rule to the filtering rule list, the pattern rule being configured to match any subsequently received query consistent with the one or more queries. In this example or any other example, evaluating a candidate query response for substantial similarity to a computer-readable entity description includes determining that A) the ratio of the length of the longest common subsequence between the entity description and a preprocessed response to B) the length of the preprocessed response exceeds a predefined threshold. In this example or any other example, evaluating a candidate query response for substantial similarity to a computer-readable entity description includes determining that A) the ratio of the length of a common token list between the entity description and the preprocessed response to B) the length of the preprocessed response exceeds a predefined threshold. In this example or any other example, evaluating a candidate query response for substantial similarity to a computer-readable entity description includes determining that A) the length of the common token list between the entity description and the preprocessed response is equal to B) the length of the query. In this example or any other example, evaluating a candidate query response for substantial similarity to a computer-readable entity description includes operating a natural language processor to evaluate natural language similarity between the candidate query response and the computer-readable entity description.In this example or any other example, the site rule is configured to match a website based on one or more of: a timestamp of the website, a URL of the website, metadata of the website, and content of the website. In this example or any other example, the pattern rule is configured to match a query based on parsing the query according to a regular expression specification. In this example or any other example, the pattern rule is configured to match a query based on one or more of: a query starts with a text pattern, the query ends with a text pattern, and the query contains a text pattern.

[0073] In an example, a method for determining whether to dynamically suppress a candidate query answer that is designated for inclusion in search results includes: instantiating multiple filtering rules for evaluating suppression of any candidate query answer, including one or both of pattern rules and site rules; receiving a query; upon receiving the query: obtaining candidate query answers previously associated with the query; dynamically suppressing the candidate query answer from a curated location with enhanced prominence relative to multiple other result entries within the search results in response to any of the following: 1) a computer-readable representation of the query matches a pattern rule from a list of filtering rules; or 2) a computer-readable representation of metadata for the candidate query answer matches a site rule from a list of filtering rules; and returning search results that include the candidate query answer in the curated location in response to the candidate query answer not being dynamically suppressed. In this example or any other example, the method also includes: identifying a computer-readable entity description of the query; evaluating whether the candidate query answer is substantially similar to the computer-readable entity description; and based on the evaluation, also dynamically suppressing one of the following: 1) the computer-readable entity description or 2) the candidate query answer from the search results. In this example or any other example, a candidate query response is dynamically suppressed from a curated position within the search results in response to the candidate query response being substantially similar to the computer-readable entity description. In this example or any other example, the search results are configured to present the computer-readable entity description in a second, different, curated position within the search results with enhanced prominence relative to multiple other result entries. In this example or any other example, the computer-readable entity description is dynamically suppressed from a curated position within the search results in response to the candidate query response being substantially similar to the computer-readable entity description, and the candidate query response is not suppressed from the search results. In this example or any other example, the search results are configured to output, via a speaker, an audio response as a spoken statement that includes the candidate query response.

[0074] In an example, a method for determining whether to dynamically suppress a candidate query answer that is indicated for inclusion in search results includes: instantiating multiple filtering rules for evaluating suppression of any candidate query answer, including one or both of pattern rules and site rules; receiving a first query from a search application configured for query answering; after receiving the first query: obtaining a candidate query answer previously associated with the first query; dynamically suppressing the candidate query answer from a curated position with enhanced prominence relative to multiple other result items within the search results in response to any of the following: 1) a computer-readable representation of an entity matches a pattern rule from a list of filtering rules; or 2) a computer-readable representation of metadata of the candidate query answer matches a site rule from a list of filtering rules; returning search results that include the candidate query answer in the curated position in response to the candidate query answer not being dynamically suppressed; and selecting a candidate query answer from a graph application Receiving a second query, the graphical application having a privileged display area for visually presenting an entity description, wherein the second query is the same as the first query; upon receiving the second query: identifying a computer-readable entity description for the second query; obtaining a candidate query answer previously associated with the first query and the second query; dynamically suppressing the candidate query answer from a curated position within the search results with enhanced prominence relative to a plurality of other result entries in response to any of the following: 1) the computer-readable representation of the query matches a pattern rule from a list of filtering rules; 2) the computer-readable representation of the metadata of the candidate query answer matches a site rule from a list of filtering rules; or 3) the query answer is substantially similar to the computer-readable entity description; and returning search results that include the entity description designated for visual presentation in the privileged display area and also include the candidate query answer in response to the candidate query answer not being dynamically suppressed. In this example or any other example, the candidate query answer is dynamically suppressed from a curated position within the search results in response to the candidate query answer being substantially similar to the computer-readable entity description. In this example or any other example, the search results for the first query are configured to output, via a speaker, an audio response as a spoken statement including the candidate query answer. In this or any other example, the computer-readable entity description is dynamically suppressed from a curated position within the search results in response to the candidate query answer being substantially similar to the computer-readable entity description, and the candidate query answer is not suppressed from the search results.

[0075] It should be understood that the configuration and / or method described herein are exemplary in nature, and these specific embodiments or examples should not be considered as restrictive, because many variations are possible. The specific routine or method described herein can represent one or more of any number of processing strategies. Like this, the various actions illustrated and / or described can be performed or omitted in the order illustrated and / or described, in other orders, in parallel. Likewise, the order of the above process can be changed.

[0076] The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations, and other features, functions, acts, and / or properties disclosed herein, as well as any and all equivalents thereof.

Claims

1. A method for suppressing a query response designated for inclusion in a search engine results page (SERP), the method comprising: Receive inquiries; identifying, based on the query, a query answer for inclusion in the SERP, wherein the query is mapped to the query answer in a mapping database that maps a plurality of queries to a plurality of query answers; Based on the query, identifying an entity description for inclusion in the SERP, wherein the entity description includes information about an entity related to the query; identifying search results for inclusion in the SERP based on the query; The query response is suppressed from a curated position with enhanced prominence within the SERP, wherein the query response is suppressed based on: 1) The query is included in the query blacklist; 2) the query corresponds to a predefined pattern; 3) the metadata assigned to the query answer matches a predefined rule; or Those 4) the query reply includes information that is duplicate of the information in the entity description; and The SERP is returned, the SERP including the entity description and the search result, but failing to include the query answer in the curated position because the query answer is suppressed.

2. The method according to claim 1, further comprising: A user disapproval signal is received regarding the query, and the query is added to the query blacklist in response to receiving the user disapproval signal.

3. The method according to claim 1, further comprising: receiving a plurality of user disapproval signals regarding a plurality of queries; as well as The predefined pattern is identified based on the plurality of queries and the plurality of user disapproval signals.

4. A method according to claim 1, wherein the information included in the query reply is determined to be a duplicate of the information in the entity description based on A) the ratio of the length of the longest common subsequence between the information in the entity description and the information included in the query reply to B) the length of the information in the query reply.

5. A method according to claim 1, wherein the information included in the query reply is determined to be a duplicate of the information in the entity description based on: A) the length of a common token list between the information in the entity description and the information included in the query reply to B) the length of the information included in the query reply.

6. A method according to claim 1, wherein the information included in the query reply is determined to be a duplicate of the information in the entity description based on the length of the common token list between the information in the entity description and the information included in the query reply being equal to the length of the query.

7. A method according to claim 1, wherein the information included in the query response is determined to be a duplicate of the information in the entity description based on the operation of a natural language processor that evaluates the natural language similarity between the information included in the query response and the information in the entity description.

8. The method of claim 1, wherein the rule is configured to identify a website corresponding to the query reply based on one or more of: a timestamp of the website, a URL of the website, metadata of the website, or content of the website.

9. The method of claim 1, wherein the query is determined to correspond to the predefined pattern based on parsing the query according to a regular expression specification.

10. The method of claim 1, wherein the predefined pattern comprises a text pattern, and further wherein the query is determined to correspond to the predefined pattern based on the query comprising the text pattern.

11. The method of claim 1, wherein the query reply is suppressed based on the query reply including information that is duplicated with the information in the entity description.

12. A method for suppressing a query response designated for inclusion in a search engine results page (SERP), the method comprising: receiving a query, wherein the query reply is linked to the query; in response to receiving the query, obtaining the query answer as the query answer is linked to the query, wherein the query answer is obtained for inclusion in the SERP; After obtaining the query answer, suppressing the query answer from a curated position with enhanced prominence within the SERP based on: 1) the query has a predefined pattern; or 2) triggering predefined rules based on the metadata of the query answer; as well as The SERP is returned, wherein the SERP includes search results identified based on the query but fails to include the query answer in the curated location.

13. The method according to claim 12, further comprising: receiving a second query, wherein a second query reply is linked to the second query; in response to receiving the second query, obtaining the second query answer as the second query answer is linked to the second query, wherein the second query answer is obtained for inclusion in a second SERP; identifying an entity description for inclusion in the second SERP in response to receiving the second query, wherein the entity description is identified based on the query, and further wherein the entity description includes information specific to an entity related to the query; determining that the information for the entity in the entity description duplicates information included in the second query reply; suppressing the second query answer from being included in the second SERP due to the information for the entity in the entity description being duplicated with the information included in the second query answer; as well as The second SERP is returned, wherein the second SERP includes the entity description but fails to include the second query answer.

14. The method of claim 13, wherein the second query reply is suppressed from a curated position within the SERP.

15. The method of claim 14, wherein the entity description in the second SERP returned is located in a second, different, curated location in the second SERP returned. The method of claim 12 , wherein the search results include entity descriptions.

17. A computing system comprising: processor; as well as A memory storing instructions, wherein when the instructions are executed by the processor, the processor performs actions, the actions comprising: receiving a query, wherein the query is related to an entity; identifying an instant reply based on the query, wherein the instant reply is mapped to the query in a database that maps a plurality of instant replies to a plurality of queries, and further wherein the instant reply is identified for inclusion in a search engine results page (SERP) to be presented to a user who submitted the query; identifying an entity description for the entity, wherein the entity description is identified based on the query, wherein the entity description includes information about the entity, and further wherein the entity description is identified for inclusion in the SERP; determining that the information about the entity in the entity description duplicates the information included in the immediate reply; suppressing one of the instant reply or the entity description from being included in the SERP based on the information about the entity in the entity description being determined to be duplicate of the information included in the instant reply; and The SERP is returned after suppressing the one of the instant reply or the entity description, wherein the returned SERP includes one of the instant reply or the entity description, but not both the instant reply and the entity description.

18. The computing system of claim 17, wherein the immediate reply is suppressed.

19. The computing system of claim 17, wherein the entity description is suppressed.

20. The computing system of claim 17, the actions further comprising identifying a ranked list of search results based on the query, wherein the SERP includes the ranked list of search results.