Method and system for search result annotation

Abstract

A flexible annotation framework normalizes auxiliary information from different sources, ranks the information for a single search result, and provides a lightweight or full display of the auxiliary information in an annotation for the search result. Thus, the annotation displays information that is not typically part of the details included in the search results. An example method includes, for at least one item in a search results page, identifying at least one annotation of a first annotation type in an annotation data store that references the item, identifying at least one annotation of a second annotation type in an annotation data store that references the item, ranking the annotations of the first annotation type and the annotations of the second annotation type, and providing the highest ranked annotation as part of the search result for the item in the search results page.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 62 / 705,413, filed on June 25, 2020, entitled “Search Results Annotation,” the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This application relates to search result annotation. Background Art

[0004] The search engine typically searches for items in an item repository that satisfy the query terms. The search engine aggregates information about the responsive items and provides search results to the query requester. The search engine may search major item repositories (e.g., documents available on the Internet, applications provided in an application store, items for sale in a market, etc.) or a combination of major repositories. Each search result typically includes a link and a snippet describing basic details about the responsive item. The link can take the user to a page with more details about the item, the original document, etc. The search engine ranks the responsive items and provides a search results page that lists several responsive items that are typically ranked highest and some basic details about the items included in the search results page. Summary of the invention

[0005] Embodiments provide annotations for search results in search results pages. The annotations represent auxiliary data about search results that are not part of the basic project details that are usually included in search results. The annotations can connect projects to secondary repositories and / or data sources that help give context to the location of the project in the search results page. In other words, some annotations can provide reasons or insights for the ranking of responsive search terms. Some annotations can also provide tangential data, such as from secondary sources or primary sources, that can enable users to more effectively evaluate the projects in the search results page. Some embodiments may include a framework that enables annotations to be obtained from one or more external sources. Embodiments process various types of annotations representing different auxiliary data. The disclosed framework can enable annotations from various sources and with various types to be scored. All annotations obtained by the machine learning scoring algorithm are sorted, and one or more annotations are provided to be displayed as part of the search results for the project. Therefore, embodiments can display annotations from multiple secondary sources. Embodiments may support various interfaces that provide less detail (i.e., "light annotations") for typical space-constrained displays of smartphones or wearable devices, and more detail (i.e., "full annotations") for conventional displays (e.g., laptops, smart TVs, or personal computers). In some embodiments, the user interface may initially display lightweight annotations and be configured to display full annotations in response to a gesture or other command from the user. In some embodiments, full annotations may be provided in highlighted results (e.g., in an entity view).

[0006] The embodiment provides a framework that enables secondary systems / teams (e.g., for different information repositories) to contribute annotations. The embodiment provides flexibility in the content of the annotations. The embodiment handles various annotation data sources, and the data source can be expanded or contracted with little code change. In addition, the embodiment can provide a fair scoring mechanism that relies on the quality of the generated annotations and prevents a specific annotation source from unfairly inflating the annotation score. The embodiment provides a solution for scoring different types of annotations, so different types of annotations can be mixed and sorted on search result pages and entity views (e.g., knowledge panels, knowledge boxes, etc.). Some search engines use entity views in the following cases: when the top search results correspond to entities, for example, when the top results correspond to a single entity with sufficient confidence. The embodiment provides useful information, i.e., auxiliary information, to users browsing search results. Utilizing the annotations disclosed herein leads to more intelligent electrodes, thereby saving computing resources and user time. Utilizing the annotations disclosed herein can also lead to fewer subsequent queries. Annotations can be contextual, adjusted according to a specific query. Annotations can be generally useful regardless of the specific query. Annotations can be configured to make sense at rest, for example, without the need for click access, thereby further saving computing resources. Certain annotations may have click targets to allow users to drill down deeper into the rationale.

[0007] One or more of the embodiments of the subject matter described herein may be implemented to achieve one or more of the following advantages. As an example, an embodiment may integrate different types of annotations from different systems to provide a flexible, dynamic environment for providing auxiliary data. An embodiment also saves computing resources by increasing the user's confidence in the presented search results, thereby reducing the likelihood that the user will submit multiple queries to obtain the answers sought. For example, an embodiment may provide auxiliary information from different sources that supports the ranking of a particular item in a search result list, which may increase the user's confidence in the presented results. Thus, an embodiment relates to a framework for providing a novel interface for associating items in a primary repository with a secondary repository that provides insights into the location of the item in the search results page. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

[0009] Figure 1 An example system is shown in accordance with some implementations.

[0010] Figure 2 An example data graph illustrating a diverse annotation ranking pipeline, according to an embodiment.

[0011] Figure 3 An example annotation framework data structure is shown in accordance with some implementations.

[0012] Figures 4A-4D Example mention and review annotation types are shown for a search results page in accordance with some implementations.

[0013] Figure 5A and Figure 5B Another review annotation type for a search results page is shown in accordance with some implementations.

[0014] Figures 6A-6C Shown are example top-pick and description annotation types for a search results page in accordance with some implementations.

[0015] Fig. 7A and Figure 7B Shows a version change annotation type for a search results page according to some implementations.

[0016] Figures 8A-8C Example multimedia and editorial rating annotation types are shown for a search results page in accordance with some implementations.

[0017] Fig. 9 Query context annotation types for a search results page are shown in accordance with some implementations.

[0018] Fig.10 A flow diagram is shown of a process for generating dynamic annotations for a search results page in accordance with some implementations.

[0019] Fig.11 An example of a computer device that can be used to implement the described techniques is shown.

[0020] Fig.12 An example of a distributed computer device that can be used to implement the described techniques is shown.

[0021] Like reference numbers in the various drawings indicate like elements. DETAILED DESCRIPTION

[0022] Embodiments include a flexible annotation framework that normalizes auxiliary information from different sources, scores and ranks annotations for a single search result, and provides options for lightweight or full display of information in annotations. The disclosed embodiments operate in a search engine environment in which a query engine generates a search results page. In some embodiments, the search engine environment can be a market environment, for example, a market environment that provides items (e.g., applications, services, products) for sale, download, ordering, etc. However, the disclosed technology can be applied to any search domain that identifies items in response to queries from a primary corpus or repository and provides annotations from other (i.e., secondary) sources. Examples of other search environments include travel-related searches, media searches, new home searches, entity searches, etc. Embodiments provide a common framework for obtaining auxiliary information from other sources and / or from primary repositories. The auxiliary information can be any information related to the item, but does not include information that is typically provided in the search results for the item. For example, a market search engine typically displays images, titles, prices, overall reviews (star ratings), and / or shipping information in search results. The auxiliary information displayed in the annotation does not include such primary information. The disclosed annotation framework is configured to accept annotations from external systems and generate annotations from internal system sources. Implementations enable display of lightweight and full annotations in user interfaces. Implementations can provide zero, one, or multiple annotations for a search term.

[0023] Figure 1 1 is a block diagram of a search service system 100 according to an example implementation. The system 100 may be used to implement a search service that provides annotations for some search results in a search result page. Figure 1 The depiction of system 100 in is described as a market search service, although implementations are not limited thereto and the disclosed techniques may be applied to other settings.

[0024] Search service system 100 may include search engine 110. Search engine 110 may be a computing device that takes the form of a variety of different devices, such as a standard server, a group of such servers, or a rack server system. In some embodiments, search engine 110 may be a single system that shares components (e.g., processors and memory). In addition, search engine 110 may be implemented in a personal computer (e.g., a laptop computer). Search engine 110 may be an example of a computer device 1200, such as Fig.12 The search engine 110 may include one or more servers that receive queries from requesters (eg, client 170) and provide search results to the requesters.

[0025] The search engine 110 may include one or more processors 113 formed in a substrate, an operating system (not shown), and one or more computer memories 114. The computer memory 114 may represent any kind of memory (e.g., RAM, flash memory, cache, disk, tape, etc.). The memory 114 may represent a variety of memories. In some embodiments (not shown), the memory 114 may include an external storage device, such as a memory that is physically remote but accessible to the search engine 110. The search engine 110 may include one or more modules or engines representing specially programmed software. For example, the search engine may include a query system 120 that enables the search engine 110 to receive and respond to queries.

[0026] The query system 120 itself may include modules. For example, the query system 120 may include a query engine 125 and an indexing engine 129. The indexing engine 129 may be configured to update the index 140. For example, the indexing engine 129 may add items to the index 140, update items in the index 140, and delete items from the index 140. In some embodiments, the indexing engine 129 may work with one or more crawlers. The crawler searches for items accessible via the Internet and returns the content (including metadata) of the items. In some embodiments, the crawler may be configured to crawl only certain types of content or crawl certain domains. The indexing engine 129 may use content and / or metadata to generate and update the index 140.

[0027] The query system 120 may also include a query engine 125. The query engine 125 may receive a query from a requester (e.g., client 170), analyze the query to determine how to search the index 140, and initiate a search of the index 140. A user may submit a query to the search engine 110, such as a word, a phrase, a word list, an image, a recording, etc. The search engine 110, in particular the query engine 125, uses one or more indexes (e.g., index 140) to identify items returned in response to a query. Items returned to the query engine 125 in response to a query may also be referred to as responsive items. The query engine 125 may generate search results for some or all responsive items. As used herein, search results are summary data related to responsive items. The data provided as part of the search results themselves are considered the primary data of the item. The primary data comes from a primary repository, such as index 140, and typically represents less than all data available in the repository for the item. The search results may include links. The links may initiate some actions related to responsive items. For example, a search result link may take a user to a web page, such as a web page that offers the item for purchase or download, may take a user to an interface, file, or web page that provides additional details about the item, etc. The search results for an item may also include a small image or icon related to the item. The search results for an item may include a brief description of the item or information extracted from the item, also known as a snippet. The search results may include a summary of the ratings of the item (e.g., number of starts, rate, etc.). Items such as products, services, or hotels may include the price of the item. The search results may include other summary information related to the item and / or describing the item as part of the main information included in the search results.

[0028] The query engine 125 can sort the responsive items. The sorting can include applying one or more sorting signals to the responsive items. The sorting signals can include many factors. Once sorted, many search engines organize the responsive items into pages. Each page may have a preset number of responsive items. Thus, for example, the first ten responsive items can be returned as the first page, the next ten responsive items can be on the second page, and so on. The user can use links, controls, or over-scrolling to move between pages, for example, to request the last page, the next page, the previous page, etc. The number of responsive items returned in the first page can depend on the size of the visible area. For example, a mobile device with a 6-inch screen may contain fewer responsive items in the first page than a laptop computer with a 17-inch screen.

[0029] When looking for responsive items, the query system 120 may be responsible for searching one or more indexes, collectively referred to as index 140. Index 140 may be, for example, an inverted index that associates terms, phrases, and / or n-grams with items. Items may be files, including files that describe products. Items may be products, books, applications, and / or services offered in stores. Items may be hotels, restaurants, flights, and the like related to travel. Items may be entities. Terms, phrases, and / or n-grams may come from item descriptions. Terms, phrases, and / or n-grams may come from reviews about items. Terms, phrases, and / or n-grams may come from files (web pages, PDFs, and the like) that describe or mention items. Index 140 may also include a knowledge base or knowledge graph. In a knowledge graph, entities (items) are modeled as nodes, and facts about entities are modeled as attributes or labeled relationships between entities. For example, an item may be linked to one or more reviews in a graph relationship. Index 140 may also include indexes for products, attributes, flights, applications, and the like. Where index 140 represents several different repositories, one repository may be considered the primary repository searched by query engine 125 and index engine 129. The remaining repositories may be secondary repositories used by query system 120, e.g., to rank or provide additional details about responsive items selected by a user.

[0030] The index 140 may be stored on a tangible computer-readable storage device, such as a disk, flash memory, cache, or a combination thereof, which is configured to store data in a semi-permanent or non-transient form. In some embodiments, the index 140 may be stored in a combination of various memories. The query engine 125 may obtain responsive items from the index 140, sort the responsive items, generate search results for at least some of the responsive items, and provide the search results to a query requester, such as a client 170.

[0031] In addition to finding responsive items, the query system 120 can also obtain annotations for one or more of the responsive items. The annotations of the responsive items represent auxiliary data related to the items. Some annotations are independent of the query. Some annotations are query-dependent. Some annotations can be provided by the query system 120. Some annotations may be provided by other systems. Annotations can belong to different types. For example, an annotation can be a mention annotation based on an article or other document that mentions the item, such as an article reviewing a product or a news article about the product. The mention annotation can be in the form of a snippet, such as selecting certain text from an article. The mention annotation can be in the form of a summary, such as with the title of the article. As another example, the annotation can be based on editorial ratings. Such ratings can be collected from certain domains, organizations, or publications. The editorial rating annotation can include a rating from one source or can include multiple ratings from multiple sources. As another example, the annotation can be a top ranked annotation. The top ranked annotation indicates that the item is ranked in a list of "top X" items in a certain category, such as "hot smartphones", "hot beach resorts", or "hot father's day gifts". As another example, the annotation may be a review annotation. The review annotation is generated based on a user's review of the item. The review annotation may be a positive / negative annotation that lists the most frequently mentioned positive aspects of the item and / or the most frequently mentioned negative aspects of the item. The review annotation may be a query-based snippet review. Most queries, especially for market searches, include words that are not resolved and / or not used to find matching items. The query-based review annotation may find and display matches for those words in user reviews. The query-based review annotation may highlight (e.g., bold, underline, italicize, use a different color, etc.) matching words in their full form.

[0032] As another example, the annotation may be a new version annotation. The new version annotation may indicate that the item is new, which may give the item a higher importance and therefore a higher relevance. As another example, the annotation may be a query context annotation. The query context annotation uses query terms related to the measurement and emphasizes the product attributes that meet those terms. In some embodiments, the query context annotation may display a graphical distribution of all terms in the same category and indicate where the results appear in the space. As another example, the annotation may be a description annotation. Although most search result pages do not include a long description of the item, the description annotation includes such a description. In some embodiments, the annotation may highlight or select (if the description is too long) a portion that is particularly relevant to the query. As another example, the annotation may be a multimedia annotation. The multimedia annotation may point to a video or audio file from an authoritative source (a reputable publisher) related to the item. The multimedia file may be a review of the item, a demonstration of the item, etc. As another example, the annotation may be a prior query annotation. If the user has agreed to retain historical data, the prior query annotation is an indication that the user has previously researched the item.

[0033] Some annotation types may be pre-computed offline and stored, for example, in offline annotations 130. Offline annotations may be indexed by item (e.g., by item identifier). Annotations may be pushed to the query system 120 from other systems. Thus, for example, mentions, top rated, editorial comments, and other annotation types may be pushed from an Internet or news search system to a product query system and stored in offline annotations 130. In some embodiments, query-dependent annotation types may be computed offline and stored in offline annotations 130. In some embodiments, query-dependent annotation types, such as query-based comment snippets, descriptions, or contextual annotations, may be generated at query time. In some embodiments, all annotations from other systems are stored in offline annotations 130. In some embodiments, some of the annotation types generated by the query system 120 may be computed offline and stored in offline annotations 130. In some embodiments, some of the annotation types generated by the query system 120 may be computed at query time. Although illustrated as part of the search system 110, in some embodiments, one or more annotation types stored in the offline annotations 130 are remote from the search engine 110, but may be accessed by the search engine 110.

[0034] In some embodiments, the query system 120 may include a search log (not shown). The search log may include an aggregated search log, aggregated data collected from a query, or other aggregated data about a previously processed query. In some embodiments, the search log may be generated by the search engine 110 in the normal process of generating search results. In some embodiments, the search log may be used as a source of annotation types.

[0035] The search engine 110 may include one or more annotators 135. In some embodiments, the annotator 135 generates a type of annotation. The annotator 135 may be remote from the search engine 110, but may be connected to the search engine 110 in a communication manner. The annotator 135 is configured to generate an annotation payload from a secondary repository or from data in a primary repository that is not usually included in the search results. In some embodiments, the annotator 135 may score the annotation. In some embodiments, the annotator 135 may generate features that can be used to score the annotation relative to other annotations. In some embodiments, the annotator 135 may score the annotation using a machine-learned model (e.g., a regression algorithm, a neural network, etc.). The machine-learned model may be trained by providing a human rater with an item and one or more annotations of the same type for the item. The human rater may score each annotation to indicate the usefulness of this annotation. These labeled annotations may be used to train one or more models to score specific types of annotations. In some embodiments, one or more of the annotators 135 may exist on an external system that pushes the normalized annotation information to the search engine 110. The external system may include multiple annotators 135 (eg, for different types of annotations).

[0036] The query system 120 includes an annotator engine 127. The annotator engine 127 obtains annotations of items from annotators 135 and aggregates the annotations. The annotator engine 127 includes a machine-learned model, such as NG3. The annotator engine 127 is configured to take each of the scoring annotations (or scoring features) produced by the annotator 135 that created the annotations and rank the different annotations relative to each other. The machine-learned model can be trained by presenting items with multiple different types of annotations to human raters and letting the human raters rank the annotation types. The model can be trained using this labeled data. If a new type of annotation is added to the system, the model can be retrained to include the new type of annotation. Therefore, a new annotation type can be added to the query system 120 by retraining the annotator engine 127. In some embodiments, if it is found that the annotator 135 exaggerates the annotation score or otherwise games the system, or if new information indicates that a specific annotation type is not very useful, the annotator engine 127 can be trained to penalize the annotations of that annotation type.

[0037] Once the annotator engine 127 has ranked the annotations for an item, the annotations may be provided for presentation. The query system 120 may provide the top-ranked annotations. The query system 120 may provide a set of x top-ranked annotations. The query system 120 may provide all the annotations ranked for an item. The query system may provide lightweight or full annotations for initial display. When screen space is at a premium, lightweight annotations may be selected. Annotations may be displayed in a carousel, a swipe-and-grow interface, an expandable row, etc. Not every search result in a search results page may have annotations. In some embodiments, only the top-ranked responsive items may have annotations. In some embodiments, if an item lacks the lowest-rated / ranked annotation, no annotations may be displayed for that query. The query system 120 may provide the information needed to display annotations as part of the search results page for a query.

[0038] The query system 120 can communicate with (multiple) clients 170 via a network 160. For example, the network 160 can be the Internet, a cellular network, a wired or wireless local area network (LAN), a wide area network (WAN), etc. The network 160 may represent a variety of network types. The query system 120 can communicate with the client 170 and other domains (not shown) via the network 160 to send data to / from them.

[0039] Client 170 may be an example of computer device 100, such as Fig.11As shown. For example, the client 170 may be a personal computer, a mobile phone, a tablet device, a laptop computer, a wearable device, a smart TV, etc. The client 170 may include one or more processors 173 formed on a substrate, and the one or more processors 173 are configured to execute one or more machine executable instructions or software, firmware fragments, or a combination thereof. The processor 173 may be semiconductor-based, that is, the processor may include semiconductor materials capable of executing digital logic. The client 170 may also include one or more computer memories 174. The memory 174 (e.g., main memory) may be configured to temporarily, permanently, semi-permanently store one or more pieces of data, or a combination thereof. The memory 174 may include any type of storage device that stores information in a format that can be read and / or executed by one or more processors 173. The memory 174 may store applications, modules, and / or engines that perform certain operations when executed by one or more processors 173. In some embodiments, the application, module, or engine may be stored in an external storage device and loaded into the memory 174.

[0040] Applications may include any number of applications configured to be executed on client 170, such as operating systems, messaging applications, shopping applications, editing applications, search assistants, maps, etc. Specifically, applications include browsers 175. Browsers 175 are operable to receive web page code (e.g., HTML, JavaScript, etc.) and present web pages to be displayed to users of client 170. Therefore, client 170 includes a display device provided with a viewport. As used herein, a viewport is a polygonal area showing the content presented by a browser. Conventionally, a viewport is rectangular. The size of the viewport depends on the device, for example, the viewport on a smart phone is smaller than the viewport of a personal computer or a tablet device. The content of the presented web page may not be fully adapted to the viewport. In this case, the user can scroll, for example, perform a scrolling action to bring content into the viewport and move the content out of the viewport. A scrolling action is any input that a browser 175 (possibly in combination with the operating system of the client 170) identifies as performing scrolling to move content into the viewport. Example scrolling actions are driving a scroll wheel on a mouse, a tap and drag action, a sliding action (e.g., using a finger or stylus), driving a scroll bar in a browser window, etc. The browser 175 thus displays scrollable content (e.g., a rendered web page) and the viewport determines which portion of the content the user of the client 170 can actually see, or in other words, which portion of the content is visible. Expanding a lightweight annotation can scroll content previously displayed in the viewport out of the viewport.

[0041] Client 170 may include one or more input devices, such as a touch screen, keyboard, mouse, pointer, microphone, camera, one or more physical buttons, etc. The input device may initiate input events, such as scrolling, link selection, cursor movement. Client 170 may also include a communication device operable to send and receive data from other computing devices (e.g., another client, server, search engine 110, etc.) via one or more networks (e.g., network 160). Figure 1 The configuration in represents one example configuration and implementations may also include other configurations.

[0042] In addition to the above, controls may be provided to the user of the client device 170 to allow the user to select whether and when the system, program, or feature described herein may enable the collection of user information (e.g., information about the user's activities, user preferences, or the user's current location), and whether content or communications may be sent to the user from a server (e.g., search engine 110). In addition, certain data may be processed in one or more ways before being stored or used to remove personally identifiable information. For example, the user's identity may be processed so that the user's personally identifiable information cannot be determined, or the user's geographic location may be generalized (e.g., city, zip code, or state level) when obtaining location information so that the user's specific location cannot be determined. Therefore, the user of the client 170 can control what information is collected about the user, how the information is used, and what information is provided to the user and / or to the search engine 110.

[0043] Figure 2 An example data graph showing a diverse annotation ranking pipeline according to an embodiment. Figure 2 In the example shown, the system includes five different types of annotations. Annotator 135(a) generates editorial snippet annotations. Annotator 135(b) generates review pro / con annotations. Annotator 135(c) generates description annotations. Annotator 135(d) generates editorial rating annotations. Annotator 135(e) generates top ranking annotations. Annotator 135(f) generates query-based review snippet annotations. Each annotator 135 may generate annotations in a normalized format. Figure 3An example normalized annotation format 300 is shown according to some embodiments. The annotation may identify an object to which it is applied, such as a file identifier, a product identifier, an entity representation, etc. The identifier may match an identifier of an item in a primary index. The annotation may also include an indication of the annotation type. The payload includes data to be included in the annotation of that type. The payload depends on the annotation type. In some embodiments, the payload may include data items for lightweight annotations and additional data items for full annotations. In some embodiments, the payload may include instructions (e.g., code) for how to display the data items. In some embodiments, the payload may include a target. The target may be a file from which the annotation is generated. The annotation may include a score. The score is set by the annotator that generates the annotation. The annotation may include a feature set. The feature set may be used to score the annotation against annotations of the same type and / or other types.

[0044] exist Figure 2 In the example of , each annotator 135 provides a score for the annotation. The annotator engine 127 takes all the annotations and ranks them relative to each other using a machine-learned ranking algorithm. The output of the annotator engine 127 is a ranked list of annotations 250. The example system can select one, two, or three of the annotations in the ranked list 150 to display as part of the search results.

[0045] Figure 4A-4D Example mention and comment annotation types are shown for search result page 400 according to some implementations. Figure 4A In the example of , a user has entered query 405 "headphones". Search results 400 for query 405 include result 410 and result 420. Both result 410 and result 420 include annotations in the form of expandable rows. Control 415 can be selected to expand the annotation of search result 410, and control 425 can be selected to expand the annotation of search result 420. Figure 4B 4, the user has selected control 415. The selection of control 415 causes annotation 430 to be displayed in a carousel including annotation 435. Both annotation 430 and annotation 435 are associated with the item of search result 410. Annotation 430 is an example of a mention annotation. Specifically, annotation 430 is in the form of a snippet, e.g., some text of an editorial review that selected this item. Annotation 435 is another mention of the item in summary form, e.g., the title of an article containing comments from a different editor.

[0046] Figure 4CIs the result of a user selecting control 425. The selection of control 425 results in an expanded row that displays annotation 440 and annotation 445 in a horizontally scrollable carousel area. In some embodiments, only annotation 440 may be displayed in the expanded row. In some embodiments, annotation 440 may display the complete annotation. Other annotations may be included in the carousel area and accessed by scrolling. Annotation 440 is an example of a positive / negative comment annotation type. Positive / negative comments may be generated from user reviews of a product. The positive / negative comment annotation may list the most frequently mentioned positive aspects of the item and / or the most frequently mentioned negative aspects of the item. In some embodiments, the positive / negative comment may include one or more click targets. Clicking on the target may trigger a query. This query may search for comments for the item. Clicking on the target may open a comment view. Annotation 445 is a comment snippet. The comment snippet may list comments that include one of the most frequently mentioned advantages (or disadvantages) of the product. Figure 4D Annotation 450 shown in FIG. 4 is another example of a review segment annotation.

[0047] Figure 5A and Figure 5B Another type of review annotation for search result 500 is shown in accordance with some embodiments. Figure 5A In the example shown, the user enters query 505 for "headphones for running". Search results 500 for query 505 include result 510 and result 520. Both result 510 and result 520 include annotations in the form of expandable rows. Control 515 can be selected to expand the annotation for search result 510, and control 525 can be selected to expand the annotation for search result 420. Figure 5A In FIG. 5 , the user selects control 525, causing annotation 530 and annotation 535 to be displayed in the carousel area. Annotation 530 is an example of a review annotation based on the query. Figure 5A In the example in , the annotation includes a snippet of a user review that mentions running with headphones. A user mentioning running in a product review may cause a particular product to be ranked higher. The annotation provides a reason for the ranking of the search results 520 and can cause a reduction in subsequent queries because the user has confidence that the results match the intent. Rare / uncommon query terms in the main repository (for example, there are not many items related to running or running compared to those related to headphones) can be used to identify appropriate review snippets. Determining query terms used to describe entities (adjectives for items) can be used to identify appropriate snippets. A portion of a review snippet that mentions a context specific to the query can be highlighted (bold, italic, underlined, different font color, etc.). The review snippet annotation can include a click target. Clicking the target can initiate a query for the user's review. Figure 5BHorizontal scrolling of the carousel is shown, which brings annotation 535 fully into the viewport and annotation 540 partially into the viewport. Annotation 535 is another example of a review mention summary.

[0048] Figure 6A-6C Example preferred and descriptive annotation types for search results 600 are shown in accordance with some implementations. Fig. 6A In the example in FIG. 6 , a user has entered a query 605 for “headphones”. Search results 600 for query 605 include result 610 and result 620. Result 610 and result 620 both include annotations in the form of slide-to-expand interface elements. In the slide-to-expand interface, a user can select an annotation and then slide and drag downward to expand the annotation interface. The slide of annotation 615 results in the following: Figure 6B 6 shows an expanded interface shown in . Annotation 615 represents an example of a top-ranked annotation type. A top-ranked annotation indicates that this item is ranked in a "top X" list of items in a category, such as "popular smartphones," "popular beach vacations," or "popular Father's Day gifts." Annotation 615 represents a lightweight version of an annotation. Annotation 615' in FIG. 6 shows a full annotation that may correspond to annotation 615. Full annotation 615' may include additional context and information not included in the lightweight version of annotation 615.

[0049] The expanded interface also includes annotations 630 and 635 that are partially displayed within the viewport. Annotation 630 is an example of an annotation type that mentions a snippet. Annotation 635 is an example of an annotation that mentions a summary. These additional annotations are optional and may not be displayed with annotation 615'. Additional annotations may also be included in a scrollable carousel area. Fig. 6A Annotation 625 shows an example description annotation. The description annotation includes a full / expanded description of the item. The description annotation can be used in situations where the item description is not included in the main search result information. The description annotation can be contextualized to select an appropriate fragment from a longer description. The description annotation can include a click target. The click target can link to a file with a full description of the item, such as a product order page, a hotel reservation page, a page providing a source for the description, etc. Annotation 635 is an example of a lightweight description annotation. Figure 6C An example of a full description annotation 635' is shown that may correspond to annotation 635. Annotation 635' may be displayed after a swipe or short drag of annotation 635. In some implementations, annotation 635' may be the only annotation displayed within the viewport. Figure 6C In the example of FIG. 6 , the viewport includes a carousel area with additional annotations 640 and 645 .

[0050] Fig. 7A and Figure 7BA version of search result 700 is shown changing annotation types according to some embodiments. Fig. 7A In the example in , a user entered the query 705 "iPhone 10". The search results 700 for query 705 include result 710 and may include other results. Result 710 includes note 715. Note 715 is an example of a version change note type. The version change note type provides an indication to the user that the item has been replaced / updated to an updated version. The version change note may include the name or other indication of the subsequent version. The version change note may include the release date of the updated version. The version change note may include a click target. Clicking the target may link to a file describing the replacement. Clicking the target may initiate a query for an updated version of the item. Note 715 represents an example of a lightweight version of a version change note. Figure 7B An expanded annotation interface is shown, for example, resulting from the sliding of annotation 715. Annotation 715' represents an example of a complete updated version of the annotation. Annotations 725 and 730 may also be displayed in Figure 7B . Annotation 725 is an example of editing a rating annotation. Annotation 730 is an example of a top-ranked annotation.

[0051] Figure 8A-8C Example multimedia and editorial rating annotation types are shown for search results 800 according to some implementations. Fig. 8A In the example in , a user entered query 805 "latest iPhone". Search result 800 for query 805 includes result 810 and result 820. Result 810 includes annotation 815. Annotation 815 is an example of a new release annotation type. The new release annotation type can provide an annotation that the product is a new model, thereby giving it higher importance and therefore higher relevance. Annotation 815 can represent a lightweight annotation. Annotation 825 is an example of a media annotation. A media annotation refers to a video that explains, rates, or otherwise relates to an item. In some embodiments, a media annotation can be associated with a curated source, such as a company or news organization that sells the item. Media annotations can include media thumbnails or titles. Media annotations can include click targets. Click targets can link to media files.

[0052] Figure 8B815. The expanded annotation interface includes annotation 815', which can be an example of a complete version of annotation 815. The new product release annotation can include a release date. The new product release annotation can include a click target. The click target can be a file (such as a web page, press release, etc.) announcing the release of this new product. The expansion interface also includes a partial view of annotation 83O. Annotation 830 is partially displayed within the viewport and is an example of an edit rating annotation. The edit rating annotation can include one rating from one source, or can include multiple ratings from multiple sources. In some embodiments, the sources can be curated. If more than one annotation is displayed in the expanded annotation interface, the user can scroll through the annotations. Figure 8C The expanded annotation interface is shown when the annotation 830 has been fully scrolled into the viewport. As shown, the edit rating annotation can provide the rating of the item and the source of the rating. The edit rating annotation can include the product rating and the highest rating. The edit rating annotation can include the source of the rating. The edit rating annotation can include a click target. The click target can be the source of the rating, for example, a web page with the details behind the rating.

[0053] Fig. 9 900 shows a query-context annotation type for search results 900 according to some embodiments. Fig. 9 In the example in FIG. 1 , a user enters query 905 “long skateboards”. Search results 900 for query 905 include result 910 and result 920. Result 910 includes annotation 915. Annotation 915 is an example of a query context annotation type. The query context annotation type may use query terms related to measurement and emphasize product attributes that satisfy those terms. Fig. 9 In the example of , annotation 915 includes product attributes indicating board length and item type, both of which match the description "long" in query 905. In some implementations, the query context annotation can display a graphical distribution. The graphical distribution can illustrate the distribution of the measured attribute across the category and can indicate where the item falls in the distribution.

[0054] Implementations are not limited to the described example interfaces, annotations, user interactions, or usability. Annotations may be presented in other interfaces and may include other annotation types not specifically disclosed.

[0055] Fig.10 1 is a flowchart of a process 1000 for generating dynamic annotations for a search result page according to some embodiments. The process 1000 may be performed by a search service system, such as Figure 1System 100. Process 1000 may be a process performed during a browsing session. Process 1000 may begin in response to receiving a query from a requester, such as from a client device (1005). The system may obtain and rank items from a primary repository that is considered responsive to the query, or in other words, obtain and rank a collection of responsive items (1010). The system may assign responsive items to pages, such as a first page, a second page. The search system may generate search results for each responsive item assigned to the first page. The search system may also obtain annotations for at least some of the responsive items assigned to the first page (1015). The responsive items assigned to the first page may have no suggested content, for example, if the item is not associated with any pre-computed annotations or any dynamically generated annotations. The annotation may have an annotation type score (1020). The score is generated by an annotator that generates the annotation. The score may be pushed using annotations from an external system. Such annotations are pre-computed for the item. The score may be dynamically generated by the search system, for example, using auxiliary information of the item from information in the primary repository. Dynamic annotations are still scored by the annotator. In some embodiments, the annotator extracts features for scoring rather than generating a score (1020'). The annotation score and annotation type are used to sort the (multiple) annotations associated with the responsiveness project (1025). Alternatively, the extracted annotation features and annotation type are used to sort the (multiple) annotations associated with the responsiveness project (1025'). The system uses the sorting to provide one or more of the annotations for the responsiveness project (1030). In some embodiments, if the top-ranked annotation does not meet the minimum score, the annotation is not provided. In some embodiments, only the top-ranked annotations are provided. In some embodiments, all annotations that meet the minimum score are provided. In some embodiments, all annotations are provided and the sorting is used only to determine the order of the annotations. Process 1000 then ends.

[0056] Fig.11 An example of a general-purpose computer device 1100 is shown, which can be used as Figure 1 The search engine and / or client 170 in the embodiment of the present invention can be used with the techniques described herein. The computer device 1100 is intended to provide various example forms of computing devices, such as laptop computers, desktop computers, workstations, personal digital assistants, cellular phones, smart phones, tablet devices, televisions, servers, and other computing devices, including wearable devices. The components, connections and relationships between components, and functions of the components shown herein are only examples and are not meant to limit the embodiments of the present invention described and / or claimed herein.

[0057] The computing device 1100 includes a processor 1102, a memory 1104, a storage device 1106, and an expansion port 1110 connected via an interface 1108. In some embodiments, the computing device 100 may include a transceiver 1146, a communication interface 1144, and a GPS (Global Positioning System) receiver module 1148, as well as other components connected via the interface 1108, such as one or more cameras, a touch sensor, a keyboard, etc. The device 1100 can communicate wirelessly via the communication interface 1144, wherein the communication interface 1144 may include a digital signal processing circuit when necessary. Each of the components 1102, 1104, 1106, 1108, 1110, 1140, 1144, 1146, and 1148 may be mounted on a common motherboard or in other appropriate ways.

[0058] The processor 1102 can process instructions for execution within the computing device 1100, including instructions stored in the memory 1104 or the storage device 1106 to display graphical information of the GUI on an external input / output device (e.g., a display 1116). The display 1116 can be a monitor or a flat touch screen display. In some embodiments, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used in combination as appropriate. In addition, multiple computing devices 1100 can be connected, each device providing a portion of the necessary operations (e.g., as a server group, a group of blade servers, or a multi-processor system).

[0059] The memory 1104 stores information within the computing device 1100. In one embodiment, the memory 1104 is one or more volatile memory units. In another embodiment, the memory 1104 is one or more non-volatile memory units. The memory 1104 may also be another form of computer-readable medium, such as a magnetic disk or an optical disk. In some embodiments, the memory 1104 may include an expansion memory provided by an expansion interface.

[0060] Storage device 1106 can provide mass storage for computing device 1100. In one embodiment, storage device 1106 can be or include computer readable media, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device, or a device array, including devices in a storage area network or other configuration. A computer program product can be tangibly implemented in these computer readable media. The computer program product can also include instructions that perform one or more methods (such as the above methods) when executed. A computer or machine readable medium is a storage device, such as memory 1104, storage device 1106, or memory on processor 1102.

[0061] The interface 1108 can be a high-speed controller that manages bandwidth-intensive operations of the computing device 1100, or a low-speed controller that manages less bandwidth-intensive operations, or a combination of these controllers. An external interface 1140 can be provided to enable short-range communications between the device 1100 and other devices. In some embodiments, the controller 408 can be coupled to the storage device 1106 and the expansion port 1114. The expansion port, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, one or more cameras, or a device connected to a network, such as a switch or router, for example, through a network adapter.

[0062] The computing device 1100 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 1130, or implemented multiple times in a group of such servers. The computing device can be implemented as a part of a rack server system. In addition, the computing device can be implemented in computing devices such as laptop computers 1132, personal computers 1134, or tablet devices / smart phones 1136. The entire system can be composed of multiple computing devices 1100 that communicate with each other. Other configurations are also possible.

[0063] Fig.11 An example of a general purpose computer device 1100 is shown, which may be Figure 1 The search engine 110 of the present invention may be used with the techniques described herein. The computing device 1100 is intended to represent various example forms of large data processing devices, such as servers, blade servers, data centers, mainframes, and other large computing devices. The computing device 1100 may be a distributed system having multiple processors, possibly including network attached storage nodes interconnected by one or more communication networks. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit the embodiments of the invention described and / or claimed herein.

[0064] The distributed computing system 1100 may include any number of computing devices 1180. The computing devices 1180 may include servers or rack servers, mainframes, etc. communicating via a local or wide area network, dedicated optical links, modems, bridges, routers, switches, wired or wireless networks, etc.

[0065] In some embodiments, each computing device may include multiple racks. For example, computing device 1180a includes multiple racks 1158a-1158n. Each rack may include one or more processors, such as processors 1152a-1152n and 1162a-1162n. The processors may include data processors, network attached storage devices, and other computer control devices. In some embodiments, one processor may operate as a master processor and control scheduling and data distribution tasks. The processors may be interconnected through one or more rack switches 1158, and the one or more racks may be connected through a switch 1178. The switch 1178 may handle communications between multiple connected computing devices 1100.

[0066] Each rack may include memory, such as memory 1154 and memory 1164, and storage devices, such as 1156 and 1166. Storage devices 1156 and 1166 may provide mass storage and may include volatile or non-volatile storage, such as a network attached disk, a floppy disk, a hard disk, an optical disk, a tape, a flash memory, or other similar solid-state storage device, or an array of devices, including devices in a storage area network or other configuration. Memory 1156 or 1166 may be shared between multiple processors, multiple racks, or multiple computing devices, and may include a computer-readable medium that stores instructions that can be executed by one or more processors. Memory 1154 and 1164 may include, for example, one or more volatile memory units, one or more non-volatile memory units, and / or other forms of computer-readable media, such as a disk or optical disk, flash memory, cache, random access memory (RAM), read-only memory (ROM), and combinations thereof. Memory (such as memory 1154) may also be shared between processors 1152a-1152n. Data structures (eg, indexes) may be stored, for example, between the memory 1156 and the storage 1154. The computing device 1100 may include other components not shown, such as a controller, a bus, an input / output device, a communication module, and the like.

[0067] The entire system may be composed of multiple computing devices 1100 that communicate with each other. For example, device 1180a may communicate with devices 1180b, 1180c, and 1180d, and these devices may be collectively referred to as search engine 110. As another example, Figure 1 The search engine 110 of the system may include two or more computing devices 1100. Some computing devices may be geographically close to each other, while other computing devices may be geographically far away. The layout of the system 1100 is only an example, and the system may adopt other layouts or configurations.

[0068] According to certain aspects of the present disclosure, a method includes, for at least one item in a search results page, identifying at least one annotation of a first annotation type in an annotation data store that references the item; identifying at least one annotation of a second annotation type in an annotation data store that references the item; ranking the annotations of the first annotation type and the annotations of the second annotation type; and providing the highest ranked annotation as part of the search results for the item in the search results page.

[0069] These and other aspects may include, alone or in combination, one or more of the following. For example, the first annotation type may be a query-dependent annotation. As another example, the first annotation type may be received from an external system. As another example, the annotation data store may receive annotations in a normalized format from an external system. In some such embodiments, the normalized format may include an item identifier, a payload, and a score or feature for sorting different types of annotations. As another example, the first annotation type may be related to user reviews of an item. As another example, the first annotation may include information that is not available in a repository used to generate the first search result. In some such embodiments, the first annotation may be pushed from a system remote from the system that generates the search results page to the system that generates the search results page. As another example, a machine-learned scoring algorithm may be trained to sort different types of annotations to perform sorting.

[0070] According to certain aspects, a system includes at least one processor and an offline annotation data store storing annotations in a normalized format, each annotation being one of a plurality of annotation types, the normalized format including an item identifier. The system may also include a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations. The operations may include identifying annotations in the offline annotation data store corresponding to items in a search results page; ranking the identified annotations, and providing at least the highest ranked annotation as part of the search results page.

[0071] These and other aspects may include, alone or in combination, one or more of the following. For example, the highest ranked annotation may initially be displayed as a lightweight annotation. As another example, the annotation may be associated with an editorial file that mentions the item. As another example, the item may be a product and at least one annotation relates to a newer version of the product. As another example, at least one annotation may be associated with positive and negative content from user reviews of the item. As another example, the operation may also include determining that a term in a query for which a search results page is generated is an adjective for the item; identifying user reviews of items associated with the adjective; and generating at least one annotation that includes at least one of the identified user reviews. At least one annotation may be included in the ranking.

[0072] In another aspect, a tangible computer-readable storage medium has recorded and embodied thereon instructions that, when executed by one or more processors of a computer system, cause the computer system to perform any of the above-described aspects or operations.

[0073] Various embodiments may include being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system that includes: at least one programmable processor that may be special purpose or general purpose, the at least one programmable processor coupled to receive data and instructions from a storage system and to send data and instructions to a storage system; at least one input device; and at least one output device.

[0074] These computer programs (also referred to as programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any non-transitory computer program product, apparatus and / or device (e.g., magnetic disk, optical disk, memory (including read access memory), programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor.

[0075] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser for a user to interact with an implementation of the systems and techniques described herein), or any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0076] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship between a client and a server is generated by computer programs running on the respective computers and having a client-server relationship to each other.

[0077] Several embodiments have been described herein. However, various modifications may be made without departing from the spirit and scope of the invention. Furthermore, the logic flows depicted in the accompanying drawings do not require the particular order or sequential order shown to achieve the desired results. Furthermore, other steps may be added to the described flows, or some of the steps may be eliminated, and other components may be added or removed from the described systems. Accordingly, other embodiments are also within the scope of the appended claims.

Claims

1. A method for annotating search results, include: In response to the query, identifying items as part of search results displayed in a search results page; determining that a term in the query for which the search results page was generated is an adjective of the item; identifying a first annotation of a first annotation type in a first annotation data store that references the item, the first annotation being independent of the query; identifying a second annotation of a second annotation type in a second annotation data store that references the item, the second annotation being independent of the query and the annotation of the second annotation type having a different payload type than the annotation of the first annotation type, wherein at least one of the first annotation and the second annotation is a user review of the item associated with the adjective; sorting the first annotations to generate a first sorting and sorting the second annotations to generate a second sorting; determining that the first ranking is higher than the second ranking; and displaying a first annotation in the search results page, Wherein the item is included in the search results page without reference to a first annotation data store. 2 . The method of claim 1 , wherein annotations of a first annotation type are received from a first external system and annotations of a second annotation type are received from a second external system. 3 . The method of claim 1 , wherein the first annotation data store and the second annotation data store receive the first annotation and the second annotation, respectively, from an external system in a normalized format. 4 . The method of claim 3 , wherein the normalized format includes a project identifier, a payload depending on the annotation type, and a score assigned by the external system.

5. The method of claim 3, wherein the normalized format includes an item identifier, a payload that depends on the annotation type, and features for sorting annotations of different types. The method of claim 1 , wherein the term is identified for the search result without using the adjective for the term.

7. The method of claim 1, wherein the first annotation is pushed to the system generating the search results page from a system remote from the system generating the search results page.

8. The method of claim 1 , wherein ranking the first annotations to generate a first ranking further comprises executing a machine learning scoring algorithm trained to rank annotations of different types, and ranking the second annotations to generate a second ranking further comprises executing the machine learning scoring algorithm.

9. A system for annotating search results, include: at least one processor; an offline annotation data store storing annotations in a normalized format, each annotation being one of a plurality of annotation categories, each annotation category having a type of payload dependent on the annotation category, the normalized format comprising an item identifier and the payload, and the plurality of annotation categories being query independent; and a memory storing instructions that, when executed by the at least one processor, cause the system to: determining that a term in the query for which a search result page was generated is an adjective of an item; identifying, by including the item in the search results page without reference to the offline annotation data store, from the offline annotation data store a first annotation having a first annotation category and a second annotation having a second annotation category corresponding to the item as part of the search results displayed in the search results page, wherein at least one of the first annotation and the second annotation is a user review of the item associated with the adjective; sorting the first annotations to generate a first sorting and sorting the second annotations to generate a second sorting; determining that the first ranking is higher than the second ranking; and A first annotation is provided for display in the search results page.

10. The system of claim 9, wherein the first annotation is initially displayed as a lightweight annotation.

11. The system of claim 10, wherein the lightweight annotation is configured to display the full annotation in response to receipt of a gesture or other intent from a user.

12. The system of claim 9, wherein the item is a product and the first annotation is associated with an updated version of the product.

13. The system of claim 9, wherein the first annotation is a user comment related to positive content and negative content from user comments on the item.

14. The system according to claim 9, in, At least one of the first annotation or the second annotation is associated with an editorial document that mentions the item.

15. The system according to claim 9, in, The normalized format includes the ratings pushed from the external system along with the annotations.

16. The system of claim 9, wherein the normalized format includes annotation features for sorting.

17. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause a computing system to perform operations, the operations include: In response to the query, identifying items as part of search results displayed in a search results page; determining that a term in the query for which the search results page was generated is an adjective of the item; identifying a first annotation of a first annotation type in a first annotation data store that references the item, the first annotation being independent of the query; identifying a second annotation of a second annotation type in a second annotation data store that references the item, the second annotation being independent of the query and the annotation of the second annotation type having a different payload type than the annotation of the first annotation type, wherein at least one of the first annotation and the second annotation is a user review of the item associated with the adjective; sorting the first annotations to generate a first sorting and sorting the second annotations to generate a second sorting; determining that the first ranking is higher than the second ranking; and displaying a first annotation in the search results page, Wherein the item is included in the search results page without reference to a first annotation data store.

18. The non-transitory computer readable medium storing instructions according to claim 17, in, The first annotation is initially displayed as a lightweight annotation that is configured to display the full annotation in response to receipt of a gesture or other intent from the user.

19. The non-transitory computer readable medium storing instructions according to claim 17, in, At least one of the first annotation or the second annotation is related to an editorial document that mentions the item.

20. The non-transitory computer readable medium storing instructions according to claim 17, in, The first annotation data store and the second annotation data store receive first annotations and second annotations, respectively, from an external system in a normalized format.

Images