Assisted search of non-document items

Abstract

The disclosed embodiments provide a streamlined, assisted search process for surfacing items from a database that enables guided exploratory searching. For example, the system may receive selected content from a client device and use a search converter to determine a first object and a second object for the selected content. The system may generate a first result by performing a first search of items in the database using the first object as a query, and may generate a second result by performing a second search of items in the database using the second object as a query. The system may select a first set of the first results based on the relevance of the first object to the selected content, and may select a second set of the second results based on the relevance of the second object to the selected content. The system may provide a combined search result including the first set and the second set.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of and claims priority to U.S. Patent Application No. 18 / 057,427, filed November 21, 2022, entitled "Assisted Searching of Non-Document Items," which in turn claims priority to U.S. Provisional Patent Application No. 63 / 266,205, filed December 30, 2021, entitled "Assisted Searching of Non-Document Items," the disclosures of both of which are incorporated by reference herein in their entireties. [Background technology]

[0002] background Currently, to search for non-document items in databases such as browser extensions, mobile applications, web applications, catalog / inventory items, etc., a user must (1) determine and provide relevant keywords, (2) scroll through the search results to identify whether the item the user wishes to find is listed, and (3) if not, repeat steps (1) and (2) until a solution is reached or the user gives up. Summary of the Invention

[0003] overview The disclosed embodiments provide a streamlined, assisted search process for surfacing items from a database that enables guided exploratory searches. The disclosed system includes an interface that allows a user to trigger a search from selected content. For example, the interface may allow a user to select content and drag and drop it onto a control (e.g., an actionable icon / button), which may interpret the drop as a query request and, for example, search a data store of individual items based on the selected content. As another example, a user may select content and then activate (e.g., click, select) a control to trigger a smart search of the selected content. Selectable content can be text, links, and / or images. The disclosed system may intelligently select keywords for the search based on an analysis of the content and identify objects suggested by the content. The objects represent keywords, topics, entities, etc., that can serve as queries for a database of items. The disclosed system can initiate a search of the data store using one, two, or more of the identified objects as queries. The content analysis may include translating objects identified in the content or suggested by the content into relevant (customized for the database) keywords for the database. A search for two or more objects (two or more queries) may be performed in parallel. The search may return the top-ranked items for each query. A novel parallel search results interface presents the top-ranked items from the search grouped by query based on the salience and confidence of the objects found and their corresponding results. [Brief explanation of the drawings]

[0004] [Figure 1]1 illustrates content displayed on a web page and a selected content search control according to some implementations. [Figure 2A] 2A-2C illustrate various content selections that trigger assisted searching of selected content of the web page of FIG. 1 according to various implementations. [Figure 2B] 2A-2C illustrate various content selections that trigger assisted searching of selected content of the web page of FIG. 1 according to various implementations. [Figure 2C] 2A-2C illustrate various content selections that trigger assisted searching of selected content of the web page of FIG. 1 according to various implementations. [Figure 2D] 2A-2C illustrate various content selections that trigger assisted searching of selected content of the web page of FIG. 1 according to various implementations. [Figure 3A] 10A-10C illustrate various assisted search result interfaces according to various implementations. [Figure 3B] 10A-10C illustrate various assisted search result interfaces according to various implementations. [Figure 3C] 10A-10C illustrate various assisted search result interfaces according to various implementations. [Figure 4] FIG. 1 illustrates a computing system and search server for implementing concepts described herein. [Figure 5] 1 is a flowchart illustrating a method for implementing at least some of the concepts described herein. [Figure 6] 1 is a flowchart illustrating a method for implementing at least some of the concepts described herein. [Figure 7] 1A and 1B illustrate examples of a typical computing device and a typical mobile computing device that may be used with the techniques described herein. DETAILED DESCRIPTION OF THE INVENTION

[0005] Detailed Description The present disclosure relates to searching a database of items initiated through analysis of user-selected web page content rather than user-entered keywords. Specifically, an area of ​​the web page can be selected (e.g., selected using gestures and / or an input device), and the selected area can be searched in response to the selection of the area. The selected area can be analyzed for objects. In some implementations, additional context may be provided along with the selected content. This additional context can aid in identifying the object. In some implementations, at least two objects may be submitted as a query for searching the database of items. The object(s) submitted as a query may be optimized / customized to the database of items to increase the likelihood of finding a responsive item.

[0006] The disclosed implementations may also include generating a combined search results page. The search results page includes multiple search results and is presented in a browser user interface. Each search result corresponds to an item identified in response to a query. The items may be identified in the search results. The search results may include additional information related to the items (e.g., attributes of the items). Thus, the combined search results page may include a set of response items (a search result set) to a first query (e.g., for a first object) and a set of response items (a search result set) to a second query (e.g., for a second object). Items in the sets of items may be ranked independently of each other. The number of items in the first set of items may be a function of the number of items returned, a confidence score associated with the first object, a salience score for the first object, and the relevance of the items to the first object. The number of items in the second set of results may be determined similarly. In some implementations, the number of items in the second set may also be a function of the number of items in the first set, in addition to other factors. In some implementations, the items in the first set may include additional information (e.g., as rich results). In some implementations, items in the first set may be presented before items in the second set. In some implementations, the combined search results page may replace the content of the web page in the browser. In some implementations, the combined search results page may be displayed in a separate tab in the browser. In some implementations, the combined search results page may be presented in a separate window in the browser tab.

[0007] A technical challenge associated with searching non-document items (e.g., individual items) in a database is that such items may lack meaningful text to index. Unlike documents, which provide sufficient text for indexing, non-document items such as browser extensions, mobile applications, web applications, catalog / inventory items, etc., may have titles, short item descriptions, and / or manifest files from which terms / phrases can be captured and indexed. However, these attributes may not contain text that aligns with how users search. Thus, there can be a disconnect between how users search and how items are indexed and made searchable.

[0008] At least one technical solution to this technical problem of searching a database of items associated with minimal searchable text includes a search converter. The search converter may be located between a client computer (e.g., operated by a user) and a search engine of the database of items to assist the user in searching the database of individual items. The search converter may be configured to analyze selected content (e.g., user-selected images, text, hyperlinks, etc.) to identify objects within the content and generate a query based on the identified objects. In generating a query, the search converter may be configured to ensure that the identified objects are related to items within the database. For example, a search converter for a repository (database) of mobile applications may be configured to predict a selfie object as a query for the repository from an image showing a person's face, rather than predicting a male, female, person, or some other object. Similarly, a search converter for a database of books may predict an author's name as a query after analyzing the same image and / or may assign a higher saliency score to an author recognized in the image as opposed to other people recognized in the image. The search converter may be configured to initiate searches within the database corresponding to at least two objects identified in the selected content. These searches may be initiated in parallel. Thus, both queries (or several queries) can be submitted simultaneously, reducing the delay between receiving a query and presenting the results page for that query.

[0009] Another technical challenge associated with non-keyword-based searches is minimizing the number of separate search requests issued to enable a user to reach a satisfactory answer. As outlined above, a user may need to perform several separate searches using various keywords as query terms to reach a satisfactory answer. A satisfactory answer represents a search result page that includes at least search results that answer the intended question. Because a user's intent cannot be objectively determined, some conventional search systems search for the most likely intent (object) and may provide additional links pointing to other objects (e.g., links related to the fruit apple), which may initiate a different search when selected. When various objects are related to the inferred intent, some conventional search systems may include top results for different objects. For example, search results for the company Apple® may be interspersed with one or two top results for the fruit apple. Both approaches can increase the number of independent search requests.

[0010] Technical solutions to this technical problem include submitting parallel queries for various inferred intents and a novel interface that can display the top response items for each parallel query together in the interface. For example, if a search converter determines that the objects videoconferencing, browser, and video recording are recognized in a selected image and / or text, the search converter may submit all three objects as parallel queries to a search engine, which will return the top results for each object. The system may select and order these search results based on the confidence and salience scores assigned to the objects, so that several response items for the most salient query are returned and displayed together, followed by several response items from the second most salient query, and so on. In some solutions, the response items for the most salient query may be presented before the response items for the next query. In some solutions, the response items for the most salient query may include additional information about each response item that is not included in the response item results for other queries. In some solutions, the number of items displayed for each query (e.g., the set of response items selected for display) may depend on the confidence and salience scores of the objects submitted as queries and the relevance of the items to the query itself.

[0011] Technical effects of the features described herein include streamlining search and facilitating item discovery and exploration in situations where minimal searchable text is available for search indexing. Specifically, the disclosed technology may use machine learning or other techniques to generate smart queries based on predicted intent, potentially eliminating the manual process of search-retry-search-retry. Furthermore, searches may be accelerated because multiple queries (multiple inferred intents) are searched in parallel. Additionally, smart queries can be tailored / customized to the database being searched, increasing the likelihood that the smart query will return salient items and further reducing repetitive search requests. Parallel query submission and a novel results interface that integrates results from various queries facilitate exploration and discovery, allowing users to discover results they might not have thought to seek.

[0012] FIG. 1 illustrates content displayed in web page W1 on a client device. Web page W1 is displayed (e.g., rendered) in display area 110 within tab 112 of browser 105. Browser 105 includes address bar area 114. Address bar area 114 may be controlled by and / or associated with browser 105 (e.g., a browser application). Address bar area 114 may be controlled by browser 105 on behalf of web page W1 and / or a provider of web page W1. An address for web page W1 may be displayed in address bar area 114 (e.g., input address area 116). Other controls and / or icons may be included in address bar area 114. For example, address bar area 114 may include a search control 130. Search control 130 may be an icon, button, or image configured to initiate a search for selected content when activated by a user. In some implementations, search control 130 may be configured as a “drop” location. A drop location represents an area of ​​a display that can receive a drop action. The drop action is the final action in a drag-and-drop sequence. A drop action traditionally corresponds to releasing a mouse button, lifting a finger or stylus, a paste (e.g., control-v) keyboard command, etc. When a search control 130 is configured as a drop location, the search control 130 is activated to receive the drop action. In some implementations, the search control 130 may be configured as a button that is activated by a user selection of the search control 130. Activation of a selection control can occur in multiple ways, depending on the configuration of the client device. For example, a user may click on the search control 130 with a pointer, tap on the search control 130 with one finger, two fingers, etc., tap on the search control with a stylus, look at the search control and blink, or the like.

[0013] Web page W1 includes several items of content 118, examples of which include images I1, I2, I3, I4, and I5, text headings T1 and T2, and text blocks T3, T4, T5, and T6. A text block may represent a sentence, a paragraph, multiple paragraphs, one or more lists, one or more tables, etc. A text block may include a link. A link has anchor text (i.e., text displayed to the user) and a document address (not necessarily displayed). If the user's selection includes a link, the selected content may include the link's anchor text and the content from the link address (i.e., the document specified / identified by the link address). A user may select an item of content, such as text heading T1 (118a) or image I1 (118c). A user may select multiple items of content, such as image I3 and text blocks T4 and T5. A user may select portions of a content item, such as portions of text block T3 (118b). A user may select one content item and a portion of another content item.

[0014] 2A-2D illustrate various content selections that trigger a database of probable-intent searches of the item-selected content on the web page of FIG. 1, according to various implementations. FIG. 2A illustrates the selection of content item 118a (representing text heading T1), with the selection indicated by a diagonal line. Selecting content item 118a can cause a user to activate search control 130, which triggers a database of probable-intent searches of the item. To activate search control 130, a user may select content item 118a, drag it to control 130, and drop content item 118a onto control 130, as represented by arrow 210 in FIG. 2A. Alternatively (or additionally), a user may select content item 118a and then select search control 130 to trigger a database of probable-intent searches. Regardless of how search control 130 is activated, the selected content (e.g., the text of heading T1) is provided to the search system. Additionally, in some implementations, additional content items may be provided as context for the search. For example, content item T3 or a portion of content item T3 (e.g., the first sentence, a predetermined number of words proximate to the selected content item 118a) may be provided as the context. In another embodiment, all content items T2-T7 and I1-I5 may be provided. In another embodiment, content items proximate to the selected content 118a (e.g., text box T2 and image I1) may be provided, or some other combination of less than all content items may be provided.

[0015] FIG. 2B illustrates the selection of a portion of content item 118b (representing text block T2), with the selection indicated by diagonal lines. The portion of content item 118b may be a word, multiple words, a sentence, multiple sentences, a list item, a table cell, a table row, a table column, a link, or the like, or any combination thereof. As discussed above with respect to FIG. 2A, after selecting a portion of content item 118b, a user may activate search control 130, which triggers a presumed-intent search of a database of items using the selected portion of content item 118b. This may include dragging and dropping the selected portion of content item 118b onto search control 130 (represented by arrow 210), or the user selecting search control 130 once a portion of content item 118b is selected. As discussed with respect to FIG. 2A, another content item, a portion of a content item, multiple content items, or multiple portions of multiple content items may provide context for the search.

[0016] FIG. 2C illustrates the selection of an image content item 118c (representing image I1), with the selection indicated by a diagonal line. The image content item 118c may be any type of image, including text, icons, or other images. As discussed above with respect to FIGS. 2A and 2B, after selecting the image content item 118c, the user may activate the search control 130, which triggers a presumed-intent search of a database of items using the selected content item 118c. This may involve dragging and dropping the selected image content item 118c onto the search control 130 (represented by arrow 210), or the user selecting the search control 130 while the image content item 118c is selected. As discussed with respect to FIGS. 2A and 2B, another content item, a portion of a content item, multiple content items, or multiple portions of multiple content items may provide context for the search.

[0017] FIG. 2D illustrates the selection of multiple content items 118d (representing image I3, image I4, and text box T5), with the selection indicated by diagonal lines. The multiple content items 118d include both textual and image content. As discussed above with respect to FIGS. 2A-2C, after selecting a content item 118d, the user may activate the search control 130, which triggers an inferred intent search of a database of items using the multiple content items 118d. This may involve dragging and dropping the selected content item 118d onto the search control 130 (represented by arrow 210), or the user selecting the search control 130 while the content item 118d is selected. As discussed with respect to FIGS. 2A-2C, another content item, a portion of another content item, multiple content items, or multiple portions of multiple content items may provide context for the search.

[0018] 3A-3C illustrate various inferred-intent search result interfaces according to various implementations. The search result interfaces are generated in response to completing an inferred-intent search query. The combined search result page is represented by search result page W2. In the example of FIG. 3A, search result page W2 is displayed as a sidebar or as a split screen with web page W1. Thus, portions of both web page W1 and search result page W2 are displayed in display area 110 of browser tab 112. The search result page generated for an inferred-intent search may include sets of response items for two or more queries. In the example of FIGS. 3A-3C, search result page W2 includes three sets of response items. In the example of FIG. 3A, the first set of response items 302 is indicated by a vertical solid line and includes search results SR1-1, SR1-2, and SR1-3. Each of search results SR1-1, SR1-2, and SR1-3 represents a respective item in the database. The search results may include the names of the items. The search results may also include images or icons of the items. The search results may include a brief description of the item, a link to a more detailed description of the item, a page to purchase or download the item, etc. The first set of response items 302 are response items to a first query performed against a database of items.

[0019] The example of FIG. 3A includes a second set of response items 303, indicated by diagonal solid lines. The second set of response items 303 includes SR2-1, SR2-2, and SR2-3 (also referred to as search results). The items represented by the second set of response items 303 are response items to a second query executed against the database of items. The search results SR2-1, SR2-2, and SR2-3 may include information about each item similar to that included in the search results SR1-1, SR1-2, and SR1-3. The example of FIG. 3A includes a third set of response items 304, indicated by horizontal solid lines. The third set of response items 304 includes SR3-1, SR3-2, and SR3-3. The items represented by the third set of response items 304 are response items to a third query executed against the database of items. Submitting the third query, the second query, and the first query in parallel may enable the search system to provide the search result page W2 more quickly, thus reducing response time after query submission. The search results SR3-1, SR3-2, and SR3-3 may include information about each item similar to that contained in the search results SR1-1, SR1-2, and SR1-3. The number of items in each set of search results may be a function of several factors, including the size of the display area of ​​the search result page W2, the confidence and salience of the estimated object used as the query, the number of response items for the query, the relevance of those items to the query, and the number of items already selected in other sets of other queries. This is described in more detail in connection with FIG. 6 . Because W2 occupies a relatively small proportion (space) of the display area 110, the number of items in each set of items 302, 303, and 304 may be smaller as a result. In addition to the split window, if the client device is a smartphone or a wearable, the search result page W2 may have a smaller display area.

[0020] In the example of FIG. 3B, search results page W2' is displayed in a new tab 312 of browser 105. This new tab 312 is the active tab, and therefore, web page W2' fills the display area 110 of browser 105. The content of web page W1 associated with tab 112 is no longer visible, but may be made visible again in response to user selection of tab 112. In the example interface of FIG. 3B, search results page W2' displays three sets of response items 302', 303', and 304'. As in FIG. 3A, the three sets of search results (response items) represent three different queries executed against a database of items. Because web page W2' occupies a larger portion (space) of display area 110 than web page W2 of FIG. 2A occupies, there may be more items in each set of items. Thus, for example, the set of response items 302′ may include five items, the set of response items 303′ may include three items, and the set of response items 304′ may include four items. The information contained in each search result may be similar to the information contained in the search results described above with respect to FIG. 3A. In both FIGS. 3A and 3B, the response items to the first query (e.g., the set of response items 302 and 302′) have more salience than the response items to the second query (e.g., the set of response items 303 and 303′). Salience may be indicated by the first result being presented earlier / before the second result. Salience may also be indicated by other means (e.g., by taking up more display area (by being larger, taking up more space on the display), and / or by including more information).

[0021] FIG. 3C illustrates an implementation in which the response items to the first query are not presented before the response items to the second query, but instead include additional information or occupy a larger proportion of the search result page W2″. In the example of FIG. 3C, the set of response items 302″ represents the response items to the first query. These search results may be presented larger or occupy a larger proportion of the display area 110 than the set of response items 303′ and 304′. This is an example in which the set of response items 302″ is more prominent than the set of response items 303′ and 304′. The set of response items 302″ may include similar information as the set of response items 302′ (but in a larger font, at a higher zoom level, etc.). The set of response items 302″ may include additional information not included in the set of response items 302′. For example, the search result SR1-1 for the response items 302″ may include ratings, sale prices, publisher information, or other attributes of the items that were not included in the SR1-1 for the response items 302 or 302′. Because it contains additional information, the set of response items 302'' is sometimes referred to as a rich result. A rich result may include any additional information about the items that is not included in other sets of search results (e.g., the set of response items 303' and 304'). The set of response items 303' and 304' may also be displayed in another area / portion of the display. In some implementations, the set of response items 303' and 304' may be presented after the rich result (e.g., the set of response items 302'').

[0022] FIG. 4 illustrates a system 400 including a computing system 402 and a search server 410 for implementing the concepts described herein. The computing system 402 may also be referred to as a client computing device or client device. The computing system 402 is a device having an operating system 429. In some embodiments, the computing system 402 includes a personal computer, a mobile phone, a tablet, a netbook, a laptop, a smart appliance (e.g., a smart TV), or a wearable. The computing system 402 can be any computing device having a display that allows a user to select displayed content. The computing system 402 may include one or more processors 464 configured to execute one or more machine-executable instructions, or portions of software, firmware, or a combination thereof, and formed within a substrate. The processor 464 may be semiconductor-based; that is, the processor may include semiconductor material capable of executing digital logic. The computing system 402 may also include one or more memory devices 463. The memory device 463 may include a main memory that stores information in a format that can be read and / or executed by the processor 464. The memory device 463 may store applications or modules (eg, operating system 429, applications 428, selection manager 430, browser 420, etc.) that perform predetermined operations when executed by the processor 464.

[0023] Operating system 429 is system software that manages a computer's hardware and software resources and provides common services to computing programs. In some embodiments, operating system 429 is operable to run on a personal computer, such as a laptop, netbook, or desktop computer. In some embodiments, operating system 429 is operable to run on a mobile computer, such as a smartphone or tablet. Operating system 429 may include multiple modules configured to provide common services and manage the resources of computing system 402. Computing system 402 may include one or more input devices 467 that enable a user to select content. Non-exclusive example input devices 467 include a keyboard, a mouse, a touch-sensitive display, a trackpad, a trackball, etc. Computing system 402 may include one or more output devices 468 that enable a user to view web pages and / or receive audio or other visual output.

[0024] The computing system 402 may include applications 428, which represent specially programmed software configured to perform various functions. One of the applications may be a browser 420. The browser 420 may be configured to display web pages, run web applications, and the like. The browser 420 may include additional functionality in the form of extensions. The operating system may also include a selection manager 430 configured to allow a user to select, copy, paste, drag, and drop content. The browser 420 is an embodiment of the browser 105 of FIG. 1 and may include controls, such as the control 130, configured to trigger a contextual search of the database 419 and / or the database 479. The browser 420 may include an intent search client 425 configured to trigger a search. Thus, the intent search client 425 may represent software that executes in response to activation of a search control. The intent search client 425 may transmit selected content to the search server 410. The intent search client 425 may capture and use the context surrounding the selected content and provide this context to the search server 410. For example, if the selected content is a sentence, the paragraph in which the sentence occurs may also be sent to the search server 410 along with the selected content (or possibly the entire page).

[0025] In some embodiments, computing system 402 may communicate with search server 410 via network 450. Search server 410 may be one or more computing devices in the form of multiple different devices, such as, for example, a standard server, a group of such servers, or a rack server system. In some embodiments, search server 410 may be a single system that shares components such as a processor and memory. Network 450 may include the Internet and / or other types of data networks, examples of which include a local area network (LAN), a wide area network (WAN), a cellular network, a satellite network, or other types of data networks. Network 450 may also include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) configured to receive and / or transmit data within network 450. Network 450 may further include any number of wired and / or wireless connections.

[0026] The search server 410 may include one or more processors 415, an operating system (not shown), and one or more memory devices 417 formed within the substrate. The memory device 417 may represent any type (or types) of memory (e.g., RAM, flash, cache, disk, tape, etc.). In some embodiments (not shown), the memory device 417 may include external storage, e.g., memory that is physically remote from the search server 410 but accessible by the search server 410. The search server 410 may include one or more modules or engines that represent specially programmed software. For example, the search server 410 may include a database search engine 418 configured to search a database of items 419. The search engine 418 includes an index of terms used to determine which items in the database 419 are responsive to a query. The database of items 419 includes non-document items. In other words, the items in the database 419 are not documents available on the Internet, but instead are a more limited repository of individual items. Examples of items in database 419 include installable items, examples of which are browser extensions, mobile applications, native applications, and progressive web applications. Other examples of items in database 419 include items for purchase (e.g., via an online store, classified ads, or catalog), items in a library, items for streaming (e.g., video, audio, media files), items for rental, etc. Database 419 may be searchable by a limited number of text fields. For example, items in database 419 may be indexed by terms in the item's description. Items in database 419 may be indexed by the item's title and / or another identifier. Items in database 419 may be indexed by terms in a manifest file (e.g., for items that are applications or extensions).The database search engine 418 may use conventional techniques to search for items in the database 419 that are responsive to the query. The database 419 may be distributed across several different computing devices and / or may be remote from the search server 410.

[0027] The search server 410 also includes an assisted search interface 412. The assisted search interface 412 may be a layer located between the computing system 402 and the database search engine 418. In some implementations, the assisted search interface 412 may be part of the database search engine 418. In some implementations, the assisted search interface 412 may be separate from the database search engine 418. The assisted search interface 412 may be invoked when a user does not provide keywords for searching the database 419. This is the case, for example, when a user selects content within a web page and triggers a search via a control 130, as shown in any of FIGS. 2A-2D. The assisted search interface 412 may take the selected content as input. The assisted search interface 412 may take the selected content and a context for the selected content as input. The assisted search interface 412 may analyze the selected content to identify an inferred (most likely) search intent and submit one or more queries for the inferred intent(s) to the database search engine 418. The assisted search interface 412 may provide a search result page as output. The search results page includes response items from database 419. The response items are grouped in the search results page by query, as shown in any of Figures 3A-3C.

[0028] The assisted search interface 412 may include a search converter 416 and / or a parallel search module 414. The search converter 416 may analyze the selected content and determine search intent that may be expressed in the selected content. Search intent is also referred to as an object. An object represents a keyword, phrase, entity, etc. that can be used as a query in a database search engine. The search converter 416 may use machine learning techniques to analyze the input (selected content). The analysis technique used by the search converter 416 may depend on the type(s) of content contained in the selected content. For example, if the selected content is text, the search converter 416 may pass the text as a query to the database search engine 418. As another example, if the selected content is text, the search converter 416 may use conventional or later-developed natural language processing techniques to parse the text and identify entities mentioned within the text to submit as a query. Such techniques may be performed by the search converter 416 and / or the object identifier 413. The mentioned entities are objects identified within the content.

[0029] Natural language processing techniques can provide a confidence level that a particular entity is the entity mentioned in the text. For example, the text jaguar can refer to several different entities, such as a car, an animal, or a soccer team. Natural language processing techniques can provide a confidence level (score) for each different type of entity. The confidence score represents the likelihood that a particular entity (e.g., a car) is the intended meaning over another entity (e.g., a soccer team). Contextual information can improve the confidence level. Contextual information may be provided with the selected content and may represent content (e.g., text and / or images) near the selected content within the source web page. In some implementations, if contextual information is not provided with the selected content and the entity confidence score does not exceed a predetermined threshold, the search converter 416 can obtain additional context, for example, by requesting it from the computing system 402. In implementations where some context is provided but the confidence score does not exceed a predetermined threshold, the search converter 416 can obtain additional context from the computing system 402. This context can be used in determining the query terms (objects), but in some implementations, this context may only be used if the intent engine has low confidence in the recognized object.

[0030] If the selected content includes image data, the search converter 416 may perform object recognition on the image data using conventional or later-developed object recognition techniques. Such techniques may be performed by the object identifier 413. The object identifier 413 may use machine learning techniques to identify objects (entities, keywords, etc.) within one or more images. The object identifier 413 may include a machine learning model, a classifier, and / or a regression model. The object identifier 413, or portions of the object identifier 413, may be a service provided by another server or computing device. The object identifier 413 may take as input text, an image, multiple images, or a combination of text and images, and may provide objects identified within the input. In some implementations, the object identifier 413 may be part of the search converter 416.

[0031] Each object may have a respective confidence score, which represents the likelihood that the object appears in the input (e.g., the selected content). The object identifier 413 may provide each object with a respective salience score. The salience score represents how important / topical the object is to the input. For example, a high salience score indicates that the object is the main topic, main point, or main object in the text or image(s). A low salience score indicates that the object is ancillary or secondary object. A high-confidence, high-salience object represents a strong inferred search intent. A high-confidence, low-salience object represents a weaker search intent. In some implementations, the object identifier 413 may include different models for analyzing different types of content, such as a text or natural language model and an image recognition model. In some implementations, the search converter 416 may determine which model to use in the analysis based on the type of content. In some implementations where the selected content includes two different types of content, a portion of the selected content (e.g., a text portion) may be provided to one model (or no model), and another portion of the selected content (e.g., an image portion) may be provided to a different model. In some implementations, the search converter 416 may use a first model to determine the type(s) of the selected content.

[0032] In some implementations, the search converter 416 may convert / translate objects identified in the selected content into keywords (objects) related to items in the database 419. For example, the object identifier 413 may be a service or model that performs object detection, but the objects may not necessarily be translated into related terms (e.g., searchable item descriptions) in the database 419. The search converter 416 may convert the identified objects into objects that are more likely to result in the identification of items in the database 419. For example, if the object identifier 413 provides biking or skiing, the search converter may convert the identified objects into sports for querying the database 419 if the database 419 is related to games. Similarly, the search converter 416 may convert beach ball or sandcastle objects into ocean views if the database 419 is related to rental properties or homes for sale. In some implementations, this conversion / translation can be performed using machine learning. For example, the object identifier 413 may provide candidate objects as input, and the search converter 416 may provide identified objects as output for use in querying the database. In this sense, the search converter 416 may be considered an additional, or final, layer of the object detection model.

[0033] The assisted search interface 412 (e.g., the search converter 416 and / or the parallel search module 414) may rank the identified objects, e.g., the objects provided by the object identifier 413. The rank of the identified objects may be determined based on a combination of the object's confidence score and salience score. In some implementations, objects that do not meet a predefined confidence score threshold may not be considered for querying the database 419. In some implementations, objects that do not meet a predefined salience threshold may not be considered for querying the database 419. In some implementations, objects that do not have a combination score that meets a predefined combination threshold may not be considered for querying the database 419. Identified objects that are considered for querying the database 419 are provided to the database search engine 418 as separate queries. In other words, if two objects are considered for querying (e.g., because they have confidence scores and / or salience scores that meet the thresholds), the assisted search interface 412 sends two separate queries to the database search engine 418. These queries may be executed in parallel, i.e., simultaneously, by the database search engine 418. This reduces the delay between the user triggering a query and the presentation of the search results page. If the search converter 416 identifies four objects, four independent queries may be submitted to the database search engine 418.

[0034] In some implementations, if the database 419 contains no or only one object that is considered for querying, the search converter 416 may provide additional context to the object identifier 413, which may use the additional context to analyze and score the object identified in the selected content. The additional context may already have been provided to the search converter 416, for example, along with the selected content. The additional context may also be obtained from the computing system 402 that triggered the query. In some implementations, the search converter 416 may start with contextual content proximate to the selected content in the source web page (e.g., W1 in FIG. 1 ) and expand the context as needed to increase the confidence score. In some implementations, the search converter 416 may simultaneously provide all contextual information it receives or has access to. In some implementations, the search converter 416 may submit at least two (or more) queries for the selected content, regardless of whether the second object meets the confidence and / or salience thresholds. This may reduce the probability of multiple queries being submitted.

[0035] The assisted search interface 412 may include a parallel search module 414 that manages queries and returned results. For example, the parallel search module 414 may submit queries to a database search engine 418 (or directly to a database 419). The parallel search module 414 may rank the objects submitted as queries. Ranking may establish first or primary objects, secondary objects, etc. Once search results, i.e., response items to a query, are returned, the parallel search module 414 may determine which response items for different queries to include in the combined search result page. In other words, the parallel search module 414 may determine a set of response items for each query to return as the combined search result page. The number of items in each set of items may depend on multiple factors. For example, the set of items for the primary query (those with the highest combined confidence and salience scores) may depend on the total number of response items returned, the relevance of the items to the query, the difference between the combined scores for the primary query and the secondary queries, and the display area available on the combined search result page. If the combined score of the highest ranked query is significantly greater than the combined score of the second-highest ranked query, then the number of items in the set of response items for the highest ranked query may be greater than the number of items in the set of response items for the second-highest ranked query, provided there are enough response items to satisfy this maximum. Similarly, if the combined scores of two highest ranked queries are close, then the number of items in the two sets of response items may also be close.

[0036] The parallel search module 414 may generate the combined search results page by grouping the response items in each set of items together. In other words, the response items in the sets may be kept together in the combined search results page. In some implementations, a response item in a first set (e.g., a response item to a highest ranked query) may have more salience than a response item in a second set (e.g., a response item to a second highest ranked query). In some implementations, salience may involve displaying the first set of items before the second set of items. In some implementations, salience may involve including more information about the response items in the first set than the response items in the second set. In some implementations, individual response items in the first set may be presented larger than individual response items in the second set, e.g., occupy a larger proportion of the display. In some implementations, salience may include giving the response items in the first set an appearance that is distinct from the response items in the second set. For example, the response items in the first set may be highlighted, may have a different color background, may have bold text, etc. The parallel search module 414 may provide a combined search results page to the computing system 402 that triggered the query for display (e.g., in a window or tab of the browser 420).

[0037] In some implementations, the assistance search interface 412 may send one or more queries to an object provider 470 that is remote from the search server 410. The object provider 470 may be a server that includes a database search engine 476. The database search engine 476 may be functionally similar to the database search engine 418. The database search engine may receive queries and search for responsive items in a database 479. The database 479 may not be a database of documents, but may be similar to the database 419 in that it may index (make searchable) items based on limited text, such as item descriptions, item titles, or item manifests. Thus, implementations are not limited to querying a search engine co-located on the search server 410. Additionally, in some implementations, the database search engine 418 may be included in (a component of) the intent search interface 412. Thus, the configuration shown in FIG. 4 is one example, and implementations may include variations of the illustrated components.

[0038] FIG. 5 is a flowchart illustrating a method 500 that may be performed on a client device in accordance with at least some of the concepts described herein. Method 500 may trigger an assisted search of a database of non-document items and present search results to a user. Method 500 may be performed by a browser on the client device (e.g., browser 420 of FIG. 4 or browser 105 of FIG. 1 or 2A-2D). Method 500 may begin with displaying a web page in a display area of ​​the browser (502). The display area may be associated with a browser tab. The browser may receive a selection of content (504). The content may be content displayed within a web page. The selected content may be text-based. The selected content may be image-based. The selected content may be a combination of text and image data. Conventional techniques may be used to achieve the selection. The browser may detect activation of a search control while content is selected and trigger an assisted search of the database of items (506). A user may activate the search control by dragging the selected content to the search control and dropping the selected content onto the search control. The search control may be activated by the user clicking / selecting it while content is selected within the web page.

[0039] In some implementations, a context for the selected content may be identified (508). The context may be text and / or images near (adjacent to, surrounding) the selected content. If the selected content includes a link, the context may be content associated with a document identified by the link's link address. The context may include a portion of the content on a web page, or may include all of the content on a web page. The context may include content within a given document object. The context item may be determined by proximity or another relationship to the selected content. The context item may be provided to an assisted search interface along with the selected content. Providing the selected content (and optionally the context item) triggers a search of a database of non-document items (510). In response to triggering the search, the browser receives a combined search results page (512). The combined search results page includes a set of items responsive to the first query (first search results) and a set of items responsive to a second query submitted to the database of items (second search results). The two sets of items are displayed as a group on the combined search results page. For example, each response item may be actionable such that clicking or tapping on a search result of the response item causes the client device to perform an action. The action may depend on the type of item represented in the database of items. The browser may display the combined search results page (514). This display may occur in a new browser tab, a new browser window, or in a separate view with the web page from which the content was selected. Method 500 then ends. While a combined search results page is illustrated, in some implementations the search results page may not be a combined search results page, but may instead be a search results page for a single submitted query (e.g., the first set of search results).

[0040] FIG. 6 is a flowchart illustrating a method 600 performed by a search server in accordance with at least some of the concepts described herein. To search a database of individual items (e.g., items that are not web-based documents but have specific text associated with them for use in the search), method 600 identifies at least one object from the selected content that is interpreted as a probable search intent. Typically, multiple objects (probable queries) may be identified from the selected content. A search is performed using the probable queries to generate a combined search results page, where response items for a particular query are grouped together rather than ranked and interleaved together. Method 600 can be performed by an assisted search interface, such as assisted search interface 412 of FIG. 4.

[0041] The method 600 may begin by receiving selected content from a client device (602). In some implementations, the selected content may be received along with a context item. The method 600 may include determining a first object and a second object for the selected content (604). In implementations where context is provided with the selected content, the context may be used in determining the first object and the second object. Determining the objects may include using at least one of a machine learning model, a classifier, and a regression model. The object detection technique used may depend on the type of selected content. For example, one technique may be used for selected text and another technique may be used for selected images. If the selected content includes text and images, multiple object detection techniques may be used. In some implementations, object detection may use a general detector, e.g., one that is not customized or specialized for the items in the database. In such implementations, determining the first object and the second object may include translating objects identified by a general detector (e.g., candidate objects identified using traditional natural language processing, entity detection, etc.) into objects that are more closely related to the individual database of items. Some implementations may use machine learning models customized for the database of items for conversion. In some implementations, the object detector may already be trained (customized) for the database of items.

[0042] Each identified object may have a respective confidence score. The confidence score indicates the likelihood (probability) that the object is reflected in the selected content. In some implementations, the first object and the second object may have a confidence score that meets a predetermined confidence threshold, such as 75% confidence, 80% confidence, or 95% confidence. Each identified object may have a respective salience score. The salience score indicates how topical or relevant the object is to the selected content. In some implementations, the first object and the second object may have a salience score that meets a predetermined salience threshold, such as 40%, 50%, or 55%, where a lower score indicates that the object is less likely to be the subject of the selected content (i.e., not the main idea or concept), and a higher score indicates that the object is more highly relevant to the selected content. For example, an object in the foreground of an image may be considered more salient to the image than an object in the background or an object that is partially obscured, etc. In some implementations, if no identified object has a confidence score that meets the confidence threshold (606, No), additional context may be obtained from the web page (608) and object detection may be repeated using the additional context for the selected content (604). This may help improve the confidence score of one or more initially detected objects. In some implementations, no context items are provided with the selected content. In some implementations, the context items provided with the selected content represent only a portion of the content of the web page from which the selected content was obtained, and additional context items may be obtained from the web page. In some implementations, only a portion of the context items provided with the selected content may be used in initially determining the object, and additional context items may be used in a second period of object detection (and possibly a third period using additional context items, if necessary).

[0043] The method 600 includes generating 610 a combined search result page for a query corresponding to a first object and a second object. Generating the combined search result page may include generating 612 a first result (a first set of results) by performing a first search of items in a database using the first object as a query. Similarly, generating 614 a second result (a second set of results) by performing a second search of items in the database using the second object as a query. The method 600 may also include selecting 616 a first set of response items to the first query from the first results for inclusion in the combined search result page (corresponding to the first set of search results) and selecting 616 a second set of response items to the second query from the second results for inclusion in the combined search result page (corresponding to the second set of search results). Each search result in the first set of search results corresponds to an item identified as responsive to the first query. Each search result in the second set of search results corresponds to an item identified as responsive to the second query. In some implementations, the first object and the second object (as well as any other objects submitted as a query) may be ranked using, for example, their respective confidence scores, their respective salience scores, or a combination thereof. In some implementations, the first object may be the highest ranked object. Although not shown in FIG. 6, additional objects may be determined and used in querying the database and generating the combined results. In some implementations, ranking may be performed before submitting the query. In some implementations, any object having a rank that meets a threshold may be submitted as a query.

[0044] In an implementation in which objects are ranked, a first set of results may have more salience than a second set of results. In one example, salience may mean that the first set of results may be displayed before the second set of results (e.g., as shown in FIG. 3A). In one example, salience may mean that an individual result in the first set of results may occupy more display area (e.g., be larger or have more information items / attributes) than an individual result in the second set of results (e.g., as shown in FIG. 3C). In one example, salience may mean that the first set of results may have a greater number of members (i.e., a greater number of items represented by search results in the result set) than the second set of results (e.g., as shown in FIG. 3B). The number of items in each set of results, or in other words, the number of members in the set of search results included in a search results page, may depend on multiple factors, examples of which include the confidence scores and / or salience scores of each of the objects corresponding to the query, the number of response items to the query, the relevance of the response items to the query, and the display area available on the combined search results page. In some implementations, the number of members (e.g., the number of items represented therein) in the first set of results (included in the search results page) may be a function of the confidence score of the first object, the salience score of the first object, the number of items identified as responsive to the first query, the size of the display, and / or the relevance of the first result to the first object. In some implementations, the number of members in the second set of results (included in the search results page) may be a function of the confidence score of the second object, the salience score of the second object, the number of items identified as responsive to the second query, the size of the display, and / or the relevance of the second result to the second object. In some implementations, the number of members in the second set of results may be influenced by the number of members in the first set of results. In some implementations, the number of members in the second set of results is determined after the number of members in the first set of results is determined.The method 600 may include providing 618 the combined search results page to the client device, where the combined search results page is displayed to the user, for example, within a browser user interface. The method 600 then ends.

[0045] 7 illustrates an example of a generic computing device 700 and a generic mobile computing device 750 that may be used with the techniques described herein. Computing device 700 is intended to represent various forms of digital computers, examples of which include laptops, desktops, tablets, workstations, personal digital assistants, televisions, servers, blade servers, mainframes, and other suitable computing devices. Computing device 750 is intended to represent various forms of mobile devices, examples of which include personal digital assistants, mobile phones, smartphones, and other similar computing devices. The components, their connections and relationships, and their functions illustrated herein are intended for illustrative purposes only and are not intended to limit the implementation of the invention(s) described and / or claimed herein.

[0046] The computing device 700 includes a processor 702, a memory 704, a storage device 706, a high-speed interface 708 connecting to the memory 704 and a high-speed expansion port 710, and a low-speed interface 712 connecting to a low-speed bus 714 and the storage device 706. The processor 702 can be a semiconductor-based processor. The memory 704 can be a semiconductor-based memory. The components 702, 704, 706, 708, 710, and 712 are interconnected using various buses and may be mounted on a common motherboard or in other manners as desired. The processor 702 can process instructions for execution within the computing device 700, including instructions stored in the memory 704 or the storage device 706, and display graphical information for a GUI on an external input / output device, such as a display 716 connected to the high-speed interface 708. Other implementations may use multiple processors and / or multiple buses as desired, along with multiple memories and types of memories. Additionally, multiple computing devices 700 may be connected together, with each device providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0047] The memory 704 stores information within the computing device 700. In one implementation, the memory 704 is a volatile memory unit(s). In another implementation, the memory 704 is a non-volatile memory unit(s). The memory 704 may also be another form of computer-readable medium, such as a magnetic disk or optical disk.

[0048] The storage device 706 can provide mass storage for the computing device 700. In one embodiment, the storage device 706 can be or include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or any number of devices, including a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. A computer program product can be tangibly embodied on an information carrier. The computer program product can also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, examples of which include memory 704, the storage device 706, or memory on the processor 702.

[0049] The high-speed controller 708 manages bandwidth-intensive operations for the computing device 700, while the low-speed controller 712 manages low-bandwidth-intensive operations. This allocation of functionality is merely exemplary. In one implementation, the high-speed controller 708 is coupled to the memory 704, the display 716 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 710, which may accept various expansion cards (not shown). In this implementation, the low-speed controller 712 is coupled to the storage device 706 and the low-speed expansion port 714. The low-speed expansion port may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet). The low-speed expansion port may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, etc., via, for example, a network adapter, or may be coupled to a network device, such as a switch or router.

[0050] Computing device 700 may be implemented in a number of different forms, as shown. For example, it may be implemented as a standard server 720, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 724. Additionally, it may be implemented in a personal computer, such as a laptop computer 722. Alternatively, the components of computing device 700 may be combined with other components in a mobile device (not shown), such as device 750. Each such device may include one or more of computing devices 700, 750, and the entire system may consist of multiple computing devices 700, 750 in communication with each other.

[0051] Computing device 750 includes, among other components, a processor 752, memory 764, an input / output device such as a display 754, a communication interface 766, and a transceiver 768. Device 750 may also be provided with a storage device such as a microdrive or other device to provide additional storage. Components 750, 752, 764, 754, 766, and 768 are interconnected using various buses, and several components may be mounted on a common motherboard or in other manners as desired.

[0052] The processor 752 can execute instructions within the computing device 750, including instructions stored in the memory 764. The processor may be implemented as a chipset of chips including separate analog and digital processors. The processor may provide, for example, coordination of the other components of the device 750, including control of a user interface, applications run by the device 750, and wireless communication by the device 750.

[0053] The processor 752 may communicate with a user via a control interface 758 and a display interface 756 coupled to a display 754. The display 754 may be, for example, a thin film transistor liquid crystal display (TFT LCD) or an organic light emitting diode (OLED) display, or other suitable display technology. The display interface 756 may include appropriate circuitry for driving the display 754 to present graphical and other information to the user. The control interface 758 may receive commands from the user and convert them for submission to the processor 752. Additionally, an external interface 762 may be provided in communication with the processor 752 to enable short-range communication between the device 750 and other devices. The external interface 762 may, for example, provide for wired communication in some implementations or wireless communication in other implementations, and multiple interfaces may be used.

[0054] Memory 764 stores information within computing device 750. Memory 764 may be implemented as one or more computer-readable media, one or more volatile memory units, or one or more non-volatile memory units. Expansion memory 774 may also be provided and connected to device 750 via expansion interface 772. Expansion interface 772 may include, for example, a single in-line memory module (SIMM) card interface. Such expansion memory 774 may provide extra storage space for device 750 or may store applications or other information for device 750. Specifically, expansion memory 774 may include instructions for performing or supplementing the processes described above and may also include secure information. Thus, for example, expansion memory 774 may be provided as a security module for device 750 and may be programmed with instructions that enable secure use of device 750. Additionally, secure applications may be provided via SIMM cards, with additional information, such as identification information, located on the SIMM card in an unhackable manner.

[0055] The memory may include, for example, flash memory and / or NVRAM memory, as described below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, examples of which include memory 764, expansion memory 774, or memory on processor 752. The information carrier may be received, for example, via transceiver 768 or external interface 762.

[0056] Device 750 may communicate wirelessly via communication interface 766, which may include digital signal processing circuitry if necessary. Communication interface 766 may provide for communication in various modes or protocols, examples of which include GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, via transceiver 768. Additionally, short-range communication may occur using Bluetooth, WiFi, or other such transceivers (not shown). Additionally, a global positioning system (GPS) receiver module 770 may provide additional navigation- and location-related wireless data to device 750, which may be used as needed by applications executing on device 750.

[0057] Device 750 may also perform voice communications using an audio codec 760, which may receive voice information from a user and convert it into usable digital information. Audio codec 760 may also generate sounds that are heard by the user, such as through a speaker (e.g., in the handset of device 750). Such sounds may include sounds from voice telephone calls, recorded sounds (e.g., voice messages, music files, etc.), and sounds generated by applications running on device 750.

[0058] Computing device 750 may be implemented in a number of different forms, as shown in the figure, including as a mobile phone 780, a smartphone 782, a personal digital assistant, or part of other similar mobile devices.

[0059] Various implementations of the systems and techniques described herein may be realized in digital electronic circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be specialized or general-purpose, coupled to receive data and instructions from the storage system, and to transmit data and instructions to the storage system.

[0060] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or an assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives the machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0061] To provide for interaction with a user, the systems and techniques described herein are implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) by which the user can provide input to the computer. Other types of devices can also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, verbal, or tactile input.

[0062] The systems and techniques described herein can be implemented in a computing system that includes a back-end component (e.g., as a data server), or a computing system that includes a middleware component (e.g., an application server), or a computing system that includes a front-end component (e.g., a client computer having a graphical user interface or web browser through which a user can interact with an implementation of the systems and techniques described herein), or a combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0063] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0064] Multiple implementations have been described. Of course, it will be understood that various modifications may be made without departing from the spirit and scope of the present invention. Furthermore, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desired results. Furthermore, other steps may be added to or eliminated from the described flows, and other components may be added to or removed from the described systems.

[0065] According to one aspect, a computer-implemented method includes receiving selected content from a client device, determining at least a first object and a second object for the selected content using a search converter, generating first results by performing a first search of items in a database using the first object as a query, and generating second results by performing a second search of items in the database using the second object as a query. The method may further include selecting a first set of first results based on relevance of the first object to the selected content and selecting a second set of second results based on relevance of the second object to the selected content, and providing a user interface for display including the first set of first results and the second set of second results.

[0066] These and other aspects may include one or more of the following features, alone or in combination. For example, a first result in a first set may have more salience than a second result in a second set. This salience may be manifested by the first result in the first set having more information than the second result in the second set, the first set taking up more space on the display than the second set, and / or the first result in the first set being displayed before the second result in the second set. As another example, the search converter may include an object detection model having a layer that converts objects identified in the selected content into keywords related to items in the database. As another example, the first set may have a first number of members, and the first result may have a second number of results, where the first number of members may be a function of at least two of the confidence score of the first object, the salience score of the first object, the second number of results, the size of the display, or the relevance of the first result to the first object. As another example, a query for the first object and a query for the second object may be submitted in parallel. As another example, the first set may be more salient than the second set due to a score calculated by the search converter for the first object being higher than a score calculated by the search converter for the second object. This score may be a combination of the confidence score and the salience score. As another example, the search converter may use at least one of a machine learning model, a classifier, or a regression model. As another example, the search converter may include a first model that classifies a type of selected content, the type of selected content being used to determine a second model for analyzing the selected content. As another example, the method may also include receiving a context for the selected content, the context including other content displayed with the selected content, the context being used in determining the first and second objects by the search converter.As another example, the database may include at least one of a searchable item description, a searchable item title, or a searchable manifest file. As another example, the first result and the second result are generated by a first score and a second score calculated by the search converter for the first object.

[0067] According to one aspect, a system can include at least one processor and a memory storing instructions that, when executed by the at least one processor, cause the system to perform any of the methods or operations disclosed herein. According to one aspect, a system can include a non-transitory computer-readable medium storing instructions that, when executed by the processor, cause the computing system to perform any of the methods or operations disclosed herein.

Claims

1. 1. A computer-implemented method comprising: receiving selected content from a client device; receiving a context for the selected content, the context including other content to be displayed with the selected content; The method further comprises: and determining at least a first object and a second object for the selected content using a search converter, wherein the context is used by the search converter in determining the first object and the second object; The method further comprises: performing a first search of items in a database using the first object as a query to generate a first result; performing a second search for the item in the database using the second object as a query to generate a second result; selecting a first set of first results based on the relevance of the first object to the selected content and selecting a second set of second results based on the relevance of the second object to the selected content; providing a user interface including the first set of first results and the second set of second results for display; A computer-implemented method, comprising:

2. The computer-implemented method of claim 1 , wherein the first result in the first set has more salience than the second result in the second set.

3. The computer-implemented method of claim 2 , wherein the salience is indicated as the first result in the first set having more information than the second result in the second set.

4. The computer-implemented method of claim 2 , wherein the salience is indicated as the first set taking up more space on the display than the second set.

5. 3. The computer-implemented method of claim 2, wherein the salience is indicated by the first result in the first set being displayed before the second result in the second set.

6. 2. The computer-implemented method of claim 1, wherein the first set has more salience than the second set because the score calculated by the search converter for the first object is higher than the score calculated by the search converter for the second object.

7. The computer-implemented method of claim 6 , wherein the score is a combination of a confidence score and a saliency score.

8. The computer-implemented method of claim 1 , wherein the database includes at least one of searchable item descriptions, searchable item titles, or searchable manifest files.

9. 2. The computer-implemented method of claim 1, wherein the first set has a first number of members and the first result has a second number of results, and the first number of members is a function of at least two of a confidence score of the first object, a salience score of the first object, the second number of results, a size of the display, or a relevance of the first result to the first object.

10. The computer-implemented method of claim 1 , wherein the query for the first object and the query for the second object are submitted in parallel.

11. 2. The computer-implemented method of claim 1, wherein the search converter includes a first model for classifying a type of the selected content, and wherein the type of the selected content is used to determine a second model for analyzing the selected content.

12. The computer-implemented method of claim 1 , wherein the search converter uses at least one of a machine learning model, a classifier, or a regression model.

13. 10. The computer-implemented method of claim 1, wherein the search converter includes an object detection model having a layer that converts objects identified in the selected content into keywords for searching items in the database.

14. 1. A system comprising: at least one processor; and a memory storing instructions that, when executed by the at least one processor, cause the system to perform operations, the operations including: receiving selected content from a client device; receiving a context for the selected content, the context including other content to be displayed with the selected content; The operation may further include: and determining at least a first object and a second object for the selected content using a search converter, wherein the context is used by the search converter in determining the first object and the second object; The operation may further include: performing a first search of items in a database using the first object as a query to generate a first result; performing a second search for the item in the database using the second object as a query to generate a second result; selecting a first set of first results based on the relevance of the first object to the selected content and selecting a second set of second results based on the relevance of the second object to the selected content; providing a user interface including the first set of first results and the second set of second results for display; Including, the system.

15. 15. The system of claim 14, wherein the search converter includes an object detection model having a layer that converts objects identified in the selected content into keywords for searching items in the database.

16. 16. The system of claim 14 or 15, wherein the first set has more salience than the second set because the score calculated by the search converter for the first object is higher than the score calculated by the search converter for the second object.

17. The system of claim 16 , wherein the score is a combination of a confidence score and a saliency score.

18. the first result and the second result are generated by a first score calculated by the search converter for the first object and a second score calculated by the search converter for the second object that satisfies a threshold; The system of claim 14 or 15, wherein the first score and the second score reflect confidence and salience.

19. A program for causing a processor of a computing device to carry out the method of any one of claims 1 to 13.

Images