Search system providing search results and generated content

Abstract

A computing system is described that includes a processor and a memory having instructions stored thereon that, when executed by the processor, cause the processor to perform several actions. The actions include receiving a query from an application executing on a client computing device in network communication with the computing system. The actions also include searching a computer-readable index for items based on the query, identifying the items based on the search of the computer-readable index, sending the query to a computer-implemented model, and retrieving content generated by the computer-implemented model, where the computer-implemented model generated the content based on the query. The actions further include returning at least one of the items or the content to the client computing device for presentation by the application executing on the client computing device.

Description

[Background technology]

[0001] background Traditionally, computer-implemented search engines receive queries from client computing devices operated by users, search a computer-readable index of items (e.g., web pages, images, videos, etc.) based on the query, identify items from the index that are relevant to the query, rank the items to create a ranked list, and return a search engine search results page (SERP) to the client computing device, the SERP including the ranked list of items. While search engine technology is continually improving at inferring user intent based on queries issued by the user, traditional search engines are limited to identifying items indexed and / or stored by the search engine based on the user's query. If a search engine receives a query submitted by a user that accurately reflects the user's information search intent, but the search engine does not index and / or store items relevant to that intent, the search engine may return nothing or irrelevant results to the user. Summary of the Invention [Means for solving the problem]

[0002] overview The following is a brief summary of subject matter described in more detail herein. This summary is not intended to limit the scope of the claims.

[0003] This specification describes various techniques for returning search results to a user in response to receiving a query from the user, where the search results may include both 1) previously indexed and / or stored items and 2) content generated by a computer-implemented model based on the query. More specifically, a computing system receives a query from an application executing on a client computing device in network communication with the computing system. The application may be a web browser, a mobile search engine application, a productivity application (such as a word processing application, an email application, a spreadsheet application, a slide presentation application, etc.).

[0004] Based on the query, the computing system searches a computer-readable index of items to identify items related to the query. The items may be web pages, images, videos, etc. The computing system can identify items indexed in the computer-readable index based on the query. Additionally, the computing system can perform operations related to generating content based on the query (wherein, in the context of content generation, a query may be referred to as a prompt). In one example, a search engine search results page (SERP) includes items identified by the computing system as related to the query and a selectable button indicating that content can be generated based on the query. The computing system transmits the SERP to a client computing device, and upon receiving an indication that a user has selected a selectable button in the SERP, the computing system provides the query as input to a computer-implemented model that generates content based on text input. For example, the computer-implemented model is a diffuser model that generates images, videos, text, etc. based on text input (and optionally other inputs, such as images, videos, where the input is multimodal). The computer-implemented model generates content based on the query, and the computing system transmits the content to the client computing device for presentation to the user by an application.

[0005] In another example, upon receiving a query, a computing system searches a computer-readable index for items related to the query, and the computing system provides the query as input to the computer-implemented model. In such an example, the SERP includes both 1) the items identified by the computing system as being related to the query and 2) the content generated by the computer-implemented model.

[0006] Further, the computing system can search the computer-readable index based on the content generated by the computer-implemented model. More specifically, the computing system can send the content generated by the computer-implemented model to a client computing device for presentation to a user by an application. The client computing device receives an instruction from a user that a search should be performed based on the content. More specifically, the content may have feature values, and the computing system can search the computer-readable index based on the feature values. The computing system identifies items indexed in the computer-readable index based on the feature values ​​of the content generated by the computer-implemented model. Thus, the computing system can identify existing items similar to the content generated (on-demand) based on the query. The computing system sends the identified items to the client computing device for presentation to a user by an application running on the client computing device.

[0007] While the examples provided above refer to SERPs, it is emphasized that the technology described herein is not limited to inclusion in search engines. Rather, the technology described herein may be used in connection with productivity applications (e.g., when a user is searching for content to include in a word processing document, a slideshow presentation, etc.), and the claims are not limited to search engines unless expressly recited in such claims.

[0008] The foregoing summary presents a simplified overview in order to provide a basic understanding of some aspects of the systems and / or methods described herein. This summary is not an extensive overview of the systems and / or methods described herein. It is not intended to identify key / critical elements or to delineate the scope of such systems and / or methods. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later. [Brief explanation of the drawings]

[0009] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] FIG. 1 is a functional block diagram of a computing system capable of generating search results and dynamically creating content based on queries received from client computing devices. [Figure 2] FIG. 1 is a functional block diagram of a computing system capable of generating search results based on a query and transmitting the query to a second computing system that can generate content based on the query. [Figure 3] A graphical user interface (GUI) for applications running on a client computing device. [Figure 4] A graphical user interface (GUI) for applications running on a client computing device. [Figure 5] A graphical user interface (GUI) for applications running on a client computing device. [Figure 6] A graphical user interface (GUI) for applications running on a client computing device. [Figure 7] 1 is a flow diagram illustrating a method for returning search results and generating content based on a query. [Figure 8]FIG. 1 is a flow diagram illustrating a method for generating a search engine search results page (SERP) that includes selectable buttons corresponding to generating content based on a query. [Figure 9] FIG. 1 is a flow diagram illustrating a method for generating content based on 1) a query and 2) a user's search history indicating that content can be generated for the user. [Figure 10] FIG. 1 is a flow diagram illustrating a method for performing a search on a computer-readable index based on content generated based on a query. [Figure 11] FIG. 1 is a schematic diagram of a computing system. DETAILED DESCRIPTION OF THE INVENTION

[0010] Detailed Description Various techniques relating to content retrieval and query-based content generation are now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are presented in order to provide a thorough understanding of one or more aspects. It will be apparent, however, that such one or more aspects may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to facilitate the description of one or more aspects. Furthermore, it should be understood that functionality described as being performed by a particular system component may be performed by multiple components. Similarly, for example, a component may be configured to perform functions described as being performed by multiple components.

[0011] Also, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, the phrase "X employs A or B" is intended to mean any of the natural inclusive permutations. That is, the phrase "X employs A or B" is satisfied by any of the following cases: X employs A; X employs B; or X employs both A and B. Furthermore, the articles "a" and "an," as used in this application and the appended claims, shall be generally construed to mean "one or more" unless otherwise specified or clear from the context that the singular form is intended.

[0012] Furthermore, as used herein, the terms "component" and "system" are intended to encompass computer-readable data storage comprised of computer-executable instructions that, when executed by a processor, cause a particular function to be performed. Computer-executable instructions may include routines, functions, etc. It should also be understood that a component or system may be localized on a single device or distributed across several devices. Furthermore, as used herein, the term "exemplary" is intended to mean serving as an example or instance of something and is not intended to indicate a preference.

[0013] This specification describes various techniques related to searching for content and generating content based on a query. More specifically, a computing system receives a query from an application executing on a client computing device, the query being submitted to the client computing device by a user of the client computing device. In one example, the computing system executes a general-purpose search engine, and the user provides the query to the search engine. Based on the query, the computing system searches a computer-readable index for items, which may be web pages, images, videos, etc. Further, the computing system provides the query to a computer-implemented model that generates content based on text input. In one example, the computer-implemented model is a diffuser model that generates images based on the input, which may be text or multimodal. The computer-implemented model generates content based on the query (e.g., a prompt), and the computing system returns search results to the client computing device. In one example, the search results include items identified in the search of the computer-readable index and content generated by the computer-implemented model.

[0014] A query can be a relatively complex query that expresses an information search intent that can be identified by a computing system, but there may be few (or no) items indexed by the computing system that satisfy the information search intent. An example query is "photo of a baseball player swinging a baseball bat at a soccer ball." In this example, the query is well-formed (the user's information search intent can be easily ascertained). However, images that exactly match the user's information search intent may not be indexed by the computing system (or may not be easily found in an index of images based on the text query). Using the techniques described herein, the information search intent of a user issuing a query can be satisfied by providing the query (as a prompt) to a computer-implemented model that can generate the requested image based on the query. Furthermore, the computing system can receive an indication that generated content should be used as a query (in a visual search). Accordingly, feature values ​​of the generated content can be calculated, and such values ​​can be used to search a computer-readable index for items (images, videos, etc.). The computing system can search the computer-readable index based on feature values ​​of the generated content and identify items in the computer-readable index based on such values. In other words, the computing system can identify items indexed in the computer-readable index based on content generated by the computer-implemented model, where the identified items are similar to the generated content. Continuing with the example above, the computer-implemented model can generate an image of a baseball player swinging a baseball bat at a soccer ball, and the feature values ​​of such generated image can be used by the computing system to identify previously indexed images that are similar to such image.Thus, if a user is searching for existing images of a baseball player swinging a bat at a soccer ball, the computing system can identify such images based on images generated by a computer-implemented model.

[0015] 1 , a functional block diagram of a computing environment 100 is shown. The computing environment 100 includes a computing system 102 and a client computing device 104 in communication with the computing system 102 via a network 106. The client computing device 104 is operated by a user 108. The client computing device 104 may be any suitable type of client computing device, including, but not limited to, a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone, a smartwatch, augmented reality or virtual reality headgear, etc. The client computing device 104 executes an application capable of receiving a query submitted by the user 108 and transmitting the query to the computing system 102 via the network 106. In one example, the application may be a web browser that has loaded a web page of a general-purpose search engine, and the web page includes a control for receiving a query from the user 108 (the control may be a text entry field, a button to activate a microphone on the client computing device 104, etc.). In another example, the application executed by the client computing device 104 may be a productivity application such as a word processing application, a spreadsheet application, a slideshow presentation application, an email application, a unified communications application, etc. In yet another example, the application may be an instant messaging application, a chat application, or other suitable communications application.

[0016] The computing system 102 includes a processor 110 and a memory 112, with the memory 112 having instructions loaded therein for execution by the processor 110. The computing system 102 also includes a data store 114, which includes a document index 116, an image index 118, and search history data 120. Although not shown, the data store 114 may also include a video index or an index of another type of item. The document index 116 is a computer-readable index that indexes documents by terms associated with such documents. For example, documents may be web pages indexed by terms contained therein, by terms in metadata assigned to the web pages, etc. The image index 118 is a computer-readable index that indexes images by terms and / or feature values ​​associated with the images. The search history data 120 may identify the number of searches performed by a user of the computing system 102, the number of days a user has performed searches using the computing system 102, etc.

[0017] The memory 112 of the computing system 102 includes a search system 122 that searches the document index 116, the image index 118, and / or other suitable computer-readable index (such as a video index) based on a query received by the computing system 102. The search system 122 also generates content based on the query received by the computing system 102, as described in further detail below. The search system 122 returns search results to a client computing device that sends the query to the computing system 102, the search results including items identified based on a search of the indexes in the data store 114 and / or content generated based on the query.

[0018] The search system 122 optionally includes an authorizer module 124 that verifies whether content can be generated on behalf of a user. Generating content based on text or multimodal input can be computationally expensive, and the authorizer module 124 can limit content generation to users who are relatively frequent users of the search system 122. Therefore, before the search system 122 generates content based on a query received from a client computing device operated by the user, the authorizer module 124 can access the search history data 120 and grant or deny the request to generate content based on information in the search history data 120 assigned to the user. If the authorizer module 124 determines that the user has submitted several queries to the search system 122 over a time window greater than a threshold, the authorizer module 124 can authorize the generation of content based on the query received from the client computing device operated by the user. In another example, if the authorizer module 124 determines that a user has submitted a query to the search system 122 for a number of days that exceeds a predefined threshold, the authorizer module 124 may authorize the generation of content based on the query received from a client computing device operated by the user.

[0019] Optionally, the computing system 102 further includes a query classifier 126 that assigns a classification label to a query received by the computing system 102, where the classification label assigned to the query by the query classifier 126 indicates whether content should be (dynamically) generated based on the query. The query classifier 126 may be a binary classifier, where a first label assignable to the query by the query classifier 126 indicates that content should not be generated based on the query, and a second label assignable to the query by the query classifier 126 indicates that content should be generated based on the query. The query classifier 126 may be trained based on queries to which labels have been assigned, where the labels indicate whether content should be generated based on the query. Some queries, such as relatively long and detailed queries like "video of a fish jumping out of water holding a banjo," are well suited to content generation. Other queries, such as factual queries like "George Washington's height" or "webpage of the United States Congress," or queries related to specific web pages, are less suited to content generation. The query classifier 126 is trained to distinguish between queries that are well suited to content generation and queries that are poorly suited to content generation.

[0020] The search system 122 includes an index search module 128 that searches the document index 116 and / or the image index 118 (or other suitable computer-readable index) based on a query received from a client computing device. The index search module 128 can identify items from the indexes 116-118 based on the query and can further rank the identified items to form a list of ranked items.

[0021] The search system 122 further includes a content generator module 130 that generates content based on a query received by the search system 122. More specifically, the content generator module 130 includes a computer-implemented model that generates content based on a textual or multimodal input. In one example, the computer-implemented model is a diffuser model. In one example, the computer-implemented model generates an image based on a prompt. In another example, the computer-implemented model generates a video based on a prompt. In yet another example, the computer-implemented model generates text and / or audio based on a prompt.

[0022] Optionally, the search system 122 further includes a feature identifier module 132 that calculates feature values ​​for content generated by the content generator module 130. For example, images in the image index 118 may be indexed by feature values ​​for such images. As the content generator module 130 generates images based on text input, the feature identifier module 132 may calculate feature values ​​for such images. The feature values ​​for images may indicate colors present in an image, the location of a particular feature within the image, etc.

[0023] The search system 122 also includes a suggestion module 134 that generates query suggestions based on one or more of: 1) content generated by the content generator module 130, 2) search results identified by the index search module 128, 3) previous queries submitted to the search system 122, etc. In one example, the queries suggested by the suggestion module 134 may be queries well suited for use by the content generator module 130 to generate content.

[0024] The search system 122 also includes a visual search module 136 that can perform a visual search, where the visual search module 136 uses the images generated by the content generator module 130 as a basis for searching the image index 118. As described above, the feature identifier module 132 can calculate feature values ​​for the images generated by the content generator module 130. The visual search module 136 can search the image index 118 based on such feature values ​​and can identify images that are similar to the images generated by the content generator module 130.

[0025] The search system 122 also includes a supplemental content module 138 that can identify supplemental content based on at least one of a query received by the search system 122, search results identified by the index search module 128, or values ​​of features of content generated by the content generator module 130. In one example, the supplemental content is an electronic advertisement that is presented with the search results.

[0026] Here is an example of the operation of the search system 122. The search system 122 receives a query submitted by a user 108 of a client computing device 104, the query being received by an application executing on the client computing device 104. Optionally, the authorizer module 124 (either prior to receiving the query or in response to receiving the query) searches the search history data 120 based on an identifier of the user 108 to verify whether the user is authorized to have content generated on their behalf. If the authorizer module 124 determines that content should not be created on behalf of the user 108, the search system 122 does not provide the query to the content generator module 130.

[0027] If the authorizer module 124 determines that content can be created for the user 108, the search system 122 optionally provides the query to the query classifier 126. The query classifier 126 assigns a classification label to the query, which indicates whether the query should be provided to the content generator module 130. If the classification label indicates that the query should not be provided to the content generator module 130, the search system 122 does not provide the query to the content generator module 130.

[0028] Regardless of the output of the authorizer module 124 and the output of the query classifier module 126, the index search module 128 searches a computer-readable index (such as the document index 116, the image index 118, or other suitable index in the data store 114) based on the query. The index search module 128 can identify items indexed in the computer-readable index based on the query (e.g., the index search module 128 identifies items as being relevant to the query).

[0029] If the authorizer module 124 determines that content can be created for the user 108 and the query classifier 126 assigns a classification label to the query indicating that the query should be provided to the content generator module 130, the search system 122 provides the query to the content generator module 130. The content generator module 130 generates content based on the query. In one example, if the query is "baseball player hitting a soccer ball with a baseball bat," the content generator module 130 generates an image including a baseball player hitting a soccer ball with a baseball bat.

[0030] The search system 122 may return search results to the client computing device 104 for presentation by a graphical user interface of an application executing on the client computing device 104, the search results including at least one of items identified by the index search module 128 as relevant to the query or content generated by the content generator module 130 based on the query. For example, the search system 122 may be included in a general-purpose search engine, and the search results may be included in a search engine search results page (SERP) that is provided by the application executing on the client computing device 104 to the client computing device 104 for presentation to the user 108.

[0031] As mentioned above, the content generator module 130 may utilize a significant amount of computing resources when generating content based on a query. Therefore, it may take some time (e.g., 1-5 seconds) for the content generator module 130 to generate the content. The search system 122 may generate an animation to present the content generated by the content generator module 130 along with the items identified by the index search module 128 based on the query. For example, the search system 122 may construct a SERP to include an area for displaying the content generated by the content generator module 130, and the generated content may "fade in" over time (so that the user 108 does not see a blank portion of the SERP until the content generation is complete). In another example, the suggestion module 134 generates query suggestions based on the query received from the client computing device 104, the items identified by the index search module 128 as being related to the query, previous queries submitted by the user 108 or other users of these search systems 122, etc. The query suggestions generated by the suggestion module 134 may be presented in the GUI of an application running on the client computing device 104 until the content generator module 130 completes generating the content. The search system 122 may then update the information presented to the user 108 to include the content generated by the content generator module 130.

[0032] The feature identifier module 132 calculates feature values ​​for the content generated by the content generator module 130, and the feature values ​​may reflect items depicted in the content, named entities in the content, positions of items in the content, gradients in the content, etc. If the content is an image, the feature identifier module 132 may update the image index 118 to index the image generated by the content generator module 130 with the feature values ​​calculated by the feature identifier module 132. Similar operations may occur if the content is video, text, multimodal, etc.

[0033] In some cases, a user 108 may wish to use content generated by the content generator module 130 to find an item included in one of the indexes 116-118 stored in the data store 114. Accordingly, the search system 122 may receive a message from the client computing device 104 indicating that the search system 122 should search one or more of the indexes 116-118 based on the content generated by the content generator module 130. This may occur if the user 108 wishes to find existing content similar to the content generated by the content generator module 130. Upon receiving such a message, the visual search module 136 may, for example, search the image index 118 based on the values ​​of features of the content generated by the content generator module 130. Based on a search of such index 118, the visual search module 136 may identify images indexed in the image index 118, and the search system 122 may return the identified images to the client computing device 104 as updated search results. Thus, again, the search system 122 can use the content generated by the content generator module 130 in connection with searching for content similar to the content generated by the content generator module 130 .

[0034] The supplemental content module 138 can identify supplemental content, such as electronic advertisements, based on queries received by the search system 122, items identified as being relevant to the queries by the index search module 128, values ​​of features of content generated by the content generator module 130, and the like.

[0035] In essence, the search system 122 can then search for existing (indexed) items based on queries received from client computing devices 104, and can additionally (dynamically) generate content based on queries received from client computing devices 104. This can be particularly advantageous in situations where an information search intent is clearly expressed in a query, but items that closely match such intent are not indexed in a computer-readable index available to the search system 122. The techniques described herein relate to improvements over conventional search systems that are limited to searching for existing (indexed) items.

[0036] 2, there is shown a computing environment 200. The computing environment 200 includes a computing system 102 and a client computing device 104. The computing environment 200 further includes a second computing system 202, which includes a processor 204 and a memory 206. The memory 206 stores instructions executed by the processor 204. In the computing environment 200, the memory 206 includes a content generator module 130.

[0037] 1. However, in computing environment 200, rather than search system 122 including content generator module 130, second computing system 202 includes content generator module 130. Thus, when search system 122 receives a query from client computing device 104, computing system 102 can transmit the query to second computing system 202 (if content generator module 130 is to generate content based on the query). In another embodiment, upon receiving the query, computing system 102 can instruct client computing device 104 to transmit the query to second computing system 202. Upon receiving such an instruction, client computing device 104 can transmit the query to second computing system 202.

[0038] The content generator module 130 generates content based on the query and sends the content to either the computing system 102 or the client computing device 104. When the second computing system 202 sends the content to the computing system 102, the search system 122 includes the generated content in search results and sends the search results to the client computing device 104. When the second computing system 202 sends the generated content to the client computing device 104, an application running on the client computing device 104 can incorporate the content into the application's GUI. Figure 2 is provided to illustrate that separate computing systems can each perform search and content generation operations. The client computing device 104 in the environment 200 receives both existing (indexed content) identified based on the query and dynamically generated content based on the query.

[0039] 3-6 illustrate example search engine GUIs that may be presented on a client computing device 104. While the GUIs illustrate particular items and arrangements of items, it should be understood that the technology described herein is not limited to items or arrangements of items. For example, some GUIs illustrate images returned as search results, and it should be understood that the search results may be video search results, multimodal search results, or the like. Furthermore, while one or more of the GUIs illustrate content generated based on text input as being images, such content may be or include text, video, images, any combination thereof, etc. (as discussed above).

[0040] 3, a schematic diagram illustrating a GUI 300 corresponding to a search engine is shown. The GUI 300 includes a text entry field 302 that receives a query submitted by the user 108. In this example, the index search module 128 has identified M images 304-306 and N web pages 308-310 as being relevant to the query. Additionally, the supplemental content module 138 has identified Q electronic advertisements 312-314, and the suggestion module 134 has identified P query suggestions 316-318, e.g., based on the query submitted in the text entry field 302.

[0041] 3, the query classifier 126 assigns a classification label to a query that indicates that the content generator module 130 can generate content based on such a query. Rather than allocating computing resources without receiving a user instruction to generate content, the search system 122 includes a selectable button 320 in the GUI 300. The selectable button 320 indicates to the user 108 that, when selected, content will be dynamically generated based on the query.

[0042] FIG. 4 is a schematic diagram of a GUI 400 that may be generated by the search system 122 when the search system 122 receives an indication that a selectable button 320 has been selected by a user 108. Upon receiving the indication that a user 108 has selected the selectable button 320, the search system 122 provides a query to the content generator module 130. The content generator module 130 generates content based on the query, and the search system 122 returns the generated content for presentation to the user 108 by an application executing on the client computing device 104. As shown in FIG. 4 , the GUI 400 includes generated content 402, which is content generated by the content generator module 130. Such content may be images, video, text, etc. The graphical user interface 400 may further include a selectable hyperlink 404. The selectable hyperlink 404 indicates to the user 108 that a search can be conducted across one or more computer-readable indexes based on the generated content 402 shown in the graphical user interface 400.

[0043] 5 is a schematic diagram of a graphical user interface 500 through which the search system 122 returns both the identified items in the indexes 116-118 and the authored content 402 to the client computing device 104. That is, in contrast to the GUI 300 shown in FIG. 3, the content generator module 130 generates the authored content 402 without receiving explicit instructions from the user 108 to create such content.

[0044] 6, a schematic diagram of another graphical user interface 600 of an application executing on a client computing device 104 is presented. The visual search module 136 receives an indication that a user 108 has selected a hyperlink 404 and searches at least one computer-readable index based on the value of the feature of the created content 402. The graphical user interface 600 includes a field 602 in which updated search results are presented, which may include images, videos, web pages, or the like.

[0045] 7-10 illustrate methods for providing search results based on a query, where the search results include existing items identified based on the query and dynamically generated content based on the query. While these methods are illustrated and described as a series of acts performed in sequence, it should be understood and appreciated that these methods are not limited by the order of the sequence. For example, some acts may occur in a different order than described herein. Additionally, some acts may occur simultaneously with other acts. Furthermore, in some cases, not all acts may be required to implement the methods described herein.

[0046] Additionally, the acts described herein may be computer-executable instructions that may be implemented by one or more processors and / or stored on one or more computer-readable media. Computer-executable instructions may include routines, subroutines, programs, and / or threads of execution, etc. Furthermore, the results of the acts of these methods may be stored on a computer-readable medium and / or displayed on a display device, etc.

[0047] 7, a method 700 is shown for returning indexed search results and generated content to a client computing device based on a query received from the client computing device. Method 700 begins at 702, where a query is received from the client computing device at 704. At 706, a computer-readable index is searched based on the received query. As discussed above, the computer-readable index may index web pages, images, and / or videos, etc.

[0048] At 708, search results are identified from the computer-readable index based on the search. At 710, content is (dynamically) generated based on the query, the content being generated by a computer-implemented model. For example, the computer-implemented model is a diffuser model. At 712, the search results identified at 708 and the content generated at 710 are returned to the client computing device. Method 700 completes at 714.

[0049] 8, a method 800 for returning search results identified based on a query and content generated based on the query is shown. The method 800 begins at 802, where a query is received from a client computing device at 804. A computer-readable index is searched based on the received query at 806. Search results are identified based on the received query at 808.

[0050] A determination is made at 810 whether the query is well suited for submission to a computer-implemented model that generates content based on text input. If it is determined at 810 that the query is not well suited for submission to a computer-implemented model, method 800 proceeds to 812, where search results are returned to the client computing device.

[0051] If, at 810, it is determined that the query is well suited for provision to a computer-implemented model, method 800 proceeds to 814, where the search results and a selectable button are returned to the client computing device. In other words, the search results are returned to the client computing device and a selectable button is presented on the display of the client computing device. The selectable button notifies the user who issued the query that selection of the button will cause content to be generated based on the query.

[0052] At 816, a determination is made whether a selectable button has been selected by the user. If at 816, it is determined that a selectable button has been selected, method 800 proceeds to 818, where the query is provided to a computer-implemented model that generates content based on the text input. At 820, the generated content is retrieved from the computer-implemented model, and at 822, the generated content is returned to the client computing device.

[0053] The method 800 completes at 824 when the search results are returned to the client computing device at 812, when a determination is made that no selectable button is selected at 816, or when the generated content is returned to the client computing device at 822.

[0054] 9 shows a method 900 for returning search results indexed in a computer-readable index based on a query and for returning dynamically generated content based on the query. Method 900 begins at 902, where a query is received from a client computing device at 904. At 906, the computer-readable index is searched based on the received query, and at 908, search results are identified based on the query.

[0055] At 910, a determination is made as to whether the query is well suited for provision to a computer-implemented model that dynamically generates content based on text input. If it is determined at 910 that the query is not well suited for provision to a computer-implemented model, method 900 proceeds to 912, where the search results identified at 908 are returned to the client computing device. If it is determined at 910 that the query is well suited for provision to a computer-implemented model, then at 914, the search history of the query issuer is accessed. At 916, a determination is made as to whether content should be generated based on the search history of the query issuer. If it is determined that content should be generated, method 900 proceeds to 918, where the query is provided to a computer-implemented model that generates content based on text input. At 920, the generated content is retrieved from the computer-implemented model, and at 922, the generated content and the search results identified at 908 are returned to the client computing device. The method 900 completes at 924 if it is determined at 910 that the query is not well suited for submission to a computer-implemented model, if it is determined at 916 that content should not be generated based on the query issuer's search history, or if the generated content and search results are returned to the client computing device at 922.

[0056] Although the method 900 shows that decision 910 occurs before decision 916, it should be understood that decision 916 can occur before decision 910.

[0057] 10, a method 1000 for performing a search based on generated content is shown. The method 1000 begins at 1002, where a query is received from a client computing device at 1004. At 1006, a determination is made that the query can be used by a computer-implemented model to (dynamically) generate content. At 1008, content is generated based on the query, where the content is generated by the computer-implemented model (e.g., a diffuser model).

[0058] At 1010, values ​​of features of the content generated by the computer-implemented model are calculated. At 1012, a request using the content as a query is received from a client computing device. At 1014, a computer-readable index of items is searched based on the values ​​of the features of the content calculated at 1010. At 1016, items are identified based on the search, and at 1018, the items are returned to the client computing device. Method 1000 completes at 1020.

[0059] Referring now to FIG. 11 , a high-level diagram of an exemplary computing device 1100 that can be used in accordance with the systems and methods disclosed herein is shown. For example, the computing device 1100 can be used in a system that searches a computer-readable index. As another example, the computing device 1100 can be used in a system that dynamically generates content based on text input. The computing device 1100 includes at least one processor 1102 that executes instructions stored in memory 1104. The instructions may be, for example, instructions for implementing a function described as being performed by one or more components above or instructions for implementing one or more of the methods described above. The processor 1102 can access the memory 1104 via a system bus 1106. In addition to storing executable instructions, the memory 1104 may also store images, video, text, and the like.

[0060] Computing device 1100 further includes a data store 1108 accessible by processor 1102 via system bus 1106. Data store 1108 may include executable instructions, computer-readable indexes, etc. Computing device 1100 also includes an input interface 1110 that allows external devices to communicate with computing device 1100. For example, input interface 1110 may be used to receive instructions from an external computer device, a user, etc. Computing device 1100 also includes an output interface 1112 that interfaces computing device 1100 with one or more external devices. For example, computing device 1100 may display text, images, etc. via output interface 1112.

[0061] It is contemplated that external devices communicating with computing device 1100 via input interface 1110 and output interface 1112 may be included in an environment that provides virtually any type of user interface with which a user can interact. Examples of types of user interfaces include graphical user interfaces, natural user interfaces, etc. For example, a graphical user interface may accept input from a user using one or more input devices, such as a keyboard, mouse, or remote control, and provide output on an output device, such as a display. Furthermore, a natural user interface may enable a user to interact with computing device 1100 in a manner that is not bound by the constraints imposed by input devices such as a keyboard, mouse, and remote control. Rather, a natural user interface may rely on speech recognition, touch and stylus recognition, on-screen and adjacent-to-screen gesture recognition, air gestures, head and eye tracking, voice and speech, vision, haptics, gestures, artificial intelligence, etc.

[0062] Furthermore, although illustrated as a single system, it should be understood that computing device 1100 may be a distributed system. Thus, for example, several devices may be in communication over network connections and may collectively perform the tasks described as being performed by computing device 1100.

[0063] The various functions described herein may be implemented in hardware, software, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer-readable storage media. A computer-readable storage medium may be any available storage medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. In this specification, the terms "disk" and "disc" include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs (BDs), where a disk typically reproduces data magnetically and a disc typically reproduces data optically using a laser.

[0064] Furthermore, propagating signals are not included within the scope of computer-readable storage media. Computer-readable media also includes communication media, including any medium that facilitates transfer of a computer program from one place to another. For example, a connection can be a communication medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of communication media. Combinations of the above are also intended to be included within the scope of computer-readable media.

[0065] Alternatively or additionally, the functions described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, example types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0066] The technology described herein relates to at least the following examples:

[0067] (A1) In an aspect, a method includes receiving a query from an application executing on a client computing device in network communication with a computing system. The method also includes, in response to receiving the query, 1) searching a computer-readable index for items based on the query, 2) identifying the items based on the search of the computer-readable index, 3) sending the query to a computer-implemented model, and 4) retrieving content generated by the computer-implemented model, where the computer-implemented model generated the content based on the query. The method also includes returning at least one of the items or the content to the client computing device for presentation by the application executing on the client computing device.

[0068] (A2) In some embodiments of the method of (A1), both the item and the content are returned to the client computing device in a search engine search results page (SERP).

[0069] (A3) In some embodiments of at least one of the methods (A1)-(A2), the computer-implemented model is a diffuser model.

[0070] (A4) In some embodiments of at least one of the methods of (A1)-(A3), the method also includes providing the query to a classifier trained to identify queries usable by the computer-implemented model to generate content, the query being provided to the classifier prior to sending the query to the computer-implemented model. The method also includes receiving an indication from the classifier that the query is usable by the computer-implemented model to generate content, the query being sent to the computer-implemented model based on the indication from the classifier.

[0071] (A5) In some embodiments of at least one of the methods of (A1)-(A4), the content is an image generated by a computer-implemented model.

[0072] (A6) In some embodiments of the method of (A5), the image is returned to the client computing device for presentation on a display of the client computing device. The method also includes receiving an indication that a second search should be performed based on the content generated by the computer-implemented model. The method further includes searching a computer-readable index of items based on the indication and the content. The method further includes identifying a second item based on the search of the computer-readable index of items. The method also includes returning the second item to the client computing device for presentation by an application executing on the client computing device.

[0073] (A7) In some embodiments of the method of (A6), the method also includes, upon obtaining the content generated by the computer-implemented model, calculating values ​​of features of the content, wherein a computer-readable index of the item is retrieved based on the calculated values ​​of the features of the content.

[0074] (A8) In some embodiments of at least one of the methods of (A1)-(A7), the item is returned to the client computing device for presentation by an application executing on the client computing device at a first time point. The method also includes generating a query suggestion based on the query. The method further includes returning the query suggestion along with the item to the client computing device for presentation by the application executing on the client computing device. The method further includes returning the content to the client computing device without further user interaction at a second time point subsequent to the first time point.

[0075] (A9) In some embodiments of method (A8), the computing system causes the client computing device to cease presenting query suggestions when the content is returned to the client computing device.

[0076] (A10) In some embodiments of at least one of the methods of (A1)-(A9), the method also includes accessing a search history associated with a user of the client computing device, wherein the query is sent to the computer-implemented model based on the search history associated with the user of the client computing device.

[0077] (A11) In some embodiments of at least one of the methods of (A1) through (A10), the application is a productivity application.

[0078] (B1) In another aspect, a method performed by a computing system includes receiving a query from an application executing on a client computing device in network communication with the computing system. The method also includes returning a search results page to the client computing device for presentation by the application based on the query. The search results include search results identified based on the query and a selectable button corresponding to generation of content based on the query. The method further includes receiving an indication from the client computing device that the selectable button was selected. The method also includes providing the query to a computer-implemented model based on the indication, wherein the computer-implemented model generates and provides content based on the query. The method further includes returning the content to the client computing device for presentation by the application.

[0079] (B2) In some embodiments of the method of (B1), the method is performed by a general-purpose web search engine running on a computing system.

[0080] (B3) In some embodiments of at least one of the methods of (B1)-(B2), the method also includes providing the query to a computer-implemented classifier, the computer-implemented classifier generating an output indicating that the query is usable by the computer-implemented model to generate content. Based on the output generated by the computer-implemented classifier, a selectable button is included in the search result page.

[0081] (B4) In some embodiments of at least one of the methods of (B1)-(B3), the content is a video.

[0082] (B5) In some embodiments of at least one of the methods of (B1) through (B4), the method also includes receiving a request from the application to search the computer-readable index based on the content subsequent to returning the content to the client computing device for presentation by the application. The method also includes identifying items in the computer-readable index based on the content in response to receiving the request. The method further includes returning a second search results page to the client computing device for presentation by the application, the second search results page including the items.

[0083] (B6) In some embodiments of at least one of the methods of (B1) through (B5), the computer-implemented model is a diffuser model.

[0084] (B7) In some embodiments of at least one of the methods of (B1) through (B6), the method also includes determining, based on a user profile of a user of the client computing device, that the computer-implemented model should generate content on behalf of the user. If the computer-implemented model determines that the content should be generated on behalf of the user, a selectable button is included in the search results.

[0085] (C1) In yet another aspect, a method includes receiving a query from a client computing device in a general-purpose web search engine. The method also includes providing the query to a computer-implemented classifier, where the computer-implemented classifier is trained to identify a query to be provided to a computer-implemented model that generates an image based on the query. The method further includes obtaining an output from the computer-implemented classifier indicating that the query should be provided to the computer-implemented model. The method further includes providing the query to the computer-implemented model based on the output from the computer-implemented classifier. The method also includes obtaining a first image from the computer-implemented model, where the computer-implemented model generated the image based on the query. The method further includes identifying search results based on the query. The method further includes generating a search engine search results page (SERP), where the SERP includes the image and search results generated by the computer-implemented model. The method also includes transmitting the SERP to the client computing device for presentation on a display of the client computing device.

[0086] (C2) In some embodiments of the method of (C1), the computer-implemented model is a diffuser model.

[0087] (D1) In another aspect, a computing system includes a processor and a memory, the memory storing instructions that, when executed by the processor, cause the processor to perform at least one of the methods disclosed herein (e.g., at least one of methods (A1)-(A11), (B1)-(B7), or (C1)-(C2)).

[0088] (E1) In yet another aspect, a computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to perform at least one of the methods disclosed herein (e.g., at least one of methods (A1)-(A11), (B1)-(B7), or (C1)-(C2)).

[0089] What has been described above includes examples of one or more embodiments. Of course, it is not possible to describe every conceivable variation and modification of the above-described devices or methods for purposes of describing the foregoing aspects, but those skilled in the art will recognize that many further modifications and permutations of the various aspects are possible. Accordingly, the described aspects are intended to embrace all such modifications, variations, and variations that fall within the spirit and scope of the appended claims. Furthermore, where the term "includes" is used in either the detailed description or the claims, such term is intended to be inclusive in the same manner as the term "comprising" when used as a transitional term in a claim.

Claims

1. 1. A computing system comprising: a processor; a memory, which when executed by the processor, causes the processor to: receiving a query from an application executing on a client computing device in network communication with the computing system; In response to receiving the query, searching a computer readable index of items based on the query; identifying an item based on said searching of said computer readable index; sending the query to a computer-implemented model; and obtaining content generated by the computer-implemented model, the computer-implemented model generating the content based on the query; and returning at least one of the item or the content to the client computing device for presentation by the application executing on the client computing device; a memory storing instructions for performing actions including: A computing system comprising:

2. The computing system of claim 1 , wherein both the item and the content are returned to the client computing device in a search engine search results page (SERP).

3. The computing system of claim 1 or 2, wherein the computer-implemented model is a diffuser model.

4. The act is: before sending the query to the computer-implemented model, providing the query to a classifier trained to identify queries usable by the computer-implemented model to generate content; receiving an indication from the classifier that the query can be used by the computer-implemented model to generate content, wherein the query is sent to the computer-implemented model based on the indication from the classifier; The computing system of any one of claims 1 to 3, further comprising:

5. The computing system of any one of claims 1 to 4, wherein the content is an image generated by the computer-implemented model.

6. The image is returned to the client computing device for presentation on a display of the client computing device, and the act of: receiving an indication that a second search should be performed based on the content generated by the computer-implemented model; searching a computer readable index of the item based on the instructions and the content; identifying a second item based on the search of a computer-readable index of the items; returning the second item to the client computing device for presentation by the application executing on the client computing device; The computing system of claim 5 further comprising:

7. The act is:

7. The computing system of claim 6, further comprising: upon obtaining the content generated by the computer-implemented model, calculating values ​​of features of the content, wherein a computer-readable index of the item is searched based on the calculated values ​​of the features of the content.

8. The item is returned to the client computing device for presentation by the application executing on the client computing device at a first time, and the action comprises: generating query suggestions based on the query; returning the query suggestions along with the items to the client computing device for presentation by the application executing on the client computing device; returning the content to the client computing device without further user interaction at a second time subsequent to the first time; The computing system of any one of claims 1 to 7, further comprising:

9. The act is:

9. The computing system of claim 1, further comprising: accessing a search history associated with a user of the client computing device, wherein the query is sent to the computer-implemented model based on the search history associated with the user of the client computing device.

10. 1. A method performed by a computing system, the method comprising: receiving a query from an application executing on a client computing device in network communication with the computing system; returning a search results page to the client computing device for presentation by the application based on the query, the search results page comprising: search results identified based on the query; and a selectable button corresponding to generating content based on the query; Including, returning and receiving an indication from the client computing device that the selectable button was selected; providing the query to a computer-implemented model based on the instructions, the computer-implemented model generating content based on the query; and returning the content to the client computing device for presentation by the application; A method comprising:

11. The method of claim 10 , wherein the method is performed by a general-purpose web search engine running on the computing system.

12. 12. The method of claim 10 or 11, further comprising providing the query to a computer-implemented classifier, the computer-implemented classifier generating an output indicating that the query can be used by the computer-implemented model to generate the content, and providing the selectable button included in the search results page based on the output generated by the computer-implemented classifier.

13. The method according to any one of claims 10 to 12, wherein the content is a video.

14. subsequent to returning the content to the client computing device for presentation by the application, receiving a request from the application to search a computer-readable index based on the content; In response to receiving the request, identifying an item in the computer-readable index based on the content; returning a second search results page to the client computing device for presentation by the application, the second search results page including the item; and The method of any one of claims 10 to 13, further comprising:

15. 15. The method of claim 10, further comprising: determining, based on a user profile of a user of the client computing device, that the computer-implemented model should generate content on behalf of the user; and determining that the selectable button is included in the search results if the computer-implemented model determines that the content should be generated on behalf of the user.

Images