Visual citations for refining information provided in response to multimodal queries

The method addresses the challenge of inaccurate text-based queries by processing multimodal queries to generate visual citations, improving the accuracy and efficiency of information retrieval through user-verified search results.

JP2024163064AActive Publication Date: 2024-11-21GOOGLE LLC

Patent Information

Application Number
JP2024075870
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-09
Filing Date
2024-05-08
Publication Date
2024-11-21
Estimated Expiration
2044-05-08

AI Technical Summary

Technical Problem

Users often struggle to express their queries accurately using text-based search services, especially when describing unfamiliar objects, leading to inefficiencies and inaccuracies in retrieving relevant information.

Method used

A computer-implemented method that processes multimodal queries, including images and textual prompts, to generate visual citations by identifying visually similar result images and refining linguistic outputs based on user feedback, allowing users to verify the accuracy of search results.

Benefits of technology

Enables users to quickly verify the accuracy of search results by providing visual citations, enhancing the precision of information retrieval and reducing the need for further research.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024163064000001_ABST
    Figure 2024163064000001_ABST
Patent Text Reader

Abstract

To present, to users, information retrieved in response to multimodal queries.SOLUTION: Result images are retrieved based on a similarity to a query image. A set of textual inputs is processed with a machine-learned language model to obtain a language output comprising textual content. The set of textual inputs comprises textual content from source documents that include the result images, and a prompt associated with the query image. The language output and the result images are provided to a user computing device. Information is received descriptive of an indication by a user that a first result image is visually dissimilar to the query image. Textual content associated with the source document that includes the first result image from the set of textual inputs is removed. The set of textual inputs is processed with the machine-learned language model to obtain a refined language output. The refined language output is provided to the user computing device.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates generally to presenting information retrieved in response to a multimodal query to a user, and more particularly, to generating and presenting visual citations to a user for refinement of information retrieved or derived in response to a multimodal query. [Background technology]

[0002] In modern society, text-based search services are ubiquitous, but users often struggle to formulate text-based queries in various situations. For example, users often find it difficult to describe objects that are unfamiliar to them. In another example, users may not be able to properly express their intent (e.g., the intended subject of the query, etc.) in text. To facilitate more efficient and accurate interactions between users and search services, multimodal queries have been proposed. A multimodal query is a query formulated using multiple types or formats of data (e.g., text content, audio data, video data, image data, etc.). For example, a user may provide a search service with a multimodal query that includes an image and an associated text prompt (e.g., an image of a bird and the text query "What kind of bird is this?"). The search service can use various multimodal query processing techniques to retrieve search results such as images and associated text content, and some portions of the text content may be presented to the user to indicate that they are associated with a particular result image. Summary of the Invention [Means for solving the problem]

[0003] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the description that follows, or may be learned from the description, or may be learned through practice of the embodiments.

[0004] One exemplary aspect of the present disclosure is directed to a computer-implemented method. The computer-implemented method includes, by a computing system including one or more computing devices, searching for two or more result images based on similarity between an intermediate representation of a query image and an intermediate representation of two or more result images. The computer-implemented method includes, by the computing system, processing a set of text inputs with a machine learning language model to obtain a linguistic output including text content, the set of text inputs including text content from a source document including the two or more result images and a prompt associated with the query image. The computer-implemented method includes, by the computing system, providing the linguistic output and the two or more result images to a user computing device for display within an interface of the user computing device. The computer-implemented method includes, by the computing system, receiving information from the user computing device describing an indication by a user of the user computing device that a first result image of the two or more result images is visually dissimilar to the query image. The computer-implemented method includes, by the computing system, removing from the set of text inputs the text content associated with the source document including the first result image. The computer-implemented method includes processing, by a computing system, a set of text inputs with a machine learning language model to obtain refined language output. The computer-implemented method includes providing, by the computing system, the refined language output to a user computing device for display within an interface of the user computing device.

[0005] Another exemplary aspect of the present disclosure is directed to a user computing device including one or more processors and one or more non-transitory computer-readable media collectively storing a first set of instructions that, when executed by the one or more processors, cause the user computing device to perform operations. The operations include obtaining a query image. The operations include obtaining text data describing a prompt. The operations include providing the query image and the text data describing the prompt to a computing system associated with a visual search service. The operations include receiving from the computing system in response to providing the query image and the prompt, (a) two or more result images and (b) a linguistic output from a machine learning language model, the linguistic output being generated based on the prompt and textual content from a source document including the two or more result images. The operations include displaying, within an interface of an application executed by the user computing device, (a) an interface element including the linguistic output and (b) two or more selectable attribute elements respectively associated with the two or more result images, each selectable attribute element including a thumbnail of the associated result image and attribute information identifying a source document including the associated result image.

[0006] Another exemplary aspect of the present disclosure is directed to one or more non-transitory computer-readable media collectively storing a first set of instructions that, when executed by one or more processors of the user computing device, cause the user computing device to perform operations. The operations include obtaining a query image. The operations include obtaining text data describing a prompt. The operations include providing the query image and the text data describing the prompt to a computing system associated with a visual search service. The operations include receiving from the computing system in response to providing the query image and the prompt, (a) two or more result images and (b) a linguistic output from a machine learning language model, the linguistic output being generated based on the prompt and textual content from a source document including the two or more result images. The operations include displaying, within an interface of an application executed by the user computing device, (a) an interface element including the linguistic output and (b) two or more selectable attribute elements respectively associated with the two or more result images, each selectable attribute element including a thumbnail of the associated result image and attribute information identifying a source document including the associated result image. The operations include receiving input from a user via an input device associated with the user computing device selecting a first selectable attribute element of the two or more selectable attribute elements. The operations include providing, in response to receiving the input, to the computing system, information indicating a selection of the first selectable attribute element. The operations include receiving, in response to providing the information, a refined linguistic output from the computing system, the refined linguistic output being generated based on the prompt and textual content from the source document including two or more result images other than the first result image associated with the first selectable attribute element.

[0007] Other aspects of the present disclosure are directed to various systems, apparatus, non-transitory computer-readable media, user interfaces, and electronic devices.

[0008] These and other features, aspects, and advantages of various embodiments of the present disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the present disclosure and, together with the description, serve to explain associated principles.

[0009] A detailed discussion of embodiments, directed to persons skilled in the art, is described herein, and this specification refers to the accompanying drawings. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of an exemplary visual search system according to some implementations of the present disclosure. [Diagram 2] FIG. 1 is a dataflow diagram for providing information and accompanying visual citations in response to a visual query, according to some implementations of the present disclosure. [Diagram 3] FIG. 1 is a flowchart diagram of an example method for implementing responses to prompts and generating corresponding visual citations in accordance with some implementations of the present disclosure. [Figure 4] A diagram showing an example interface of a user computing device for displaying textual content and corresponding interface elements in accordance with some implementations of the present disclosure. [Figure 5A] A diagram showing an example interface of a user computing device for displaying textual content and corresponding interface elements in accordance with certain other implementations of the present disclosure. [Figure 5B] 5B illustrates an exemplary interface of a user computing device that is displayed subsequent to the interface of FIG. 5A in response to receiving user input, according to some implementations of the present disclosure. [Figure 6A] A data flow diagram for dynamic refinement of visual search information in response to user feedback in a first time period T1 according to some implementations of the present disclosure. [Figure 6B] A data flow diagram for dynamic refinement of visual search information in response to user feedback in a second time period T2 according to some implementations of the present disclosure. [Figure 7A] A diagram showing an example interface of a user computing device for collecting user feedback on derived text content and corresponding result images in accordance with some implementations of the present disclosure. [Figure 7B] A diagram illustrating an exemplary interface of a user computing device for displaying visual search information refined based on user feedback, according to some implementations of the present disclosure. [Figure 8] FIG. 1 is a flowchart diagram of an example method for providing visual search information derived from documents containing images retrieved based on visual similarity to a query image, according to some implementations of the present disclosure. [Figure 9] 1 is a flowchart diagram of an example method for refining visual search information based on user feedback, according to some implementations of the present disclosure. [Figure 10] FIG. 1 is a flowchart diagram of an example method for implementing collection of user feedback for refining visual search information in accordance with some implementations of the present disclosure. [Figure 11A] FIG. 1 is a block diagram of an example computing system for implementing a visual or multimodal search service in accordance with some implementations of the present disclosure. [Figure 11B] A block diagram of an example computing system for performing visual search operations and / or refining visual search information in accordance with some implementations of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Reference numbers that are repeated among the figures identify like features in various implementations.

[0012] In general, the present disclosure is directed to presenting retrieved information to a user in response to a multimodal query. More specifically, the present disclosure relates to generating visual citations that visually identify sources of retrieved information provided in response to a query, such as a visual or multimodal query, such that a user can refine the information based on the visual citation. A multimodal query is a query that is formulated using various types of data (e.g., text content, audio data, video data, image data, etc.). In response to a multimodal query, a search system can retrieve and / or derive information using various multimodal query processing techniques.

[0013] As an example, assume that a user provides a multimodal query consisting of a query image of a bird and a corresponding prompt, such as "What bird is that?" The visual search system may search for result images that are visually similar to the query image. Based on an assumption that the source of the visually similar result images (e.g., a document containing images and textual content) is likely to contain information related to the query image, the visual search system may extract information from the source of the result images. The visual search system may then derive textual content from the extracted information based on the prompt (e.g., a summary of the information, etc.). For example, the visual search system may process the textual content and the prompt with a machine learning language model to generate a linguistic output that includes the textual content.

[0014] The visual search system can provide the text content and visually similar images for display to a user in an interface of a user computing device. The interface can include an attribute element for the result image. The attribute element can include a representation of the result image (e.g., a thumbnail) and information identifying the source of the image. Following the previous example, if one result image shows the same species of bird as shown by the query image and the source of the result image is a website, the attribute element can include a thumbnail of the result image and information identifying the website (e.g., the website title, URL, etc.). In this way, a user can quickly verify the accuracy of the text content based on the visual similarity between the result image and the query image or by navigating to the source of the result image. For example, if a result image shows a bird that is not obviously the same species as the bird shown by the query image, the user can quickly determine that the corresponding text content provided to the user is relatively likely to be inaccurate.

[0015] In some implementations, a user can select an attribute element to indicate that the corresponding result image is inaccurate, and therefore any information derived from the source of the result image is likely to be inaccurate. Based on the user's selection, the visual search system can derive text content. Following the previous example, the visual search system can search for four result images, each showing a bird. The visual search system can extract information from the source of the four result images, and can process the extracted information along with a prompt provided by the user using a machine learning language model to obtain a language output including the text content. The visual search system can provide the text content and the four attribute elements to a user computing device associated with the user.

[0016] For example, assume that one of the four result images shows a bird that is a distinct species from the bird shown in the query image and the other three result images. The user can select an attribute element that includes the result image (e.g., via a touch screen device, etc.), and the user computing device can indicate the selection of the result image to the visual search system. Previously, the visual search system may have generated the textual content provided to the user by processing the prompt and a corpus of information extracted from the sources of the four result images with a machine learning language model. Thus, in response to the selection of the result image, the visual search system may remove any information extracted from the source of the result image from the corpus of information, and then process the remaining information with the machine learning language model to generate a second language output that includes different textual content. This textual content can be provided to the user computing device. In this manner, the visual search system can iteratively enhance the results based on user feedback on the visual citations.

[0017] Aspects of the present disclosure provide several technical effects and benefits. As one exemplary technical effect and benefit, a search service that can provide a direct response to a query is much more desirable to a user than a service that only provides a list of related documents, which still requires the user to spend significant time and energy to conduct further research. However, most search services that can provide a response to a user query do not provide the user with the ability to verify the accuracy of the response. Lacking the ability to verify the response, many users may refuse to use such a search service.

[0018] However, implementations of the present disclosure allow for the provision of visual citations to users to quickly and efficiently indicate the accuracy of a reply. More specifically, by deriving a response to a query from information associated with result images that are visually similar to the query image, a user can quickly determine the accuracy of the response based on what is shown in the result images. In this manner, implementations of the present disclosure can provide a response to a query while also allowing a user to quickly verify the accuracy of the response.

[0019] It should be noted that, as used herein, "text unit," "text content," and "text" may be used interchangeably. In general, each of the above terms may refer to one or more alphanumeric units. For example, text content, text unit, and text may refer to individual paragraphs, words, single numbers, alphanumeric strings, lines of program code or instructions, machine language, machine-readable code, etc.

[0020] Further, it should be noted that any text, text content, and / or text units referred to herein may be derived from audio data, image data, audiovisual data, and the like. For example, a "document" as further defined herein may be a news article that has been scanned and saved as an image. From such an image, text can be extracted using conventional optical character recognition techniques. Thus, an image showing text may be referred to as text, even if intermediate stage processing techniques have been used to extract the text from the image. This is also applicable to audio and audiovisual media, such as recordings of conversations, videos, podcasts, music, video games with dialogue, and the like. More generally, those skilled in the art will appreciate that spoken utterances, textual descriptions, or any other media from which text can be derived may generally be referred to as "text" throughout this specification.

[0021] Referring now to the drawings, exemplary embodiments of the present disclosure will be discussed in further detail.

[0022] 1 shows a block diagram of an exemplary visual search system 100 according to some implementations of the present disclosure. More specifically, a user computing device 102 may include an input device 104 and a communication module 106. The input device 104 may be or may otherwise include a device (e.g., a microphone, a camera, a touch screen, a physical button, an infrared camera, a mouse, a keyboard, etc.) that can directly or indirectly receive input from a user. The communication module 106 may be or may otherwise include hardware and / or software collectively configured to communicate with a visual search computing system 108 via a network 110. For example, the communication module 106 may include a device that facilitates wireless connection to a network.

[0023] The user computing device 102 can obtain a query image 112 and an associated prompt 114. The query image 112 can be a selected image that serves as a query to the visual search computing system 108. For example, a user of the user computing device 102 can capture the query image 112 using an input device 104 of the user computing device 102. Alternatively, the user can obtain the query image 112 in some other manner (e.g., by performing a screen capture, downloading an image, creating an image with an image creation tool, etc.).

[0024] The prompt 114 associated with the query image 112 may include textual content provided by a user. For example, the user may directly prompt the textual content of the prompt 114 via a keyboard or some other input method included in the input device 104. Alternatively, the user may provide the prompt 114 indirectly. For example, the user may generate a spoken utterance, and the user computing device 102 may capture the spoken utterance with the input device 104. The user computing device 102 may process the spoken utterance using speech recognition techniques (e.g., machine learning text-to-speech models, etc.) to generate the prompt 114.

[0025] In some implementations, the query image 112 may be provided without a corresponding prompt 114, and the user computing device 102 may determine an appropriate prompt associated with the query image 112. For example, if the query image 112 shows a bird as the subject matter of interest in the image, the user computing device 102 may select an appropriate prompt 114 (e.g., "identify this object," "describe this," "tell me more," etc.) to provide with the query image 112. Alternatively, in some implementations, the user computing device 102 may modify a prompt 114 provided by a user of the user computing device 102. For example, the user computing device 102 may modify the prompt 114 to add contextual information to the prompt (e.g., time of day, geolocation, user information, information describing the preceding query image and / or the user-provided prompt, etc.).

[0026] The user computing device 102 can provide a visual search request 116 to the visual search computing system 108 over the network 110. The visual search request 116 can include a query image 112 and a prompt 114.

[0027] The visual search computing system 108 may include a visual search module 118. The visual search module 118 may process the visual search request 116 to obtain textual content 120 and result images 122. The textual content 120 may respond to the prompt 114 and the query image 112. For example, if the query image 112 shows an animal and the prompt 114 is "what is this animal", the textual content 120 may provide a response to the prompt (e.g., identifying the species of the animal or, in the case of a known animal, the name of the animal). The result images 122 may be images that are visually similar to the query image 112. These result images 122 are included in the document from which the textual content 120 was derived. The visual search module 118 may extract at least a portion of the text included in one or more of the documents. In some implementations, the visual search module 118 may perform various processing techniques to identify portions of text within the document that are likely to be related to the result images included in the document.

[0028] More specifically, the visual search module 118 can search for the result images 122 based on similarities between the query image 112 and the result images 122. For example, the visual search module 118 can include a machine learning based model that can be used to identify images that are visually similar to the query image 112. The visual search module 118 can obtain text from a document that includes the result images. The document can be any type or modality of source material that includes the result images, such as a website, a journal, a book, a newspaper, an article, a social media post, a transcript, a blog, etc., as described herein. In some implementations, the visual search module 118 can select the text content 120 from text extracted from the document. Additionally or alternatively, in some implementations, the visual search module 118 can derive the text content 120 from text extracted from the document. For example, the visual search module 118 can include or otherwise have access to a machine learning based model, such as a large-scale language model, and can process the text extracted from the document and prompts to obtain the text content 120.

[0029] The visual search module 118 may provide interface data 124 to the user computing device 102 over the network 110. The interface data 124 may include textual content 120 and result images 122. For example, the interface data 124 may include instructions for highlighting the textual content 120 and for including thumbnail representations of the result images 122, such that a user of the user computing device 102 can easily verify the accuracy of the result images 122, and, accordingly, the accuracy of the textual content 120. In this manner, the visual search computing system 108 can facilitate rapid and accurate verification of a reply by a user while providing a response to a query.

[0030] 2 illustrates a data flow diagram 200 for providing information and accompanying visual citations in response to a visual query, according to some implementations of the present disclosure. More specifically, a visual search computing system 202 (e.g., a physical server computing system, a cloud computing system, a virtualized and / or physical computing node in a network (e.g., an edge computing node, etc.)) may include a visual search module that can obtain a query image 206 and a prompt 208 from a user computing device. For example, a user computing device 203 can provide a visual search request to the visual search computing system 202 over a network.

[0031] In some implementations, the query image 206 may be received from the visual search computing system 202 without an associated prompt. In such a situation, the visual search module 204 may determine to generate a likely prompt associated with the query image 206. For example, the visual search module 204 may include a machine learning based semantic image model trained to generate a semantic description of the query image 206. In some implementations, the visual search module 204 may use the semantic description of the query image 206 as the prompt 208. Alternatively, in some implementations, the visual search module 204 may process the semantic description of the query image 206 with another machine learning based model, such as a large-scale language model, to generate the prompt 208.

[0032] In some implementations, the visual search module 204 may modify the prompt 208 received from the user computing device 203. For example, the visual search module 204 may modify the prompt 208 to add contextual information to the prompt (e.g., the time of day, the geolocation of the user computing device 203, stored user information associated with the user of the user computing device 203, information describing the preceding query image and / or the prompt provided by the user computing device 203, etc.).

[0033] The visual search module 204 may include an image assessment module 210. The image assessment module 210 may implement various processing techniques to identify result images 212 that are visually similar to the query image 206. For example, the image assessment module 210 may include a machine learning visual search model 214 that is trained to identify images that are visually similar to the query image 206 from a corpus of stored image data. For example, in some implementations, the visual search model 214 may be a machine learning coding model, such as an embedding model, that may be used to generate an intermediate representation (e.g., an embedding, etc.) of the query image 206. The image assessment module 210 may include or have access to an image search space 215. The image search space 215 may include intermediate representations for a plurality of stored images. For example, the image search space 215 may be an embedding space that includes embeddings generated for images stored in a data store (e.g., a database, etc.) that stores and indexes a large number of images to facilitate visual search services. The image assessment module 210 may select the result image 212 having an embedding in the embedding space that is closest to the embedding generated for the query image 206 .

[0034] It should be noted that this example illustrates a single result image 212 only to more clearly illustrate an exemplary implementation of the present disclosure, however, such implementation is not limited to obtaining a single result image 212. Instead, the result image 212 may be any number of result images obtained according to the similarity between the result image and the query image 206.

[0035] As previously described, the visual search module 204 can index a large number of images to facilitate visual search services. The visual search module 204 can also index information indicative of source documents that include or are otherwise associated with the result images in the document indexing information 216. A document, as described herein, can be any type or modality of source material, such as a website, journal, book, newspaper, article, social media post, transcript, blog, etc., that includes the result images. A result image can be "associated" with a document if the result image is generated, created, hosted, etc., by the same entity as the document. For example, a document. A result image can be associated with a document if, for example, the result image is used as cover art for the document, is derived from the document (e.g., output of a generative model, etc.), is a frame of a video from which the document was transcribed, etc. A result image can be "included" in a document if it is currently in the document or was in the document when the result image and / or document was indexed.

[0036] The resultant image 212 may be included in a document 220. The document 220 may include the resultant image 212 and textual content 222. In some implementations, the document indexing information 216 may include the document 220 that includes or is otherwise associated with the resultant image 212, or may include textual content extracted from the document 220. Additionally or alternatively, in some implementations, the document indexing information 216 may describe a source location of the document 220 (e.g., a file location in a network, a website URL, an FTP address, etc.). Additionally or alternatively, in some implementations, the document indexing information 216 may include a compressed version of the document 220.

[0037] As described with respect to the result image 212, the present example illustrates a single document 220 only to more clearly illustrate an example implementation of the present disclosure. However, such implementations are not limited to obtaining a single document 220. Instead, in some implementations, multiple documents 220 may include multiple respective result images 212 (e.g., five documents for five result images). Additionally or alternatively, in some implementations, a single document 220 may include multiple result images 212. Additionally or alternatively, in some implementations, multiple documents 220 may each include an instance of a single result image 212.

[0038] As a specific example, assume that the query image 206 shows a speedboat, and that the image assessment module 210 selects a result image 212 that shows the same speedboat from a different angle. If the selected result image 212 is hosted or was originally hosted within a website (i.e., a document) for speedboat enthusiasts, the visual search module 204 may store a link to the website, an archived version of the website, or text content extracted from the website in the document indexing information 216. More generally, the visual search module 204 can store information indicative of an association between a document and the corresponding result image in the document indexing information 216.

[0039] The visual search module 204 may include a document content selection module 218. The document content selection module 218 may search the document 220, including the result image 212. Once the document 220 is searched, the document content selection module 218 may extract a first text unit 224 from the text content 222 of the document 220. The first text unit 224 may include some or all of the text content 222 of the document 220. In some implementations, the document content selection module 218 may perform various processing techniques to identify portions of text within the text 224 that are likely to be related to the result image 212 contained in the document 220. For example, if the document 220 is an online article and the result image 212 is in the middle of the online article, the document content selection module 218 may heuristically select text (e.g., paragraphs, sentences or word counts, lines, etc.) that precede or follow the result image for inclusion in the first text unit 224. Alternatively, in some implementations, the document content selection module 218 can extract all of the text contained within the document for inclusion in the first text unit 224 .

[0040] The visual search module 204 may include a text determination module 226. The text determination module 226 may determine the second text unit 228 based on the first text unit 224 and the prompt 208. In some implementations, the text determination module 226 may determine the second text unit 228 using a machine learning language model 230. For example, the text determination module 226 may process the first text unit 224 and the prompt 208 to obtain the second text unit 228. In some implementations, the machine learning language model 230 may be a large-scale language model trained on a large corpus of training data to perform multiple generative tasks. Furthermore, in some implementations, the machine learning language model 230 may have undergone additional training iterations to tune or optimize the model to a specific implementation of the language task related to the generation of the second text unit 228.

[0041] The visual search module 204 may include an interface data generation module 232. The interface data generation module 232 may generate and transmit interface data 234 to the user computing device 203. The interface data 234 may include the second text unit 228 and the result image 212. The interface data 234 may indicate a manner in which the second text unit 228 and the result image 212 are to be displayed within an interface of the user computing device 203. For example, the interface data 234 may indicate a manner for displaying the second text unit 228 and the result image 212 within an interface of an application (e.g., a visual search application, etc.) executed by the user computing device 203. The display of the second text unit 228 and the result image 212 within an interface of an application executed by the user computing device 203 is discussed in more detail with respect to Figures 4, 5A, and 5B.

[0042] In some implementations, the interface data generation module 232 can generate attribute information 236. The attribute information 236 can be or otherwise include information that identifies the document 220. For example, if the document 220 is a news article, the attribute information 236 can be the title of the news article and the name of the publishing news organization. In another example, if the document 220 is an academic paper, the attribute information 236 can be the title of the academic paper, the first author, a list of authors, literature citations, etc. In yet another example, if the document 220 is a website or some other form of document accessible to the user computing device 203, the attribute information 236 can include a link (e.g., a URL, etc.) that facilitates access to the document 220.

[0043] 3 is a flow chart diagram of an example method 300 for performing responses to prompts and generating corresponding visual citations, according to an example embodiment of the present disclosure. Although FIG. 3 depicts operations performed in a specific order for purposes of explanation and discussion, the methods of the present disclosure are not limited to the specifically depicted order or sequence. Various operations of method 300 may be variously omitted, rearranged, combined, and / or adapted without departing from the scope of the present disclosure.

[0044] At 302, a computing system can search for result images based on similarity between the query image and the result images. In some implementations, to search for result images, the computing system can process the query image with a machine learning visual search model to obtain an intermediate representation of the query image, and select result images based on similarity between the intermediate representation of the query image and the intermediate representation of the result images. For example, the machine learning visual search model can be a machine learning embedding model that generates an embedding of the query image for an image embedding space. The computing system can evaluate an embedding space that includes an embedding of the result image and a number of other image embeddings to search for result images. The embedding of the result image can be an embedding of the query image embedding in the embedding space.

[0045] In some implementations, prior to retrieving the result images, the computing system can obtain a query image from a user computing device. In some implementations, the interface includes a user interface of an application executed by the user computing device. For example, the user computing device can execute a visual search application associated with a visual search service provided by the computing system. The visual search application can facilitate capture of the query image and a prompt for transmission to the computing system.

[0046] In some implementations, obtaining the query image may include obtaining the query image and a prompt associated with the query image from a user computing device. Further, in some implementations, the prompt may be modified by the computing system. For example, the prompt may be modified to include instructions that facilitate processing of the prompt by a large-scale language model.

[0047] In some implementations, retrieving the result images further includes providing the result images to the user computing device for display within an interface, and receiving a prompt from the user computing device in response to providing the result images, the prompt being associated with the query image. For example, the computing system may receive the query image and perform a visual search to obtain the result images. The computing system may provide the result images for display within an interface of an application executed by the user computing device. In response, a user of the user computing device may input a query into the user computing device, which may be provided to the computing system.

[0048] At 304, the computing system can obtain a first text unit, the first text unit including at least a portion of the text content of the source document including the result image. In some implementations, the document includes one or more web pages of a website, an article, a newspaper, a book, or a transcript. For example, if the source document is a website article, the computing system can obtain a first text unit including the text content of the article, the title of the article, other text content associated with the website hosting the article, etc.

[0049] At 306, the computing system can determine a second text unit in response to a prompt associated with the query image, the second text unit including one or more of: (a) at least a portion of the first text unit; (b) text derived from the first text unit. In some implementations, determining the second text unit in response to a prompt associated with the query image includes processing the second text unit and the prompt associated with the query image with a machine learning language model to obtain a language output including the second text unit. In some implementations, the second text unit includes a subset of the first text unit. In some implementations, the second text unit includes text derived from the first text unit, and the text derived from the first text unit describes a summary of the first text unit.

[0050] In some implementations, prior to determining the second text unit, the computing system can generate a prompt associated with the query image based at least in part on the query image. For example, the computing system can process the query image with a machine learning model, such as a semantic image analysis model, to generate a semantic output that describes the query image. The computing system can use the semantic output as the prompt.

[0051] At 308, the computing system may provide the result image and the second text unit for display within the interface. For example, the computing system may transmit the result image and the second text unit to the user computing device that provided the query image and the prompt. In some implementations, providing the result image and the second text unit includes providing data describing an interface element including the second text unit and an attribute element including attribute information identifying a document including the result image for display within the interface. In some implementations, the document includes a web page and the attribute information includes an address of the web page. Alternatively, in some implementations, the document includes a magazine and the attribute information includes a citation indicating a location of the result image within the magazine.

[0052] 4 illustrates an example interface 400 of a user computing device for displaying textual content and corresponding interface elements according to some implementations of the present disclosure. FIG. 4 is discussed in conjunction with FIG. 2. More specifically, a visual search request 402 may include the query image 206 and the prompt 208 of FIG. 2. The visual search request 402 may be provided to the visual search computing system 202 as described with respect to FIG. 2. The visual search computing system 202 may process the visual search request 402 to obtain interface data 234. The interface data 234 may include the second text unit 228 and the result image 212.

[0053] In particular, the interface data 234 may indicate the manner in which the result image 212 and the second text unit 228 are displayed within an interface 400 of the user computing device 203. For example, the user computing device 203 may be running a visual search application, or may already be running an application integrated into the operating system of the user computing device 203. The application may display the interface 400 on a display device of the user computing device 203.

[0054] Following the illustrated example, the query image 206 may show a particular breed of dog, such as a beagle. The prompt 208 may be a question, such as "What breed of dog is this?" The visual search computing system 202 may process the visual search request 402 to obtain a second text unit 228 and a result image 212, which may be included in the interface data 234. As shown, the second text unit 228 may include a response to the query posed by the prompt 208, such as "Response: Beagle." Similarly, the result image 212 may be searched for by visual similarity between the query image 206 and the result image 212.

[0055] In some implementations, the interface data 234 may describe the manner in which the second text unit 228 and the resultant image 212 are presented. For example, the interface data 234 may indicate that the second text unit 228 should be presented within the primary interface element 404 in a manner that emphasizes the second text unit 228. The interface data 234 may further indicate that the primary interface element 404 should include an attribute element 405.

[0056] The attribution element 405 may include the result image 212 and corresponding attribution information 236. The attribution information 236 may identify the document that includes the result image 212 (e.g., document 220 of FIG. 2) from which the second text unit 228 was extracted or derived. If the document is a website or otherwise accessible by the user computing device 203, the attribution information 236 may also provide a link to access the document. In this manner, the attribution element 405 may function as a “visual citation” by which a user of the user computing device 203 can easily verify the accuracy of the second text unit 228.

[0057] In some implementations, in addition to the result image 212 included in the document from which the second text unit 228 was derived, the interface data may include multiple other result images 212. The interface data 234 may indicate instructions for displaying the other result images 212 in the result image elements 406A, 406B, 406C, and 406D (general, the result image elements 406). In some implementations, the result images 212 included in the result image elements 406 may be result images that are less similar to the query image 206 than the result images included in the primary interface element 404. For example, the visual search computing system 202 may select five result images 212 for inclusion in the interface data 234. The result image 212 that is most similar to the query image 206 (e.g., the image with the closest embedding in the embedding space to that of the query image 206) may be shown for inclusion in the primary interface element 404. The result image element 406 may include the other four result images 212.

[0058] Similar to the main interface elements 404, the result image elements 406 may include attribute information 236 that identifies the document that contains the result image 212 of the result image element 406. Following the illustrated example, each result image element 406 may include a link to the website document that contains the result image 212 contained within the respective result image element 406.

[0059] 5A illustrates an example interface 500A of a user computing device for displaying textual content and corresponding interface elements according to some other implementations of the disclosure. FIG. 5 is discussed in conjunction with FIG. 2 and FIG. 4. A visual search request 502 may be provided to the visual search computing system 202 as described with respect to FIG. 2. The visual search request may include the query image 206 and the prompt 208. The visual search computing system 202 may process the visual search request 502 to obtain interface data 234. The interface data 234 may include the second text unit 228 and the result image 212.

[0060] Following the illustrated example, the query image 206 may show a particular type of passenger jet. The prompt 208 may be a statement such as "is it a good plane?", which may or may not function as a query. The visual search computing system 202 may process the visual search request 502 to obtain a second text unit 228, a result image 212, and attribute information 236, which may be included in the interface data 234. As shown, the second text unit 228 may include a response to the query posed by the prompt 208. Similarly, the result image 212 may be searched for by visual similarity between the query image 206 and the result image 212.

[0061] The interface 500A is similar to the interface 400 of FIG. 4, except that the interface 500A may display interface elements in a format that differs from the format in which the interface elements are displayed in the interface 400 of FIG. 4. For example, in FIG. 4, the primary interface element 404 includes the textual content from the second text unit 228 and the attribute element 405 in a format that provides a clear response to the query posed by the user. However, unlike the primary interface element 404, the primary interface element 504 of FIG. 5A includes the first portion 228A of the textual content that includes an excerpt from the first document that provides more contextual information regarding the query posed by the user in the prompt 208. Additionally, the primary interface element 504 includes an emphasis element 506 that highlights or emphasizes information that is expected to serve as a response to the query posed by the prompt 208. The first document may be identified by the attribute element 505 in the same manner as described with respect to the attribute element 405 of FIG. 4.

[0062] Specifically, when processing the prompt 208, the visual search computing system 202 may determine whether to generate interface data 234 for an interface element that includes a direct response to the query, such as interface element 404 of FIG. 4, or an interface element that includes contextual information that may assist the user, such as interface element 504 of FIG. 5A. This determination may be based on a degree of certainty associated with the information retrieved in response to the prompt 208, a semantic understanding of the prompt 208, and the like. In the illustrated example, because "is it a good plane" is a relatively subjective question, the visual search computing system 202 may determine to generate the illustrated interface data 234 based on a semantic understanding of the prompt 208. Determining the type, style, format, and the like of interface elements to include in the interface data 234 is discussed in more detail with respect to FIGS. 6A-7B.

[0063] In some implementations, the interface data 234 may include information for display in multiple interface elements. In other words, the interface data 234 may include, or be used to generate, multiple interface elements that include different text content. According to the illustrated example, the interface data 234 may include information for inclusion in the primary interface element 504 and the second interface element 508. The second text unit 228 included in the interface data 234 may include first text content from a first document and second text content from a second document. The first text content may be provided for inclusion in the primary interface element 504 and the second text content may be provided for inclusion in the second interface element 508.

[0064] The visual search computing system 202 may make a determination as to whether to generate interface data 234 that includes information for inclusion in a plurality of interface elements. As with determining the format for the primary interface element 504, this determination may be made based on a semantic understanding of the prompt 208, the amount, quality, and / or semantic understanding of the text retrieved in response to the prompt 208, etc. Additionally, in some implementations, the visual search computing system 202 may determine the order in which the interface elements 504 and 508 are to be presented to the user.

[0065] Assume that a user of the user computing device 203 does not find the information presented in the primary interface element 504 sufficient. The user may provide an input 510 to the user computing device 203 that instructs the user computing device 203 to display the secondary interface element 508. Following the illustrated example, the user may provide a "swipe" touch input 510 that moves the secondary interface element 508 from a position where most of the element is hidden to a position where the entirety of the element is visible.

[0066] For example, Figure 5B illustrates an exemplary interface 500B of a user computing device that is displayed subsequent to the interface 500A of Figure 5A in response to receiving a user input, according to some other implementations of the present disclosure. Turning to Figure 5B, the interface 500B is displayed in response to receiving an input 510 from a user. As shown, in the interface 500B, the primary interface element 504 has been shifted to a fully hidden position, while the secondary interface element 508 has been shifted to a fully visible position. Like the primary interface element 504, the secondary interface element may include a second emphasis element 512 that emphasizes, highlights, or otherwise indicates portions of the second textual content 228B that are predicted to be particularly relevant to the prompt 208.

[0067] In some implementations, the interface 500B of the user computing device 203 may include an information request element 514 that the user can select to indicate a request for additional information. For example, assume that the visual search computing system 202 determines that the information included in the second text unit 228 is relatively likely to be sufficient for the prompt 208. Rather than continuing to search for information for inclusion in a third, fourth, or fifth interface element, the visual search computing system 202 may determine to include only the first text content 228A and the second text content 228B in the interface data 234 to reduce consumption of computational resources (e.g., computation cycles, memory usage, power, storage, bandwidth, network resources, etc.), reduce latency, and increase efficiency.

[0068] However, if the user determines that the information contained in the primary interface element 504 and the secondary interface element 508 is insufficient, the user may select an information request element 514. Upon selecting the information request element 514, the user computing device 203 may send a request to the visual search computing system 202. In response, the visual search computing system 202 may generate additional interface data for inclusion in a third interface element (or more). In that way, the visual search computing system 202 may facilitate repeated exploration of information in response to the prompt 208 while eliminating unnecessary use of computational resources.

[0069] 6A is a data flow diagram for dynamic refinement of visual search information in response to user feedback in a first time period T1 according to some implementations of the present disclosure. In particular, a visual search computing system 602 (e.g., such as the visual search computing system 202 of FIG. 2) may include a visual search module 604 (e.g., such as the visual search module 204 of FIG. 2). In the first time period T1, the visual search computing system 602 may obtain a query image 606 and a prompt 608, and the query image 606 and the prompt 608 may be processed in the visual search module 604.

[0070] More specifically, at a first time T1, the visual search module 604 may process the query image 606 with an image assessment module 610 as described with respect to previous figures, such as the image assessment module 210 of FIG. 2, to obtain a result image 612. The result image 612 may include a first result image 612A, a second result image 612B, and a third result image 612C. The visual search module 604 may obtain text units 614 from a document associated with the result image 612. Specifically, the visual search module 604 may use a document content selection module 616 to obtain information from the document that includes the result image 612 based on document indexing information 618. The document indexing information 618 may store information indicative of the document in which the result image was when indexed by the visual search computing system.

[0071] To follow the illustrated example, assume that the result image 612A was included in two separate documents 618A and 618B when indexed by the visual search computing system 602. The document indexing information 618 can store the text content included in the documents 618A and 618B at the time of indexing, or may store information indicating the location (e.g., URL, download link, file location, etc.) from which to access the documents 618A and 618B. For example, if the documents 618A and 618B are published journal articles, the visual search computing system 602 may store the text content directly included in the documents, since the text content is relatively unlikely to change over time. Conversely, if the documents 618A and 618B are both website pages, the visual search computing system 602 may store the URL from which the documents 618A and 618B can be accessed, since the information included in the website pages is relatively likely to be updated or repeated over time.

[0072] Continuing with the previous example, text unit 614 may include a first text unit 614A, a second text unit 614B, and a third text unit 614C. Each of the text units may include text content contained in the document that includes result image 612. For example, because result image 612A is contained in documents 618A and 618B, text unit 614A corresponding to result image 612A may include text content from both documents 618A and 618B. Text unit 614B corresponding to result image 612B may include text content from document 618C that includes result image 612B. Text unit 614C may include text content from document 618D that includes result image 612C.

[0073] The visual search computing system 602 can process the text units 614 and the prompts 608 in a text determination module 620 to obtain derived text units 622, as described with respect to the text determination module 226 of FIG. 2. More specifically, the text determination module can process a set of text inputs including (a) the text units 614 and (b) the prompts 608 to obtain derived text units 622. For example, the text determination module 620 can include a large-scale language model 621. The large-scale language model 621 can be a model trained on a large and diverse corpus of data for performing various types of language tasks. The large-scale language model 621 can process the set of text inputs to generate derived text units 622. The set of text inputs can include the text units 614 and the prompts 608.

[0074] In some implementations, the derived text units 622 may be linguistic output from a machine learning language model included in the text determination module 620. Thus, the derived text units 622 may be generative linguistic output that is generated from, but includes some textual content not included in, the text units 614. Additionally or alternatively, the derived text units 622 may be linguistic output that includes some (or all) of the textual content of the text units 614.

[0075] The visual search computing system 602 may provide the result images 612 and derived text units 622 to a user computing device 624 for display within an interface of the user computing device 624. Additionally, in some implementations, the visual search computing system 602 may provide attribute information 626 to the user computing device 624 along with the result images 612 and derived text units 622. For example, the attribute information 626 may include information stored in document indexing information 618 that identifies and / or provides access to documents 618A-618D.

[0076] In some implementations, in response to receiving the result images 612, the derived text units 622, and the attribute information 626, the user computing device 624 may provide result image selection information 628 to the visual search computing system 602. The result image selection information 628 may be information generated in response to user input collected at the user computing device 624 selecting one of the result images 612 within an interface to indicate that the selected result image 612 is incorrect.

[0077] For example, turning to Figure 7A, Figure 7A illustrates an example interface 700A of a user computing device for collecting user feedback on derived textual content and corresponding result images, according to some implementations of the present disclosure. Figure 7A is discussed with respect to Figure 6A. In particular, assume that the query image 606 is an image of a passenger jet and the prompt 608 is the query "What is the maximum distance?" In response, the visual search computing system 602 can generate and provide attribute information 626, derived text units 622, and result images 612 for display at the interface 700A of the user computing device 624.

[0078] User computing device 624 can display this information in interface 700A. Interface 700A may include an interface element 702 that includes a derived text unit 622. Following the illustrated example, derived text unit 622 may include information related to a maximum distance of a passenger jet aircraft shown in query image 606 retrieved in response to prompt 608. Here, derived text unit 622 is information related to a maximum distance of a passenger jet aircraft summarized from multiple source documents.

[0079] Additionally, interface 700A may include selectable attribute elements 704A, 704B, and 704C (generally, selectable attribute element 704). Selectable attribute element 704 is an interface element that includes a result image and attribute information that identifies a document that includes the result image. In particular, the document identified by selectable attribute element 704 is the document from which derived text unit 622 was derived. Based on the assumption that the text content of the document is closely related to the image contained therein, a user can quickly and efficiently evaluate the relevance of the document used to derive derived text unit 622 by viewing the result image included in the attribute element associated with the document. To indicate that the result image (and therefore the document that includes the result image) is not relevant, the user may select the selectable attribute element 704 that includes the result image.

[0080] For example, selectable attribute element 704A includes result image 612A and attribute information 626 that indicates the identity of the document (e.g., document 618A) that contains result image 612A. Because result image 612A included in selectable attribute element 704A is an exact visual match to query image 606, the user is unlikely to select selectable attribute element 704A. However, result image 612B included in selectable attribute element 704B is clearly visually dissimilar to query image 606 because query image 606 shows a passenger jet plane and result image 612B shows a fighter jet. This visual mismatch may cause the user to provide an input 706 that selects attribute element 704B.

[0081] 7A, the textual content included in document 618C (i.e., the "source" of result image 612B) that includes result image 612B is related to fighter aircraft rather than passenger jets, and is therefore not relevant to prompt 608 and query image 606. Because derived text unit 622 is generated based at least in part on textual content from document 618C, there is a relatively high likelihood that derived text unit 622 is at least partially inaccurate. This is illustrated in the summarized information included in interface element 702, which includes information related to fighter aircraft (e.g., "The F-37 is in a VTOL configuration and takes off relatively easily from aircraft carriers," "US allies have purchased over 200 F-37 planes," etc.).

[0082] By providing an input 706 selecting attribute element 704B, the user can indicate to the user computing device 624, and thus the visual search computing system 602, that the document including result image 612B is not relevant to the prompt 608 and therefore should not be used to generate derived text unit 622. In response to receiving the input 706, the user computing device 624 can generate and provide result image selection information 628 to the visual search computing system 602.

[0083] 6B, which is a data flow diagram for dynamic refinement of visual search information in response to user feedback at a second time period T2, according to some implementations of the present disclosure. Specifically, the visual search computing system 602 may receive result image selection information 628. The result image selection information 628 may indicate that the result image 612B is not visually similar to the query image 606. In response, at time T2, the visual search computing system 602 may generate a second derived text unit 630, which is generated based on each of the previous text units 614, except for the text unit 614B extracted from the document 618B that contained the result image 612B.

[0084] The visual search module 604 may receive result image selection information 628 indicating a selection of attribute element 704B that includes result image 612B. In response, the visual search module 604 may identify each text unit previously used to generate derived text unit 622. The visual search module 604 may then remove any text units obtained from document 618C that served as a source document for result image 612B (e.g., the document that included result image 612B).

[0085] For example, to generate derived text unit 622, the visual search module may have processed a first set of text inputs that included text unit 614A, text unit 614B, text unit 614C, and prompt 608. In response to the result image selection information, the visual search module 604 may determine a second set of text inputs that includes each text unit of the first set of text inputs other than text unit 614B associated with result image 612B included in the selectable attribute element 704B indicated by result image selection information 628. Here, the second set of text inputs may include text unit 614A, text unit 614C, and prompt 608.

[0086] Upon determination, the visual search module 604 may process the second set of text inputs with the large-scale language model 621 to generate a second derived text unit 630. Because the second derived text unit 630 is not based on information indicated by the user as inaccurate, the second derived text unit 630 may be assumed to contain more accurate information than the information contained within the derived text unit 622. In this manner, the visual search computing system 602 may dynamically and iteratively refine the visual search information (e.g., derived text units, attribute information, result images, etc.) in response to user feedback. The second text unit 630 may be provided to the user computing device 624 for display within the interface of the user computing device 624.

[0087] In some implementations, the visual search module 604 can generate second attribute information 632. The second attribute information 632 can include all of the information included in the attribute information 626 except for the attribute information associated with the document 618C. Alternatively, in some implementations, the second attribute information 632 can include instructions for not displaying the information associated with the document 618C in the interface of the user computing device 624. Additionally or alternatively, in some implementations, the visual search module 604 can retransmit result images 612 other than those included in the selectable attribute element 704B.

[0088] For example, turning to Figure 7B, Figure 7B illustrates an example interface 700B of a user computing device for display of visual search information refined based on user feedback, according to some implementations of the present disclosure. In particular, at time T2, upon receiving the result image selection information 628, the visual search computing system 602 can generate and provide second attribute information 632 and second derived text units 630 to the user computing device 624 for display in the interface 700B.

[0089] The interface 700B may include an interface element 708 that includes a second derived text unit 630. As illustrated, the second derived text unit 630 is not generated based on information contained within the document 618C, and therefore the second derived text unit 630 does not include errors associated with the content of the document 618C (e.g., information about fighter aircraft). Further, in response to an input 706 selecting the selectable attribute element 704B, the selectable attribute element 704B has been removed from the interface 700B. Instead, an additional selectable attribute element may be displayed in place of the attribute element 704B. In this manner, the user computing device 624 may communicate with the visual search computing system 602 to refine the visual search information based on user feedback.

[0090] 8 illustrates a flowchart diagram of an example method 800 for providing visual search information derived from documents containing images retrieved based on visual similarity to a query image, according to some example embodiments of the present disclosure. Although FIG. 8 illustrates steps performed in a particular order for purposes of illustration and explanation, the methods of the present disclosure are not limited to the order or sequence specifically shown. Various steps of method 800 may be variously omitted, rearranged, combined, and / or adapted without departing from the scope of the present disclosure.

[0091] At 802, a computing system may search for the multiple result images based on a similarity between the intermediate representation of the query image and each of the multiple intermediate representations associated with the multiple result images, respectively. In some implementations, searching for the multiple result images may include processing the query image with a machine learning visual search model to obtain an intermediate representation (e.g., an embedding, encoding, latent representation, etc.) of the query image. The computing system may search for the result images based on a degree of similarity between the intermediate representation of the query image and the intermediate representations of the multiple result images.

[0092] In some implementations, processing the query image with the machine learning based visual search model may include processing the query image with the machine learning based embedding model to obtain a query image embedding for the query image. The computing system may search for the multiple result images based on a distance in the embedding space between the query image embedding and the embeddings of the multiple result images.

[0093] Prior to processing the query image, the operations include obtaining the query image from a user computing device. For example, a user can use the user computing device to capture an image showing an unfamiliar object. To learn more about the object, the user can use a visual search service by providing the image and an associated prompt (e.g., "what is this object," etc.) to the computing system. Alternatively, in some implementations, the computing system can receive the image and the associated prompt from an automated service or software program. For example, an indexing service can provide the computing system with an image with an associated prompt corresponding to an indexing task (e.g., "what primary keywords should be associated with this image," etc.). Additionally or alternatively, in some implementations, the user computing device can automatically capture the image, generate a prompt, and send the image and the prompt to the computing system. For example, the user computing device can be a wearable augmented reality (AR) / virtual reality (VR) device. The user computing device can capture an image of the object and send the image along with an automatically generated prompt (e.g., "identify this object and provide related summary information," etc.) to the computing system. The user computing device may then display such information in an AR / VR context.

[0094] At 804, the computing system may identify a plurality of source documents. Each of the plurality of source documents may include a result image of the plurality of result images and textual content associated with the result image. In some implementations, identifying the plurality of source documents further includes obtaining attribute information for each of the source documents. The attribute information may include (a) identifying information that identifies the source document (e.g., a title, a citation, a numeric identifier such as a Digital Object Identifier (DOI), etc.) and / or (b) information that describes a location from which the source document can be accessed (e.g., a file path, a link to download or purchase an application, a URL, a hot link, an API call to a library or other information repository that may hold a physical copy of the document, etc.).

[0095] At 806, the computing system may determine a plurality of first text units for the plurality of result images, respectively. Each first text unit may include at least a portion of text content associated with the result image from one or more source documents that include the result image. As an example, assume that the first document that includes the first result image is an online article for a popular blog. In some instances, if the first result image is one of many images included in the article, only the text content that is closest to the first result image in the article is relatively likely to be relevant to the first result image, and thus the computing system may determine to select the text content that is closest to the first result image in the article for inclusion in the first text unit. Alternatively, if the article is relatively small, includes only a few paragraphs, or includes only the first result image, the computing system may determine to select all of the text content of the article for inclusion in the first text unit.

[0096] It should therefore be understood that the computing system may use any conventional technique for determining which portions of textual content from the source document to include in the first text unit. In some implementations, the computing system may process the textual content of the source document with a machine learning type model, such as a classification model, to predict the relevance of various portions of the textual content to the result image. Additionally or alternatively, in some implementations, the computing system may use a heuristic approach to select textual content for inclusion in the first text unit. For example, the computing system may use a rule-based schema such as: IF doc_type==Article; THEN search for sentence X-5 to X+5, where X is the location of the image in the document; IF doc_length<=1000 words; THEN Search for all words.

[0097] At 808, the computing system may process the set of text inputs with the machine learning language model to obtain a linguistic output. The linguistic output may include the second text units. The set of text inputs may include (a) two or more first text units respectively associated with two or more result images of the plurality of result images and (b) a prompt associated with the query image.

[0098] At 810, the computing system may provide the second text unit and the two or more result images to a user computing device for display within an interface of the user computing device. In some implementations, providing the second text unit and the two or more result images to the user computing device for display within an interface of the user computing device includes providing interface data to the user computing device. The interface data may include instructions for generating (a) an interface element including the second text unit and (b) two or more selectable attribute elements respectively associated with the two or more result images. Each selectable attribute element may include an associated result image or some representation of the result image, such as a thumbnail. The selectable attribute elements may also include attribute information about one or more source documents that include the associated result image.

[0099] In some implementations, the computing system may receive data from the user computing device indicating a selection of a first selectable attribute element of the two or more selectable attribute elements by a user of the user computing device. The first selectable attribute element may be associated with a first result image of the two or more result images. The computing system may identify a first text unit of the two or more first text units that includes at least a portion of text content from a source document that includes the first result image. The computing system may remove the first text unit from the set of text inputs to obtain a second set of text inputs. The computing system may process the second set of text inputs with a machine learning language model to obtain a second language output that includes the refined second text unit. The computing system may provide the refined second text unit to the user computing device.

[0100] In some implementations, removing the first text unit from the set of text inputs to obtain a second set of text inputs further includes removing information associated with the source document including the first result image from the attribute information to obtain refined attribute information. Providing the refined second text unit to the user computing device may further include providing the refined attribute information to the user computing device.

[0101] In some implementations, the linguistic output may further include prediction information predicting a portion of the second text unit as most relevant to the prompt, and the interface data may further include instructions for generating an emphasis element that highlights the portion of the second text unit.

[0102] In some implementations, providing the second text unit and the two or more result images to the user computing device for display within an interface of the user computing device may include providing interface data to the user computing device. The interface data may include instructions for generating a first interface element, a second interface element, and a first and second attribute element. The first interface element may include a first portion of the second text unit. The first portion of the second text unit may be associated with a first result image of the two or more result images. For example, if the second text unit is a summary of a first document including the first result image and a second document including the second result image, the first portion of the second text unit may be a portion summarizing the first document. Similarly, the second interface element may include a second portion of the second text unit. The second portion of the second text unit may be associated with a second result image of the two or more result images. The first selectable attribute element may include a thumbnail of the first result image, the result image itself, or an image derived from the result image, and may include attribute information about the source document that contains the first result image. The second selectable attribute element may include the second result image (or a thumbnail or an image derived therefrom) and attribute information about the source document that contains the second result image.

[0103] 9 illustrates a flow chart diagram of an exemplary method 900 for refining visual search information based on user feedback, according to an exemplary embodiment of the present disclosure. Although FIG. 9 illustrates steps performed in a particular order for purposes of illustration and explanation, the method of the present disclosure is not limited to the order or sequence specifically shown. Various steps of the method 900 may be variously omitted, rearranged, combined, and / or adapted without departing from the scope of the present disclosure.

[0104] At 902, a computing system may retrieve the two or more result images based on a similarity between the intermediate representation of the query image and the intermediate representation of the two or more result images. For example, the intermediate representation may be an image embedding, and the computing system may retrieve the two or more result images based on a distance in an embedding space between the image embedding and the image embedding for the two or more result images.

[0105] At 904, the computing system may process the set of text inputs with the machine learning language model to obtain a language output that includes the text content. The set of text inputs may include text content from a source document that includes two or more result images and a prompt associated with a query image.

[0106] In some implementations, processing the set of text inputs with the machine learning language model may include obtaining attribute information for each of the source documents. The attribute information may include (a) identifying information that identifies the source documents and / or (b) information that describes a location from which the source documents can be accessed.

[0107] At 906, the computing system may provide the linguistic output and the two or more result images to a user computing device for display within an interface of the user computing device. For example, if the user computing device is running a visual search application associated with a visual search service provided by the computing system, the computing system may provide the linguistic output and the result images for display within an interface of the visual search application. In some implementations, the computing system may also provide attribute information.

[0108] In some implementations, providing the linguistic output and the two or more result images to the user computing device for display within an interface of the user computing device may include providing interface data to the user computing device. The interface data may include instructions for generating (a) an interface element including the linguistic output and (b) two or more selectable attribute elements respectively associated with the two or more result images. Each attribute element may include a thumbnail of the associated result image and attribute information for one or more source documents that include the associated result image.

[0109] At 908, the computing system may receive, from the user computing device, information describing an indication by a user of the user computing device that a first result image of the two or more result images is visually dissimilar to the query image. In some implementations, receiving the information describing an indication by a user of the user computing device that a first result image of the two or more result images is visually dissimilar to the query image may include receiving data indicating a selection, by a user of the user computing device, of a first selectable attribute element of the two or more selectable attribute elements. The first selectable attribute element may be associated with the first result image of the two or more result images.

[0110] At 910, the computing system may remove text content associated with the source document including the first result image from the set of text inputs. In some implementations, removing the text content associated with the source document including the first result image from the set of text inputs further includes removing information associated with the source document including the first result image from the attribute information to obtain refined attribute information. In some implementations, providing the refined language output to the user computing device further includes providing the refined attribute information to the user computing device.

[0111] At 912, the computing system may process the set of text inputs with a machine learning language model to obtain refined language output.

[0112] At 914, the computing system may provide the refined language output to a user computing device for display within an interface of the user computing device.

[0113] 10 illustrates a flow chart diagram of an exemplary method 1000 for implementing collection of user feedback for refining visual search information, according to an exemplary embodiment of the present disclosure. Although FIG. 10 illustrates steps performed in a particular order for purposes of illustration and explanation, the method of the present disclosure is not limited to the order or sequence specifically shown. Various steps of the method 1000 may be variously omitted, rearranged, combined, and / or adapted without departing from the scope of the present disclosure.

[0114] At 1002, a user computing device may acquire a query image. In some implementations, acquiring the query image includes acquiring an input indicating a request to capture an image using an image capture device associated with the user computing device. In response to acquiring the input, the user computing device may capture the query image using the image capture device associated with the user computing device.

[0115] At 1004, the user computing device may obtain text data describing the prompt. In some implementations, obtaining the text data describing the prompt may include obtaining a spoken utterance from a user with an audio capture device associated with the user computing device. The user computing device may determine the text data describing the prompt based at least in part on the spoken utterance. For example, the user computing device may process the spoken utterance with a machine learning speech recognition model to obtain the text data.

[0116] At 1006, the user computing device may provide the query image and text data describing the prompt to the computing system. For example, the computing system may be a system associated with a visual search service, such as a multimodal search service that provides information in response to multimodal queries that include images and associated prompts.

[0117] At 1008, the user computing device may receive from the computing system in response to providing the query image and the prompt: (a) two or more result images and (b) linguistic output from the machine learning language model, the linguistic output being generated based on the prompt and textual content from the source document that includes the two or more result images.

[0118] At 1010, the user computing device may display, within an interface of an application executed by the user computing device, (a) an interface element including the language output and (b) two or more selectable attribute elements respectively associated with two or more result images, each selectable attribute element including a thumbnail of the associated result image and attribute information identifying a source document that includes the associated result image.

[0119] At 1012, the user computing device may receive an input from a user via an input device associated with the user computing device selecting a first selectable attribute element of the two or more selectable attribute elements.

[0120] In some implementations, each selectable attribute element may include a first selectable portion and a second selectable portion. The user computing device may receive an input from a user via an input device associated with the user computing device to select a first selectable portion of a first selectable attribute element of the two or more selectable attribute elements. In response to receiving the input to the first selectable portion of the first selectable attribute element, the user computing device may provide information indicating a selection of the first selectable attribute element to the computing system.

[0121] Alternatively, in some implementations, the user computing device may receive an input from a user via an input device associated with the user computing device, selecting a second selectable portion of a first selectable attribute element of the two or more selectable attribute elements. In response to receiving the input selecting the second selectable portion of the first selectable attribute element, the user computing device may cause a display of a source document identified by the attribute information included in the first selectable attribute element. For example, if the source document is a website, the user computing device may execute a web browser application and navigate to the website. In another example, if the source document is a PDF, the user computing device may execute a PDF reader application and open the PDF.

[0122] At 1014, in response to receiving the input, the user computing device may provide information to the computing system indicating a selection of the first selectable attribute element.

[0123] At 1016, the user computing device may receive a refined linguistic output from the computing system in response to providing the information. The refined linguistic output may be generated based on the prompt and textual content from the source document that includes two or more result images other than the first result image associated with the first selectable attribute element.

[0124] 11A illustrates a block diagram of an example computing system 1100 for implementing a visual or multimodal search service in accordance with an example embodiment of the present disclosure. The system 1100 includes a user computing system 1102, a server computing system 1130, and / or a third party computing system 1150 that are communicatively coupled via a network 1180.

[0125] The user computing system 1102 may include any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0126] The user computing system 1102 includes one or more processors 1112 and memory 1114. The one or more processors 1112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 1114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 1114 may store data 1116 and instructions 1118 that are executed by the processor 1112 to cause the user computing system 1102 to perform operations.

[0127] In some implementations, the user computing system 1102 can store or include one or more machine learning models 1120. For example, the machine learning models 1120 can be or otherwise include various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models including nonlinear and / or linear models. The neural networks can include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks.

[0128] In some implementations, one or more machine learning models 1120 may be received from a server computing system 1130 over a network 1180, stored in a user computing device memory 1114, and then used or otherwise implemented by one or more processors 1112. In some implementations, a user computing system 1102 may implement multiple parallel instances of a single machine learning model 1120 (e.g., to perform parallel machine learning model processing across multiple instances of input data and / or detected features).

[0129] More specifically, the one or more machine learning models 1120 may include one or more detection models, one or more classification models, one or more segmentation models, one or more augmentation models, one or more generative models, one or more natural language processing models, one or more optical character recognition models, and / or one or more other machine learning models. The one or more machine learning models 1120 may include one or more transformer models. The one or more machine learning models 1120 may include one or more neural radiation field models, one or more diffusion models, and / or one or more autoregressive language models.

[0130] One or more machine learning based models 1120 may be used to detect one or more object features. The detected object features may be classified and / or embedded. The classification and / or embedding may then be used to perform a search to determine one or more search results. Alternatively and / or additionally, the one or more detected features may be used to determine that an indicator (e.g., a user interface element indicating the detected feature) should be provided to indicate that the feature was detected. A user may then select an indicator to cause the feature classification, embedding, and / or search to be performed. In some implementations, the classification, embedding, and / or search may be performed before the indicator is selected.

[0131] In some implementations, the one or more machine learning models 1120 may process image data, text data, audio data, and / or latent coding data to generate output data that may include image data, text data, audio data, and / or latent coding data. The one or more machine learning models 1120 may perform optical character recognition, natural language processing, image classification, object classification, text classification, audio classification, context determination, action prediction, image correction, image enlargement, text enlargement, sentiment analysis, object detection, error detection, repair, video stabilization, audio correction, audio enlargement, and / or data segmentation (e.g., mask-based segmentation).

[0132] Additionally or alternatively, one or more machine learning based models 1140 may be included in or otherwise stored and implemented by a server computing system 1130 that communicates with the user computing system 1102 according to a client-server relationship. For example, the machine learning based models 1140 may be implemented by the server computing system 1130 as part of a web service (e.g., a viewfinder service, a visual search service, an image processing service, an ambient computing service, and / or an overlay application service). Thus, one or more models 1120 may be stored and implemented in the user computing system 1102 and / or one or more models 1140 may be stored and implemented in the server computing system 1130.

[0133] The user computing system 1102 may also include one or more user input components 1122 that receive user input. For example, the user input component 1122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be useful for implementing a virtual keyboard. Other exemplary user input components include a microphone, a traditional keyboard, or other means by which a user can provide user input.

[0134] In some implementations, the user computing system may store and / or provide one or more user interfaces that may be associated with one or more applications. The one or more user interfaces may be configured to receive input and / or provide data for display (e.g., image data, text data, audio data, one or more user interface elements, augmented reality experiences, virtual reality experiences, and / or other data for display). The user interfaces may be associated with one or more other computing systems (e.g., the server computing system 1130 and / or the third-party computing system 1150). The user interfaces may include a viewfinder interface, a search interface, a generative model interface, a social media interface, a media content gallery interface, etc.

[0135] The user computing device 1102 may include and / or receive data from one or more sensors 1126. The one or more sensors 1126 may be housed in a housing component that houses one or more processors 1112, memory 1114, and / or one or more hardware components that may store and / or execute one or more software packets. The one or more sensors 1126 may include one or more image sensors (e.g., cameras), one or more LIDAR sensors, one or more audio sensors (e.g., microphones), one or more inertial sensors (e.g., inertial measurement units), one or more biometric sensors (e.g., heart rate sensors, pulse sensors, retinal sensors, and / or fingerprint sensors), one or more infrared sensors, one or more location sensors (e.g., GPS), one or more touch sensors (e.g., conductive touch sensors and / or mechanical touch sensors), and / or one or more other sensors. The one or more sensors may be used to obtain data associated with the user's environment (eg, an image of the user's environment, a record of the environment, and / or the user's location).

[0136] The user computing system 1102 may include and / or be part of a user computing device 1104. The user computing device 1104 may include a mobile computing device (e.g., a smartphone or tablet), a desktop computer, a laptop computer, a smart wearable, and / or a smart appliance. Additionally and / or alternatively, the user computing system may acquire data from and / or generate data with one or more user computing devices 1104. For example, a smartphone camera may be used to capture image data describing the environment, and / or an overlay application of the user computing device 1104 may be used to track and / or process data provided to the user. Similarly, one or more sensors associated with a smart wearable may be used to acquire data about the user and / or about the user's environment (e.g., image data may be acquired with a camera housed in the user's smart glasses). Additionally and / or alternatively, data may be acquired and uploaded from other user devices that may be specialized in data acquisition or generation.

[0137] The server computing system 1130 includes one or more processors 1132 and memory 1134. The one or more processors 1132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operatively connected. The memory 1134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 1134 may store data 1136 and instructions 1138 that are executed by the processor 1132 to cause the server computing system 1130 to perform operations.

[0138] In some implementations, the server computing system 1130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 1130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0139] As described above, the server computing system 1130 can store or otherwise include one or more machine learning models 1140. For example, the models 1140 can be or otherwise include a variety of machine learning models. Exemplary machine learning models include neural networks or other multi-layer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

[0140] Additionally and / or alternatively, the server computing system 1130 may include and / or be communicatively connected to a search engine 1142 that may be used to crawl one or more databases (and / or resources). The search engine 1142 may process data from the user computing system 1102, the server computing system 1130, and / or the third party computing system 150 to determine one or more search results associated with the input data. The search engine 1142 may perform term-based searches, label-based searches, Boolean-based searches, image searches, embedding-based searches (e.g., nearest neighbor searches), multi-modal searches, and / or one or more other search techniques.

[0141] The server computing system 1130 can store and / or provide one or more user interfaces 1144 for obtaining input data and / or providing output data to one or more users. The one or more user interfaces 1144 can include one or more user interface elements, which may include input fields, navigation tools, content tips, selectable tiles, widgets, data display carousels, dynamic animations, information popups, image magnification, text to speech, speech to text, augmented reality, virtual reality, feedback loops, and / or other interface elements.

[0142] The user computing system 1102 and / or the server computing system 1130 can train the models 1120 and / or 1140 through interaction with a third party computing system 1150 that is communicatively coupled via a network 1180. The third party computing system 1150 can be separate from the server computing system 1130 or can be part of the server computing system 1130. Alternatively and / or additionally, the third party computing system 1150 can be associated with one or more web resources, one or more web platforms, one or more other users, and / or one or more contexts.

[0143] The third party computing system 1150 may include one or more processors 1152 and memory 1154. The one or more processors 1152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be a single processor or multiple processors operably connected. The memory 1154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 1154 may store data 1156 and instructions 1158 executed by the processor 1152 to cause the third party computing system 1150 to perform operations. In some implementations, the third party computing system 1150 includes or is otherwise implemented by one or more server computing devices.

[0144] Network 1180 may be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over network 1180 may be carried over any type of wired and / or wireless connections using a wide variety of communications protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0145] The machine learning models described herein may be used in a variety of tasks, applications, and / or use cases.

[0146] In some implementations, the input to the machine learning based model of the present disclosure may be image data. The machine learning based model may process the image data to generate an output. As an example, the machine learning based model may process the image data to generate an image recognition output (e.g., recognition of the image data, latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine learning based model may process the image data to generate an image segmentation output. As another example, the machine learning based model can process the image data to generate an image classification output. As another example, the machine learning based model may process the image data to generate an image data modification output (e.g., alteration of the image data, etc.). As another example, the machine learning based model may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning based model may process the image data to generate an upscaled image data output. As another example, the machine learning based model may process the image data to generate a prediction output.

[0147] In some implementations, the input to the machine learning based model of the present disclosure may be text or natural language data. The machine learning based model may process the text or natural language data to generate an output. As an example, the machine learning based model may process the natural language data to generate a language encoding output. As another example, the machine learning based model may process the text or natural language data to generate a latent text embedding output. As another example, the machine learning based model may process the text or natural language data to generate a transformation output. As another example, the machine learning based model may process the text or natural language data to generate a classification output. As another example, the machine learning based model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning based model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning based model may process the text or natural language data to generate an upscaled text or natural language output (e.g., text or natural language data that is of higher quality than the input text or natural language, etc.). As another example, the machine learning based model may process the text or natural language data to generate a prediction output.

[0148] In some implementations, the input to the machine learning based model of the present disclosure may be audio data. The machine learning based model may process the audio data to generate an output. As an example, the machine learning based model may process the audio data to generate a speech recognition output. As another example, the machine learning based model may process the audio data to generate a speech translation output. As another example, the machine learning based model may process the audio data to generate a latent embedding output. As another example, the machine learning based model may process the audio data to generate an encoded audio output (e.g., an encoded and / or compressed representation of the audio data, etc.). As another example, the machine learning based model may process the audio data to generate an upscaled audio output (e.g., audio data of higher quality than the input audio data, etc.). As another example, the machine learning based model may process the audio data to generate a text representation output (e.g., a text representation of the input audio data, etc.). As another example, the machine learning based model may process the audio data to generate a predicted output.

[0149] In some implementations, the input to the machine learning based models of the present disclosure may be sensor data. The machine learning based models may process the sensor data to generate an output. As an example, the machine learning based models may process the sensor data to generate a recognition output. As another example, the machine learning based models may process the sensor data to generate a prediction output. As another example, the machine learning based models may process the sensor data to generate a classification output. As another example, the machine learning based models may process the sensor data to generate a segmentation output. As another example, the machine learning based models may process the sensor data to generate a segmentation output. As another example, the machine learning based models may process the sensor data to generate a visualization output. As another example, the machine learning based models may process the sensor data to generate a diagnostic output. As another example, the machine learning based models may process the sensor data to generate a detection output.

[0150] In some cases, the input includes visual data and the task is a computer vision task. In some cases, the input includes pixel data for one or more images and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing a likelihood that one or more images show an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images and, for each region, a likelihood that the region shows an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, a respective likelihood for each category in a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines, for each pixel in one or more images, a respective depth value. As another example, the image processing task may be motion estimation, where the network input includes multiple images and the image processing output defines, for each pixel of one of the input images, the scene motion represented in pixels between the images in the network input.

[0151] The user computing system 1102 may include several applications (e.g., applications 1-N). Each application may include its own respective machine learning library and machine learning model. For example, each application may include a machine learning model. Exemplary applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, etc. In some implementations, each application may communicate with a central intelligence layer (and models stored therein) using an API (e.g., a common API across all applications).

[0152] The central intelligence layer may include several machine learning models. For example, each machine learning model (e.g., a model) may be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model (e.g., a single model) to all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by the operating system of the computing system 1100.

[0153] The central intelligence layer can communicate with a central device data layer, which can be a centralized repository of data for the computing system 1100. The central device data layer can communicate with several other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0154] FIG. 11B illustrates a block diagram of an exemplary computing system 1250 for performing visual search operations and / or refining visual search information, according to an exemplary embodiment of the present disclosure. In particular, the exemplary computing system 1250 may include one or more computing devices 1252 that may be used to acquire and / or generate one or more data sets that may be processed by the sensor processing system 1260 and / or the output determination system 1280 that may provide information about features in the one or more acquired data sets to be fed back to the user. The one or more data sets may include image data, text data, audio data, multimodal data, data for latent coding, and the like. The one or more data sets may be acquired by one or more sensors associated with the one or more computing devices 1252 (e.g., one or more sensors in the computing device 1252). Additionally and / or alternatively, the one or more data sets may be stored data and / or retrieved data (e.g., data retrieved from a web resource). For example, images, text, and / or other content items may interact with the user. The interacted content items can then be used to generate one or more decisions.

[0155] The one or more computing devices 1252 may acquire and / or generate one or more datasets based on image capture, sensor tracking, data storage retrieval, content download (e.g., downloading images or other content items from a web resource, over the Internet), and / or by one or more other techniques. The one or more datasets may be processed by a sensor processing system 1260. The sensor processing system 1260 may implement one or more processing techniques using one or more machine learning based models, one or more search engines, and / or one or more other processing techniques. The one or more processing techniques may be implemented in any combination and / or individually. The one or more processing techniques may be implemented sequentially and / or in parallel. In particular, the one or more datasets may be processed by a context determination block 1262, which may determine a context associated with one or more content items. The context determination block 1262 may identify and / or process metadata, user profile data (e.g., preferences, user search history, user browsing history, user purchase history, and / or user input data), previous interaction data, global trend data, location data, time data, and / or other data to determine a particular context associated with the user. The context may be associated with an event, a determined trend, a particular action, a particular type of data, a particular environment, and / or another context associated with the user and / or the searched or retrieved data.

[0156] The sensor processing system 1260 may include an image pre-processing block 1264. The image pre-processing block 1264 may be used to adjust one or more values ​​of the acquired and / or received images to prepare the images to be processed by one or more machine learning based models and / or one or more search engines 1274. The image pre-processing block 1264 may resize the images, adjust saturation values, adjust resolution, remove and / or add metadata, and / or perform one or more other operations.

[0157] In some implementations, the sensor processing system 1260 may include one or more machine learning based models, which may include a detection model 1266, a segmentation model 1268, a classification model 1270, an embedding model 1272, and / or one or more other machine learning based models. For example, the sensor processing system 1260 may include one or more detection models 1266 that can be used to detect particular features in a processed dataset. In particular, one or more images may be processed with the one or more detection models 1266 to generate one or more bounding boxes associated with detected features in the one or more images.

[0158] Additionally and / or alternatively, one or more segmentation models 1268 may be used to segment one or more portions of a dataset from one or more datasets. For example, one or more segmentation models 1268 may use one or more segmentation masks (e.g., one or more segmentation masks generated manually and / or based on one or more bounding boxes) to segment a portion of an image, a portion of an audio file, and / or a portion of text. Segmentation may include isolating one or more detected objects and / or removing one or more detected objects from an image.

[0159] The one or more classification models 1270 may be used to process image data, text data, audio data, latent coding data, multimodal data, and / or other data to generate one or more classifications. The one or more classification models 1270 may include one or more image classification models, one or more object classification models, one or more text classification models, one or more audio classification models, and / or one or more other classification models. The one or more classification models 1270 may process the data to determine one or more classifications.

[0160] In some implementations, data may be processed with one or more embedding models 1272 to generate one or more embeddings. For example, one or more images may be processed with one or more embedding models 1272 to generate one or more image embeddings in the embedding space. The one or more image embeddings may be associated with one or more image features of the one or more images. In some implementations, the one or more embedding models 1272 may be configured to process multimodal data to generate multimodal embeddings. The one or more embeddings may be used for classification, search, and / or learning embedding space distributions.

[0161] The sensor processing system 1260 may include one or more search engines 1274 that can be used to perform one or more searches. The one or more search engines 1274 may crawl one or more databases (e.g., one or more local databases, one or more global databases, one or more private databases, one or more public databases, one or more specialized databases, and / or one or more general databases) to determine one or more search results. The one or more search engines 1274 may perform feature matching, text-based searches, embedding-based searches (e.g., k-nearest neighbor searches), metadata-based searches, multimodal searches, web resource searches, image searches, text searches, and / or application searches.

[0162] Additionally and / or alternatively, the sensor processing system 1260 may include one or more multimodal processing blocks 1276, which may be used to aid in processing the multimodal data. The one or more multimodal processing blocks 1276 may include generating multimodal queries and / or multimodal embeddings to be processed by one or more machine learning based models and / or one or more search engines 1274.

[0163] The output of the sensor processing system 1260 may then be processed in an output determination system 1280 to determine one or more outputs to provide to a user. The output determination system 1280 may include heuristic-based determination, machine learning model-based determination, user selection-based determination, and / or context-based determination.

[0164] The output determination system 1280 may determine how and / or where to provide one or more search results in the search result interface 1282. Additionally and / or alternatively, the output determination system 1280 may determine how and / or where to provide one or more machine learning model outputs in the machine learning model output interface 1284. In some implementations, one or more search results and / or one or more machine learning model outputs may be provided for display by one or more user interface elements. The one or more user interface elements may be overlaid on the displayed data. For example, one or more detection indicators may be overlaid on the detected object in the viewfinder. The one or more user interface elements may be selectable to perform one or more additional searches and / or one or more additional machine learning model processes. In some implementations, the user interface elements may be provided as specialized user interface elements for a particular application and / or may be provided uniformly across different applications. The one or more user interface elements may include pop-up displays, interface overlays, interface tiles and / or chips, carousel interfaces, audio feedback, animations, interactive widgets, and / or other user interface elements.

[0165] Additionally and / or alternatively, data associated with the output of the sensor processing system 1260 can be used to generate and / or provide an augmented reality experience and / or a virtual reality experience 1286. For example, one or more acquired datasets may be processed to generate one or more augmented reality rendering assets and / or one or more virtual reality rendering assets, which can then be used to provide an augmented reality experience and / or a virtual reality experience 1286 to a user. The augmented reality experience may render information associated with an environment into the respective environment. Alternatively and / or additionally, objects associated with the processed datasets may be rendered into the user environment and / or the virtual environment. The rendering of the dataset generation may include training one or more neural radiation field models to learn three-dimensional representations for one or more objects.

[0166] In some implementations, one or more action prompts 1288 may be determined based on the output of the sensor processing system 1260. For example, a search prompt, a purchase prompt, a create prompt, a reservation prompt, a call prompt, a redirect prompt, and / or one or more other prompts may be determined to be associated with the output of the sensor processing system 1260. The one or more action prompts 1288 may then be provided to the user via one or more selectable user interface elements. In response to a selection of the one or more selectable user interface elements, a respective action of the respective action prompt may be performed (e.g., a search may be performed, a purchasing application programming interface may be used, and / or another application may be opened).

[0167] In some implementations, one or more datasets and / or outputs of the sensor processing system 1260 may be processed with one or more generative models 1290 to generate model-generated content items, which may then be provided to a user. Generation may be prompted based on user selection and / or performed automatically (e.g., based on one or more conditions, which may be associated with a threshold amount of unidentified search results).

[0168] The output determination system 1280 may process one or more data sets and / or the output of the sensor processing system 1260 in a data augmentation block 1292 to generate augmented data. For example, one or more images may be processed in the data augmentation block 1292 to generate one or more augmented images. Data augmentation may include data correction, data cropping, removal of one or more features, addition of one or more features, resolution adjustment, illumination adjustment, saturation adjustment, and / or other augmentations.

[0169] In some implementations, one or more data sets and / or outputs of the sensor processing system 1260 may be stored based on the determination of the data storage block 1294.

[0170] The output of the output determination system 1280 may then be provided to a user by one or more output components of the user computing device 1252. For example, one or more user interface elements associated with the one or more outputs may be provided for display by a visual display of the user computing device 1252.

[0171] The process may be performed iteratively and / or sequentially. One or more user inputs to provided user interface elements may condition and / or affect subsequent processing loops.

[0172] The techniques described herein refer to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of possible configurations, combinations, and divisions of tasks and functionality among components. For example, the processes described herein may be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate serially or in parallel.

[0173] Although the present subject matter has been described in detail with respect to various specific exemplary embodiments thereof, each example is provided as an explanation, not a limitation of the present disclosure. Those skilled in the art, upon understanding the above content, can easily produce modifications, variations, and equivalents to such embodiments. Thus, the present disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the present subject matter as would be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may be used with another embodiment to produce yet a further embodiment. Thus, it is intended that the present disclosure encompass such alternatives, modifications, and equivalents.

[0174] Embodiment Several embodiments of the present disclosure will now be described. However, it should be noted that the following embodiments are not a comprehensive list of all embodiments of the present disclosure. Instead, the following embodiments are provided to illustrate various scenarios in which embodiments of the present disclosure may be used.

[0175] Embodiment 1: A computer-implemented method comprising: - searching, by a computing system comprising one or more processor devices, for result images based on similarities between the query image and the result images; - obtaining, by a computing system, a first text unit, the first text unit including at least a portion of the text content of a source document including a result image; - determining, by the computing system, a second text unit in response to a prompt associated with the query image, the second text unit comprising: (a) at least a portion of a first text unit; or (b) text derived from the first text unit; - providing, by the computing system, the second unit of text and the resultant image for display within the interface.

[0176] Embodiment 2: The method of embodiment 1, wherein the step of searching for the result images includes: processing, by a computing system, the query image with a machine learning visual search model to obtain an intermediate representation of the query image; and searching, by the computing system, for the result images based on a degree of similarity between the intermediate representation of the query image and the intermediate representation of the result images.

[0177] Embodiment 3: The computer-implemented method of embodiment 2, wherein the step of processing the query image with the machine learning visual search model includes a step of processing, by the computing system, the query image with the machine learning embedding model to obtain a query image embedding for the query image, and the step of searching for the result images includes a step of searching, by the computing system, for the result images based on a distance in the embedding space between the query image embedding and the embedding of the result images.

[0178] Embodiment 4: The computer-implemented method of embodiment 1, wherein prior to processing the query image, the method includes obtaining, by the computing system, the query image from a user computing device.

[0179]

[0023] Embodiment 5: The computer-implemented method of embodiment 4, wherein the interface comprises a user interface of an application executed by the user computing device.

[0180]

[0023] Embodiment 6: The computer-implemented method of embodiment 4, wherein obtaining the query image includes obtaining, by the computing system, the query image and a prompt associated with the query image from a user computing device.

[0181] Embodiment 7: The computer-implemented method of embodiment 4, wherein the step of retrieving the result images further includes the steps of providing the result images to a user computing device for display within the interface by the computing system, and receiving a prompt associated with the query image from the user computing device in response to providing the result images.

[0182] Embodiment 8: The computer-implemented method of embodiment 1, wherein determining the second text unit in response to a prompt associated with the query image includes processing, by the computing system, the second text unit and the prompt associated with the query image with a machine learning language model to obtain language output including the second text unit.

[0183]

[0023] Embodiment 9: The computer-implemented method of embodiment 8, wherein the second text unit comprises a subset of the first text unit.

[0184] Embodiment 10: The computer-implemented method of embodiment 8, wherein the second text unit includes text derived from the first text unit, the text derived from the first text unit describing a summary of the first text unit.

[0185] Embodiment 11: The source document is - one or more web pages of the website; - article, - Newspaper, - Books, or - a transcript.

[0186] Embodiment 12: The computer-implemented method of embodiment 1, wherein the step of providing the second text unit and the resultant image further includes the step of providing attribute information, for display within the interface by the computing system, that (a) identifies the source document and / or (b) indicates a location from which the source document can be accessed.

[0187]

[0023] Embodiment 13: The computer-implemented method of embodiment 12, wherein the source document includes a web page and the attribute information includes an address of the web page.

[0188] Embodiment 14: The computer-implemented method of embodiment 12, wherein the source document includes a magazine and the attribution information includes a citation indicating a location of the resultant image within the magazine.

[0189] Embodiment 15: The computer-implemented method of embodiment 1, wherein prior to determining the second text unit, the method includes generating, by the computing system, a prompt associated with the query image based at least in part on the query image.

[0190] Embodiment 16: The computer-implemented method of embodiment 15, wherein the step of generating a prompt associated with the query image includes a step of processing the query image with a machine learning model by a computing system to generate a semantic output describing the image, and a step of generating, by the computing system, the prompt based at least in part on the semantic output.

[0191] Embodiment 17: A computing system comprising: - one or more processors; one or more non-transitory computer-readable media that collectively store a first set of instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations including: Obtaining a query image and an associated prompt from a user computing device; processing the query image with a machine learning embedding model to obtain a query image embedding; Retrieving result images based on similarity between the query image embeddings and the result image embeddings; Identifying a source document for a resultant image, the source document including the resultant image and text content associated with the resultant image; determining a first text unit from the source document that includes at least a portion of the text content associated with the result image; processing the first text unit and the prompt with a machine learning language model to obtain a language output including a second text unit, the second text unit comprising: (a) at least a portion of a first text unit; or (b) text derived from the first text unit; and providing the second text unit and the resultant image for display within an interface of an application executed by the user computing device.

[0192] Embodiment 18: The operation is - receiving information from a user computing device indicating a request for additional information; - retrieving a second result image based on a similarity between the query image embeddings and the embeddings of the second result image; - identifying a first source document and a second source document for a result image, each of the first source document and the second source document including a result image and text content associated with the result image, the text content associated with the result image of the first source document being different from the text content associated with the result image of the second source document; - determining an additional first text unit from one or more of the first source document or the second source document, the additional first text unit including at least a portion of the text content associated with the result image; - processing the additional first text unit and the prompt with a machine learning language model to obtain a second language output including an additional second text unit, the additional second text unit comprising: (a) at least a portion of the additional first text unit, or (b) text derived from the additional first text unit; and - providing the additional second text unit and the second result image for display within an interface of an application executed by the user computing device.

[0193] Embodiment 19: The computing system of embodiment 17, wherein providing the second text unit and the resultant image further includes providing attribute information identifying the source document for display within an interface of an application executed by the user computing device.

[0194] Embodiment 20: One or more non-transitory computer-readable media collectively storing a first set of instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: - retrieving the result images based on similarity between the query image and the result images; - obtaining a first text unit, the first text unit including at least a portion of a text content of a source document including a result image; - determining a second text unit in response to a prompt associated with the query image, the second text unit comprising: (a) at least a portion of a first text unit; or (b) text derived from the first text unit; and - providing the second text unit and the resultant image for display within the interface.

[0195]

[0082] Embodiment 21: A computing system comprising: - one or more processors; one or more non-transitory computer-readable media that collectively store a first set of instructions that, when executed by one or more processors, cause a computing system to perform operations, the operations including: Retrieving a plurality of result images based on a similarity between an intermediate representation of the query image and each of a plurality of intermediate representations respectively associated with the plurality of result images; identifying a plurality of source documents, each of the plurality of source documents including a result image of the plurality of result images and text content associated with the result image; determining a plurality of first text units for a plurality of result images, each first text unit including at least a portion of text content associated with the result image from one or more source documents including the result image; processing a set of text inputs with a machine learning language model to obtain a language output comprising second text units, the set of text inputs comprising: (a) two or more first text units respectively associated with two or more result images of the plurality of result images; and (b) a prompt associated with the query image; and providing the second text unit and the two or more resultant images to a user computing device for display within an interface of the user computing device.

[0196] Embodiment 22: The computing system of embodiment 21, wherein searching the plurality of result images includes processing the query image with a machine learning visual search model to obtain an intermediate representation of the query image, and searching the result images based on a degree of similarity between the intermediate representation of the query image and the intermediate representations of the plurality of result images.

[0197] Embodiment 23: The computing system of embodiment 22, wherein processing the query image with the machine learning visual search model includes processing the query image with the machine learning embedding model to obtain a query image embedding for the query image, and searching the multiple result images includes searching the multiple result images based on a distance between the query image embedding and the embeddings of the multiple result images in the embedding space.

[0198]

[0071] Embodiment 24: The computing system of embodiment 21, wherein prior to processing the query image, the operation includes obtaining the query image from a user computing device.

[0199]

[0046] Embodiment 25: The computing system of embodiment 21, wherein obtaining the query image includes obtaining the query image and a prompt associated with the query image from a user computing device.

[0200] Embodiment 26: The computing system of embodiment 21, wherein identifying the plurality of source documents further includes obtaining attribute information, and for each of the plurality of source documents, the attribute information includes (a) identification information that identifies the source document, and / or (b) information that describes a location from which the source document can be accessed.

[0201] Embodiment 27: The computing system of embodiment 26, wherein providing the second text unit and the two or more result images to the user computing device for display within an interface of the user computing device includes providing interface data to the user computing device, the interface data including instructions for generating (a) an interface element including the second text unit and (b) two or more selectable attribute elements respectively associated with the two or more result images, each selectable attribute element including a thumbnail of the associated result image and attribute information for one or more source documents that include the associated result image.

[0202] Embodiment 28: The operation is - receiving, from a user computing device, data indicating a selection, by a user of the user computing device, of a first selectable attribute element of the two or more selectable attribute elements, the first selectable attribute element being associated with a first result image of the two or more result images; - identifying a first text unit of the two or more first text units, the first text unit including at least a portion of text content from a source document including the first result image; - removing the first text unit from the set of text inputs to obtain a second set of text inputs; - processing the second set of text inputs with the machine learning language model to obtain a second language output including refined second text units; and - providing the refined second text unit to a user computing device.

[0203] Embodiment 29: The computing system of embodiment 28, wherein removing the first text unit from the set of text inputs to obtain a second set of text inputs further includes removing information associated with the source document including the first result image from the attribute information to obtain refined attribute information, and providing the refined second text unit to the user computing device further includes providing the refined attribute information to the user computing device.

[0204] Embodiment 30: The computing system of embodiment 27, wherein the language output further includes prediction information predicting a portion of the second text unit as being most relevant to the prompt, and the interface data further includes instructions for generating an emphasis element that highlights that portion of the second text unit.

[0205]

[0023] Embodiment 31: Providing the second text unit and the two or more resultant images to a user computing device for display within an interface of the user computing device includes providing interface data to the user computing device, the interface data comprising: - a first interface element including a first portion of a second text unit, the first portion of the second text unit being associated with a first result image of the two or more result images; - a second interface element including a second portion of the second text unit, the second portion of the second text unit being associated with a second result image of the two or more result images; and 27. The computing system of embodiment 26, comprising instructions for generating a first selectable attribute element and a second selectable attribute element, the first selectable attribute element including a thumbnail of the first result image and attribute information for a source document including the first result image, and the second selectable attribute element including a thumbnail of the second result image and attribute information for the source document including the second result image.

[0206]

[0036] Embodiment 32: The computing system of embodiment 21, wherein the second text unit includes a summary of two or more of the first text units.

[0207] Embodiment 33: A computer-implemented method comprising: - searching, by a computing system including one or more computing devices, for a plurality of result images based on a similarity between an intermediate representation of the query image and each of a plurality of intermediate representations respectively associated with the plurality of result images; - identifying, by a computing system, a plurality of source documents, each of the plurality of source documents including a result image of the plurality of result images and textual content associated with the result image; - determining, by the computing system, a plurality of first text units for a plurality of result images, respectively, each first text unit including at least a portion of text content associated with the result image from one or more source documents including the result image; - processing, by a computing system, a set of text inputs with a machine learning language model to obtain a language output comprising second text units, the set of text inputs comprising: (a) two or more first text units respectively associated with two or more result images of the plurality of result images; and (b) a prompt associated with the query image; and - providing, by the computing system, the second text unit and the two or more resultant images to a user computing device for display within an interface of the user computing device.

[0208] Embodiment 34: The computer-implemented method of embodiment 33, wherein the step of searching for the plurality of result images includes a step of processing, by a computing system, the query image with a machine learning visual search model to obtain an intermediate representation of the query image, and a step of searching, by the computing system, for the result images based on a degree of similarity between the intermediate representation of the query image and the intermediate representations of the plurality of result images.

[0209] Embodiment 35: The computer-implemented method of embodiment 34, wherein the step of processing the query image with the machine learning visual search model includes a step of processing, by a computing system, the query image with a machine learning embedding model to obtain a query image embedding for the query image, and the step of searching for the multiple result images includes a step of searching, by the computing system, for the multiple result images based on a distance in the embedding space between the query image embedding and the embeddings of the multiple result images.

[0210] Embodiment 36: The computer-implemented method of embodiment 33, wherein prior to processing the query image, the method includes a step of obtaining, by the computing system, the query image from a user computing device.

[0211] Embodiment 37: The computer-implemented method of embodiment 33, wherein obtaining the query image includes obtaining, by the computing system, the query image and a prompt associated with the query image from a user computing device.

[0212] Embodiment 38: The computer-implemented method of embodiment 33, wherein the step of identifying a plurality of source documents further includes a step of obtaining, by the computing system, attribute information, for each of the plurality of source documents, the attribute information including: (a) identification information that identifies the source document, and / or (b) information that describes a location from which the source document can be accessed.

[0213] Embodiment 39: The computer-implemented method of embodiment 38, wherein the step of providing the second text unit and the two or more result images to the user computing device for display within an interface of the user computing device includes a step of providing, by the computing system, interface data to the user computing device, the interface data including instructions for generating (a) an interface element including the second text unit and (b) two or more selectable attribute elements respectively associated with the two or more result images, each attribute element including a thumbnail of the associated result image and attribute information for one or more source documents containing the associated result image.

[0214] Embodiment 40: One or more non-transitory computer-readable media collectively storing a first set of instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: - obtaining a query image and an associated prompt from a user computing device; - processing the query image with the machine learning embedding model to obtain a query image embedding; - retrieving result images based on similarity between the query image embeddings and the result image embeddings; - identifying a source document for a resultant image, the source document including the resultant image and text content associated with the resultant image; - determining a first text unit from the source document that includes at least a portion of text content associated with the result image; - processing the first text unit and the prompt with a machine learning language model to obtain a language output including a second text unit, the second text unit comprising: (a) at least a portion of a first text unit; or (b) text derived from the first text unit; and - providing the second text unit and the resultant image for display within an interface of an application executed by the user computing device.

[0215] Embodiment 41: A computer-implemented method comprising: - retrieving, by a computing system comprising one or more computing devices, the two or more result images based on similarities between the intermediate representation of the query image and the intermediate representations of the two or more result images; - processing, by a computing system, a set of text inputs with a machine learning language model to obtain a language output comprising text content, the set of text inputs comprising text content from a source document comprising two or more result images and a prompt associated with a query image; - providing, by the computing system, the linguistic output and two or more result images to a user computing device for display within an interface of the user computing device; - receiving, by the computing system, information from a user computing device describing an indication by a user of the user computing device that a first result image of the two or more result images is visually dissimilar to the query image; - removing, by the computing system, text content associated with the source document including the first result image from the set of text inputs; - processing, by a computing system, a set of text inputs with a machine learning language model to obtain a refined language output; - providing, by the computing system, the refined linguistic output to a user computing device for display within an interface of the user computing device.

[0216] Embodiment 42: The computer-implemented method of embodiment 41, wherein the step of searching for two or more result images includes a step of processing, by a computing system, the query image with a machine learning visual search model to obtain an intermediate representation of the query image, and a step of searching, by the computing system, for the result images based on a degree of similarity between the intermediate representation of the query image and the intermediate representations of the two or more result images.

[0217] Embodiment 43: The computer-implemented method of embodiment 42, wherein prior to processing the query image, the method includes a step of obtaining, by the computing system, the query image from a user computing device.

[0218] Embodiment 44: The computer-implemented method of embodiment 41, wherein the step of processing the set of text inputs with a machine learning language model further includes a step of obtaining attribute information, and for each source document, the attribute information includes (a) identification information that identifies the source document, and / or (b) information that describes a location from which the source document can be accessed.

[0219] Embodiment 45: The computer-implemented method of embodiment 44, wherein the step of providing the linguistic output and two or more result images further includes a step of providing, by the computing system, attribute information to a user computing device for display within an interface of the user computing device.

[0220] Embodiment 46: The computer-implemented method of embodiment 44, wherein the step of providing the linguistic output and the two or more result images to a user computing device for display within an interface of the user computing device includes a step of providing interface data to the user computing device, the interface data including instructions for generating (a) an interface element including the linguistic output and (b) two or more selectable attribute elements respectively associated with the two or more result images, each attribute element including a thumbnail of the associated result image and attribute information for one or more source documents containing the associated result image.

[0221] Embodiment 47: The computer-implemented method of embodiment 46, wherein the step of receiving information describing an indication by a user of the user computing device that a first result image of the two or more result images is visually dissimilar to the query image includes a step of receiving data from the user computing device indicating a selection by the user of the user computing device of a first selectable attribute element of the two or more selectable attribute elements, wherein the first selectable attribute element is associated with the first result image of the two or more result images.

[0222] Embodiment 48: The computer-implemented method of embodiment 47, wherein the step of removing text content associated with the source document including the first result image from the set of text inputs further includes removing information associated with the source document including the first result image from the attribute information to obtain refined attribute information, and the step of providing the refined language output to the user computing device further includes providing the refined attribute information to the user computing device.

[0223] Embodiment 49: The computer-implemented method of embodiment 46, wherein the linguistic output further includes prediction information predicting a portion of the linguistic output to be most relevant to the prompt, and the interface data further includes instructions for generating an emphasis element that highlights the portion of the linguistic output.

[0224] Embodiment 50: A computer-implemented method comprising: - acquiring, by a user computing device comprising one or more processors, a query image; - obtaining, by a user computing device, text data describing a prompt; - providing, by a user computing device, a query image and text data describing a prompt to a computing system associated with the visual search service; - receiving, by the user computing device from the computing system in response to providing the query image and the prompt, (a) two or more result images and (b) linguistic output from the machine learning language model, where the linguistic output is generated based on the prompt and textual content from a source document that includes the two or more result images; by the user computing device, within the interface of an application executed by the user computing device, (a) interface elements that contain linguistic output; and ○(b) displaying two or more selectable attribute elements respectively associated with two or more result images, each selectable attribute element including a thumbnail of the associated result image and attribute information identifying a source document containing the associated result image.

[0225] Embodiment 51: The computer-implemented method of embodiment 50, wherein each selectable attribute element includes a first selectable portion and a second selectable portion.

[0226] Embodiment 52: The computer-implemented method of embodiment 51, wherein the method further includes a step of receiving, by the user computing device, an input from a user via an input device associated with the user computing device selecting a first selectable portion of a first selectable attribute element of the two or more selectable attribute elements.

[0227] Embodiment 53: The computer-implemented method of embodiment 52, further comprising the steps of: in response to receiving input into a first selectable portion of the first selectable attribute element, providing information to the computing system by the user computing device indicating selection of the first selectable attribute element; and in response to providing the information, receiving a refined linguistic output from the computing system by the user computing device, wherein the refined linguistic output is generated based on text content from a source document that includes a prompt and two or more result images other than the first result image associated with the first selectable attribute element.

[0228] Embodiment 54: The computer-implemented method of embodiment 53, wherein the method further includes a step of displaying, by the user computing device, within an interface of an application executed by the user computing device, (a) an interface element including the refined language output and (b) one or more selectable attribute elements, the one or more selectable attribute elements including each of the two or more selectable attribute elements other than the first selectable attribute element.

[0229] Embodiment 55: The computer-implemented method of embodiment 52, wherein the method further includes the steps of receiving, by the user computing device, an input from a user via an input device associated with the user computing device selecting a second selectable portion of a first selectable attribute element of the two or more selectable attribute elements, and in response to receiving the input selecting the second selectable portion of the first selectable attribute element, causing, by the user computing device, a display of a source document identified by attribute information included in the first selectable attribute element.

[0230]

[0081] Embodiment 56: Each of the source documents includes: -One or more web pages of the website; - article, - Newspaper, - Books, or - a transcript.

[0231] Embodiment 57: The computer-implemented method of embodiment 50, wherein the step of obtaining text data describing the prompt includes the steps of obtaining, by the user computing device, a spoken utterance from the user via an audio capture device associated with the user computing device, and determining, by the user computing device, the text data describing the prompt based at least in part on the spoken utterance.

[0232] Embodiment 58: The computer-implemented method of embodiment 50, wherein the step of acquiring a query image includes the steps of acquiring, by a user computing device, an input indicating a request to capture an image using an image capture device associated with the user computing device, and, in response to acquiring the input, capturing, by the user computing device, the query image using the image capture device associated with the user computing device.

[0233]

[0071] Embodiment 59: A user computing device, comprising: - one or more processors; - one or more non-transitory computer-readable media collectively storing a first set of instructions that, when executed by the one or more processors, cause the user computing device to perform operations, the operations including: Obtaining a query image; obtaining text data describing a prompt; providing a query image and text data describing a prompt to a computing system associated with a visual search service; receiving, from the computing system in response to providing the query image and the prompt, (a) two or more result images and (b) linguistic output from the machine learning language model, the linguistic output being generated based on the prompt and textual content from a source document that includes the two or more result images; Within the interface of an application executed by the user computing device, Interface elements, including language output, and displaying two or more selectable attribute elements respectively associated with two or more result images, each selectable attribute element including a thumbnail of the associated result image and attribute information identifying a source document that includes the associated result image; receiving an input from a user via an input device associated with the user computing device selecting a first selectable attribute element of the two or more selectable attribute elements; in response to receiving the input, providing to the computing system information indicative of a selection of the first selectable attribute element; a user computing device including: receiving a refined language output from the computing system in response to providing the information, the refined language output being generated based on the prompt and text content from a source document including two or more result images other than the first result image associated with the first selectable attribute element.

[0234] Embodiment 60: One or more non-transitory computer-readable media collectively storing a first set of instructions that, when executed by one or more processors of a user computing device, cause the user computing device to perform an operation, the operation comprising: - obtaining a query image; - obtaining text data describing a prompt; and - providing a query image and text data describing a prompt to a computing system associated with the visual search service; - receiving, from the computing system in response to providing the query image and the prompt, two or more result images and a linguistic output from the machine learning language model, the linguistic output being generated based on the prompt and textual content from a source document that includes the two or more result images; - within the interface of an application executed by the user computing device, ○ Interface elements that contain language output, and one or more non-transitory computer-readable media, comprising: displaying two or more selectable attribute elements each associated with two or more result images, each selectable attribute element including a thumbnail of the associated result image and attribute information identifying a source document that contains the associated result image. [Explanation of symbols]

[0235] 100 Visual Search System 102 User Computing Devices 104 Input Devices 106 Communication Module 108 Visual Search Computing System 110 Network 112 Query Images 114 Prompt 116 Visual Search Requests 118 Visual Search Module 120 Text Content 122 Result Image 124 Interface Data 202 Visual Search Computing System 203 User Computing Devices 204 Visual Search Module 206 Query Images 208 Prompt 210 Image Evaluation Module 212 Result Image 214 Machine Learning Visual Search Model, Visual Search Model 215 Image Search Space 216 Document Indexing Information 218 Document Content Selection Module 220 Documents 222 Text Content 224 First Text Unit 226 Text Judgment Module 228 Second Text Unit 230 Machine Learning Language Models 234 Interface Data 236 Attribute information 232 Interface Data Generation Module 400 Interface 402 Visual Search Request 404 Major Interface Elements 405 Attribute element 406 Result Image Element 500A Interface 500B Interface 502 Visual Search Request 504 Major Interface Elements 505 attribute element 506 Emphasis element 508 Secondary Interface Element 510 Input 514 Information request element 602 Visual Search Computing System 604 Visual Search Module 606 Query Images 608 Prompt 610 Image Evaluation Module 612 Result Image 614 Text Units 616 Document Content Selection Module 618 Document Indexing Information 620 Text Judgment Module 621 Large-scale Language Model 622 Derived Text Units 624 User Computing Devices 626 Attribute information 628 Result Image Selection Information 630 Second Derived Text Unit 632 Second attribute information 700A Interface 700B Interface 702 Interface Elements 704 Selectable Attribute Elements 706 Input 1100 Computing system, system 1180 Network 1102 User Computing System 1104 User Computing Device 1112 Processor 1114 Memory 1116 Data 1118 command 1120 Machine learning model, model 1122 User Input Components 1126 Sensor 1130 Server Computing System 1132 processor 1134 Memory 1136 Data 1138 command 1140 Machine learning model, model 1142 Search Engine 1144 User Interface 1150 Third Party Computing Systems 1152 processor 1154 Memory 1156 Data 1158 Command 1250 Computing System 1252 Computing Device, User Computing Device 1260 Sensor Processing System 1262 Context Decision Block 1264 Image Pre-processing Block 1266 Detection Model 1268 Segmented Model 1270 Classification Model 1272 Embedded Model 1274 Search Engine 1276 Multimodal Processing Block 1280 Output Judgment System 1282 Search Results Interface 1284 Machine Learning Model Output Interface 1286 Augmented and / or Virtual Reality Experiences 1288 Action Prompt 1290 Generative Model 1292 Data Expansion Block 1294 data storage blocks

Claims

1. 1. A computer-implemented method comprising: searching, by a computing system comprising one or more computing devices, two or more result images based on similarity between the intermediate representation of the query image and the intermediate representation of the two or more result images; processing, by the computing system, a set of textual inputs with a machine learning language model to obtain a language output comprising textual content, the set of textual inputs comprising textual content from a source document comprising the two or more result images and a prompt associated with the query image; providing, by the computing system, the linguistic output and the two or more result images to the user computing device for display within an interface of the user computing device; receiving, by the computing system, information from the user computing device describing an indication by a user of the user computing device that a first result image of the two or more result images is visually dissimilar to the query image; removing, by the computing system, textual content associated with the source document including the first result image from the set of text inputs; processing, by the computing system, the set of text inputs with the machine learning language model to obtain refined language output; providing, by the computing system, the refined linguistic output to the user computing device for display within the interface of the user computing device; The method includes:

2. The step of retrieving two or more result images includes: processing, by the computing system, the query image through a machine learning visual search model to obtain the intermediate representation of the query image; retrieving, by the computing system, the result images based on a degree of similarity between the intermediate representation of the query image and the intermediate representations of the two or more result images.

2. The computer-implemented method of claim 1, comprising:

3. The computer-implemented method of claim 2 , wherein prior to processing the query image, the method includes obtaining, by the computing system, the query image from the user computing device.

4. 2. The computer-implemented method of claim 1, wherein processing the set of text inputs with the machine learning language model further comprises obtaining attribute information, for each of the source documents, the attribute information including: (a) identification information that identifies the source document; and / or (b) information that describes a location where the source document can be accessed.

5. 5. The computer-implemented method of claim 4, wherein providing the linguistic output and the two or more result images further includes providing, by the computing system, the attribute information to the user computing device for display within the interface of the user computing device.

6. providing the linguistic output and the two or more result images to the user computing device for display within the interface of the user computing device, comprising: providing interface data to the user computing device; The interface data includes: (a) an interface element including the linguistic output; (b) two or more selectable attribute elements respectively associated with the two or more result images, each attribute element including a thumbnail of the associated result image and the attribute information for one or more source documents containing the associated result image; 5. The computer-implemented method of claim 4, further comprising instructions for generating:

7. receiving the information describing the indication by the user of the user computing device that the first result image of the two or more result images is visually dissimilar to the query image, receiving, from the user computing device, data indicative of a selection, by the user of the user computing device, of a first selectable attribute element of the two or more selectable attribute elements; The computer-implemented method of claim 6 , wherein the first selectable attribute element is associated with the first of the two or more result images.

8. removing the text content associated with the source document including the first result image from the set of text inputs further includes removing information associated with the source document including the first result image from the attribute information to obtain refined attribute information; 8. The computer-implemented method of claim 7, wherein providing the refined linguistic output to the user computing device further comprises providing the refined attribute information to the user computing device.

9. the linguistic output further includes predictive information predicting a portion of the linguistic output to be most relevant to the prompt; The computer-implemented method of claim 6 , wherein the interface data further comprises instructions for generating an emphasis element that highlights the portion of the language output.

10. 1. A user computing device, comprising: one or more processors; and one or more non-transitory computer-readable storage media that collectively store a first set of instructions that, when executed by the one or more processors, cause the user computing device to perform operations, the operations including: Obtaining a query image; obtaining text data describing a prompt; providing the query image and the text data describing the prompt to a computing system associated with a visual search service; receiving, from the computing system in response to providing the query image and the prompt, (a) two or more result images and (b) linguistic output from a machine learning language model, the linguistic output being generated based on the prompt and textual content from a source document that includes the two or more result images; within an interface of an application executed by said user computing device; (a) an interface element including said linguistic output; and (b) two or more selectable attribute elements respectively associated with the two or more result images, each selectable attribute element including a thumbnail of the associated result image and attribute information identifying a source document containing the associated result image; Displaying the a user computing device comprising:

11. The user computing device of claim 10 , wherein each selectable attribute element includes a first selectable portion and a second selectable portion.

12. The operation includes:

12. The user computing device of claim 11, further comprising receiving an input from a user via an input device associated with the user computing device selecting the first selectable portion of a first selectable attribute element of the two or more selectable attribute elements.

13. Receiving the input into the first selectable portion of the first selectable attribute element includes: providing to the computing system information indicative of a selection of the first selectable attribute element; receiving, in response to providing the information, from the computing system, a refined linguistic output, the refined linguistic output being generated based on the prompt and textual content from a source document including the two or more result images other than a first result image associated with the first selectable attribute element; The user computing device of claim 12 further comprising:

14. The operation includes: within the interface of the application executed by the user computing device; (a) an interface element including the refined linguistic output; (b) one or more selectable attribute elements, the one or more selectable attribute elements including each of the two or more selectable attribute elements other than the first selectable attribute element; The user computing device of claim 13, further comprising: displaying:

15. The operation includes: receiving an input from the user via the input device associated with the user computing device selecting the second selectable portion of the first selectable attribute element of the two or more selectable attribute elements; in response to receiving the input selecting the second selectable portion of the first selectable attribute element, causing a display of the source document identified by the attribute information included in the first selectable attribute element; The user computing device of claim 12 further comprising:

16. Each of the source documents comprises: One or more web pages of the website; article, newspaper, Books, or A user computing device as described in claim 10, further comprising:

17. Obtaining the text data describing the prompt includes: acquiring a spoken utterance from a user with an audio capture device associated with the user computing device; determining the text data describing the prompt based at least in part on the spoken utterance; and 11. The user computing device of claim 10, comprising:

18. Obtaining the query image includes: obtaining an input indicating a request to capture an image using an image capture device associated with the user computing device; in response to obtaining the input, capturing the query image using the image capture device associated with the user computing device; 11. The user computing device of claim 10, comprising:

19. One or more non-transitory computer-readable storage media collectively storing a first set of instructions that, when executed by one or more processors of a user computing device, cause the user computing device to perform operations, the operations including: Obtaining a query image; obtaining text data describing a prompt; providing the query image and the text data describing the prompt to a computing system associated with a visual search service; receiving, from the computing system in response to providing the query image and the prompt, (a) two or more result images and (b) linguistic output from a machine learning language model, the linguistic output being generated based on the prompt and textual content from a source document that includes the two or more result images; within an interface of an application executed by said user computing device; (a) an interface element including said linguistic output; and (b) two or more selectable attribute elements respectively associated with the two or more result images, each selectable attribute element including a thumbnail of the associated result image and attribute information identifying a source document containing the associated result image; and receiving an input from a user via an input device associated with the user computing device selecting a first selectable attribute element of the two or more selectable attribute elements; in response to receiving the input, providing to the computing system information indicative of a selection of the first selectable attribute element; receiving, in response to providing the information, from the computing system, a refined linguistic output, the refined linguistic output being generated based on the prompt and textual content from a source document including the two or more result images other than a first result image associated with the first selectable attribute element; [0023] In one or more non-transitory computer readable storage media,

20. Each of the source documents comprises: One or more web pages of the website; article, newspaper, Books, or 20. The one or more non-transitory computer-readable storage media of claim 19, comprising:

Citation Information

Patent Citations

  • Document retrieving apparatus, document retrieving method, program, and storage medium

    JP2012084179A

  • Text domain image retrieval

    JP2020115321A

  • Utilizing a digital canvas to conduct a spatial-semantic search for digital visual media

    US20180121768A1

Cited By

  • Text conversion support device and text conversion support method

    JP2026162217A