Visual and audio multi-modal search system

Abstract

A multi-modal search system is described. The system may receive image data captured by a camera of a user device. Additionally, the system may receive audio data associated with the image data. Audio data may be captured by a microphone of a user device. In addition, the system may process the image data to generate visual features. Further, the system may process the audio data to generate a plurality of words. The system may generate a plurality of search terms based on a plurality of words and visual features. The system may then determine one or more search results associated with the plurality of search terms and provide the one or more search results as output.

Description

[0001] CLAIM OF PRIORITY

[0002] This application is based on and claims priority to U.S. Application 18 / 306,638, filed April 25, 2023, which is incorporated by reference herein. TECHNICAL FIELD

[0003] The present disclosure relates generally to processing multiple modalities (e.g., both visual data and audio data) in a search query to provide search results. More particularly, the present disclosure relates to a multi-modal search based on visual data captured by a camera of a user device and audio data captured from a microphone of the user device. BACKGROUND

[0004] In conventional systems, a search query can include a text input or an audio input to search for a particular item or a particular piece of knowledge. However, when a search request is limited to text and audio data, it can be difficult to understand the entire world. Whether an individual is trying to understand what an object in front of them is, trying to determine where else the object can be found, or trying to determine where an image on the internet was captured from, a text and audio search alone can be difficult. In particular, a user can have difficulty determining which words to use. Additionally, the words can not be descriptive enough and / or rich enough to generate a desired result.

[0005] Additionally, content being requested by a user can not be readily provided to the user based on the user not knowing where to search. The user can be requesting search results without an explicit way to express the concept.

[0006] Furthermore, search results based on text or audio input alone can be limited because the user is limited by the mode of input expressing their request. Thus, in conventional systems, when search results are not satisfactory to a user because the search results are limited or incorrect, the user can have to post content to social media or discussion sites in order to crowdsource an answer from other users. SUMMARY

[0007] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or can be obvious from the description, or can be learned through practice of the embodiments.

[0008] One example aspect of the present disclosure relates to a computer-implemented method for multimodal search. The method can include receiving, by a computing system comprising one or more processors, image data captured by a camera of a user device. Additionally, the method can include receiving audio data associated with the image data. The audio data is captured by a microphone of the user device. Further, the method can include processing the image data to generate a visual feature. Moreover, the method can include processing the audio data to generate a plurality of words. The method can also include generating a plurality of search terms based on the plurality of words and the visual feature. Subsequently, the method can include determining one or more search results associated with the plurality of search terms, and providing the one or more search results as output.

[0009] In some implementations, the plurality of search terms are generated based on query refinement. In some implementations, the plurality of search terms are generated using one or more machine-learned models.

[0010] In some implementations, the plurality of search terms can be generated by replacing a word of the plurality of words with an updated word. The updated word can be derived from the visual feature.

[0011] In some implementations, the method can include processing the audio data to generate an audio signature. Additionally, the one or more search results can be determined based on the plurality of search terms and the audio signature.

[0012] In some implementations, the method can include causing the output to be rendered on a graphical user interface. The graphical user interface can be rendered on a display of the user device.

[0013] In some implementations, the audio data can be a refinement query or a pivot query. Additionally, determining the one or more search results includes processing the plurality of search terms to obtain one or more images. The one or more search results can include the one or more images. Moreover, generating the plurality of search terms can include generating one or more text embeddings and one or more image embeddings. The text embeddings and the image embeddings can be combined and input into an image search machine-learned model to obtain the one or more images.

[0014] In some implementations, determining the one or more search results can include processing the plurality of search terms using one or more image search machine-learned models to obtain one or more images. The one or more search results provided as output can include the one or more images. In some instances, the plurality of words can be text embeddings and the visual feature can be an image embedding. For example, the text embeddings and the image embedding can be combined and input into an image search machine-learned model to generate the one or more images.

[0015] In some implementations, determining one or more search results can include processing the plurality of search terms using a web search engine to obtain web results. The one or more search results provided as output can include the web results. In some instances, the visual features can include object names, and the plurality of search terms are keywords derived from the plurality of words and the object names. In some instances, the visual features can include location data, and the plurality of search terms are keywords derived from the plurality of words and the location data.

[0016] In some implementations, determining one or more search results can include processing the plurality of search terms using one or more large language models (LLMs) to obtain generated results. The one or more search results provided as output can include the generated results.

[0017] In some implementations, the method can include causing the one or more search results to be presented on a display of the user device. Additionally, in response to the presentation, the method can include receiving updated image data captured by a camera of the user device. Moreover, the method can include detecting updated visual features in the updated image data using one or more machine-learned models. Furthermore, the method can include determining one or more updated search results associated with the plurality of search terms and the updated visual features. Subsequently, the method can include providing the one or more updated search results as updated output.

[0018] In some implementations, the method can include causing the one or more search results to be presented on a display of the user device. In response to the presentation, the method can include receiving updated audio data captured by a microphone of the user device. Additionally, the method can include processing the updated audio data using one or more machine-learned models to generate updated words. Moreover, the method can include determining one or more updated search results associated with the plurality of search terms and the updated words. Furthermore, the method can include providing the one or more updated search results as updated output.

[0019] In some implementations, the audio data can include contextual information associated with an object in the image data. Additionally, the one or more search results can be determined based on the contextual information. For example, the contextual information can be a brand name of the object. In another example, the contextual information can be a website that sells the object.

[0020] In some implementations, the audio data can include attributes associated with an object in the image data. Additionally, the one or more search results can be determined based on the attributes. For example, the attributes can be a color and / or a pattern associated with the object.

[0021] In some implementations, the contextual information can be a brand name requested by the user and / or a website requested by the user.

[0022] Another example aspect of the disclosure relates to a computing system for conducting a multimodal search. The system can include one or more processors and one or more non-transitory computer-readable media collectively storing instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations can include receiving, by a computing system comprising one or more processors, image data captured by a camera of a user device. Additionally, the operations can include receiving audio data associated with the image data. The audio data is captured by a microphone of the user device. Moreover, the operations can include processing the image data to generate visual features. Further, the operations can include processing the audio data to generate a plurality of words. The operations can also include generating a plurality of search terms based on the plurality of words and the visual features. Subsequently, the operations can include determining one or more search results associated with the plurality of search terms, and providing the one or more search results as output.

[0023] Another example aspect of the disclosure relates to one or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. The operations can include receiving, by a computing system comprising one or more processors, image data captured by a camera of a user device. Additionally, the operations can include receiving audio data associated with the image data. The audio data is captured by a microphone of the user device. Moreover, the operations can include processing the image data to generate visual features. Further, the operations can include processing the audio data to generate a plurality of words. The operations can also include generating a plurality of search terms based on the plurality of words and the visual features. Subsequently, the operations can include determining one or more search results associated with the plurality of search terms, and providing the one or more search results as output.

[0024] Other aspects of the disclosure relate to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices.

[0025] These and other features, aspects, and advantages of various embodiments of the present disclosure will be better understood when considered with reference to the following description and accompanying drawings. The drawings are included to provide a description of the example embodiments of the present disclosure and, together with the description, to explain principles of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0026] With reference to the appended drawings, a detailed discussion of embodiments of the application is set forth herein below, with specific reference to the drawings.

[0027] FIG. 1 A block diagram of an example multimodal search system is depicted in accordance with example embodiments of the present disclosure.

[0028] FIG. 2 A block diagram of an example multimodal search system is depicted in accordance with example embodiments of the present disclosure.

[0029] FIGS. 3A-3G A diagram of an example use case using a multimodal search system is depicted in accordance with example embodiments of the present disclosure.

[0030] FIG. 4 A flow diagram of an example method for conducting a multimodal search is depicted in accordance with example embodiments of the present disclosure.

[0031] FIG. 5 A flow diagram of an example method for conducting a subsequent multimodal search in response to receiving updated image data is depicted in accordance with example embodiments of the present disclosure.

[0032] FIG. 6 A flow diagram of an example method for conducting a subsequent multimodal search in response to receiving updated audio data is depicted in accordance with example embodiments of the present disclosure.

[0033] FIG. 7A A block diagram of an example computing system for conducting a multimodal search is depicted in accordance with example embodiments of the present disclosure.

[0034] FIG. 7B A block diagram of an example computing system for conducting a multimodal search is depicted in accordance with example embodiments of the present disclosure.

[0035] Reference numerals repeated across multiple drawings are intended to identify identical features in various implementations. DETAILED DESCRIPTION

[0036] SUMMARY

[0037] Generally, the present disclosure relates to systems and methods for improving search results by using visual data and audio data as part of a search query. In particular, the systems and methods disclosed herein can utilize visual data and audio data to provide multimodal (e.g., multi-search, search that incorporates audio and visual inputs) search capabilities and multimodal outputs.

[0038] For example, a user can take a picture of a rosemary plant and simultaneously or subsequently ask “what can I cook with this?” The techniques described herein can input the image data and audio data into one or more machine-learned models to generate improved results. Continuing this example, the system can determine that the object in the image is a rosemary plant and replace the term “this” with “rosemary” and input this query into a web search to obtain web results for this query. In another example, a user can be using a live streaming application and simultaneously open a search application (e.g., a viewfinder) so that the user verbally submits a search request related to an image captured in the live streaming application. In yet another example, a user can verbally instruct a mobile phone or an auxiliary device to take a photo and issue a query such as “hey take a photo and identify what is in the photo.”

[0039] According to some embodiments, the systems described herein improve a multi-modal experience by introducing a user’s voice input and images captured directly from a camera by the user to reduce the friction of formulating search queries. The systems support a multi-modal experience, which is a novel way of letting users utilize multiple modalities (e.g., image and text, image and audio) simultaneously to formulate a question. Thus, users can search using multiple modalities (e.g., image and text, image and audio, text and audio) simultaneously. The systems can support the growth of information-seeking query flows and introduce new types of queries. Moreover, the systems can utilize large language models (LLMs) to improve search results associated with opinion and action-seeking queries. The systems can utilize machine-learned models to present artificial intelligence (AI)-generated answers and outputs.

[0040] The techniques described herein significantly reduce the friction of formulating multi-modal queries, which improves the user experience. For example, the user experience can be improved by enabling a user to take a picture of a product with their mobile device and simultaneously state “add it to my shopping list.” In response to this request, the system can determine the product captured by the camera of the mobile device and add the product to the user’s shopping list.

[0041] Further, by supporting new ways of inputting data for a search request, the system increases the search space for potential search results. By incorporating audio input in conjunction with image input, the system is able to increase the search space for potential search results, which in turn improves the user experience. In some embodiments, the system transcribes the audio input into a plurality of words and combines the words with features extracted from the image data, which makes it easier for the user to conduct a search. In some other embodiments, the system not only transcribes the audio data and addresses use cases that require the system to truly understand the audio and image. For example, when a user's dishwasher makes a particular gurgling sound, it can be difficult, if not impossible, to describe the sound or problem in words. The system described herein can receive audio data associated with the gurgling sound and an image of the dishwasher and return search results that fix the dishwasher. The system can determine the model and brand of the dishwasher from features extracted in the image of the dishwasher. Additionally, the system can determine that the audio data is associated with a clogged drain by extracting a signature from the audio data and determining that the signature of the audio data is similar to a signature of audio associated with a clogged drain for that particular model and brand. Continuing with this example, the search results can include that the dishwasher is a particular brand and model, the problem is that the dishwasher is clogged, and present a video tutorial that solves this problem. In some implementations, the search results can include augmented reality instructions for solving the problem with the dishwasher.

[0042] The systems and methods of the present disclosure provide a number of technical effects and benefits. As one example, the systems and methods can improve search results by supporting multi-modal search capabilities. Additionally, by simultaneously using image data (e.g., image embeddings) and audio data (e.g., text embeddings), the system can provide more accurate search results by augmenting queries with additional signals that provide helpful context for the search. For example, word embeddings and image embeddings can be real-valued vectors that encode the meaning of words and / or images, and can be simultaneously searched in a multi-dimensional vector space to provide more accurate search results. Moreover, by supporting multi-modal search capabilities, the system increases the search space for potential search results by now being able to conduct searches that were not previously possible. In some instances, the system can process audio data to generate an input audio signature and determine search results based on the input audio signature. As highlighted by the clogged dishwasher example above, the user is able to conduct searches that were not previously possible, and the system is now able to provide tutorials on how to solve the problem by simultaneously analyzing audio and image data. In some instances, search results are provided to the user more quickly by reducing the number of interactions with the user. In particular, the systems and methods disclosed herein can utilize an interactive user interface that enables the user to use both image data and audio data to provide better, faster, and more accurate search results. Moreover, by extracting only a subset of known audio signatures from an audio database that are associated with visual features extracted from image data (e.g., accessing known audio signatures for a particular brand and model of dishwasher) and then comparing the input audio signature to this subset of known audio signatures, search queries can be conducted more quickly than conventional systems in order to present search results in real-time.

[0043] Another technical effect and benefit relates to improved computational efficiency and improvements in the functionality of computing systems. For example, the systems and methods disclosed herein can utilize a multi-modal search system to provide more comprehensive multi-modal search queries that can reduce the use of additional searches and additional search result page browsing, which can save time, processing resources, energy, and computing power.

[0044] FIG. 1 A block diagram of an example multi-modal search system 100 in accordance with example embodiments of the present disclosure is depicted. The multi-modal search system 100 can receive multi-modal input 101 to generate multi-modal output 125. For example, the multi-modal search system 100 can be an image and audio search system that processes image input 102 and voice input 104 to generate image results 116, web results 118, and / or generated results 120. The image input 102 can be captured by a camera of a user device. The audio input 104 can be captured by a microphone of the user device. Additionally, the audio input 104 can be associated with the image input 102.

[0045] According to some embodiments, the audio input 104 can be a plurality of words associated with a particular category, such as a search query, a question, an inquiry, a command, an operation to be performed, a text string, and / or a response to a prompt. The system 100 can determine the multi-modal output 125 based on the category of the audio input. For example, if the audio input 104 is a question (e.g., “what can I do with the object in this image”), then the multi-modal output 125 can be a web result based on a web search 116. Alternatively, if the particular category is an action to be performed (e.g., summarize the text in this image), then the multi-modal output 125 can be a generative result 120 generated by a large language model 118. In yet another alternative output, if the particular category is an image search query (e.g., find similar dresses like this image), then the multi-modal output 125 can be an image result 116 obtained from an image search 110. The image search 110 can be based on the image features (e.g., image embeddings 114) determined from the image input 102 based on the query refinement 108. Additionally, the image search 110 can be based on one or more machine-learned models from the text embeddings 112 and / or image embeddings determined from the audio input 104 and / or transcribed text 106. For example, the text embeddings 112 and image embeddings 114 can be input into an image search 110 model (e.g., a reverse image search engine, an image classification model, a convolutional neural network (CNN) model) to provide a plurality of image results as an output of the image search 110.

[0046] Query refinement can include a process of reformulating a given query to improve retrieval performance in information retrieval operations, particularly in the context of query understanding. In the context of a search engine, query refinement can involve determining the intent of a user by evaluating a user input (e.g., image input 102, audio input 104) and refining the search query based on the intent of the user. Query refinement involves techniques such as finding synonyms of words and searching for the synonyms as well, finding semantically related words (e.g., antonyms, partial words, hyponyms, hypernyms), finding various morphological forms of words by performing stemming on each word in the search query, fixing spelling errors and automatically searching for the corrected forms, and re-weighting terms in the original query.

[0047] In some examples, the system 100 can receive the multi-modal input 101 and process the data to generate a multi-modal output 125 that depends on the audio input 104. For example, the image input can be an image of a rosemary plant and the audio input 104 can be the question "what can I make with this?" The system can use a machine-learned model (e.g., a natural language processing (NLP) speech-to-text model) to convert the audio input 104 to transcribed text 106. The transcribed text can be a plurality of words. In some examples, both the transcribed text 106 and the image input 102 can be input into the query refinement 108 to replace one of the words in the transcribed text 106 with an updated word. The updated word can be derived from a visual feature of the image input. The machine-learned model can determine the visual features (e.g., image embeddings 114) from the image input 102. In this example, one of the determined visual features can be that the image is of a rosemary plant. Thus, the query refinement 108 can replace the word "this" in the transcribed text 106 with the updated word "rosemary." Subsequently, the system 100 can conduct a web search 116 based on the category of the audio input being a question. The web results 118 provided by the system 100 based on the web search 116 can be cooking instructions that include rosemary as an ingredient.

[0048] FIG. 2 A block diagram of an example multi-modal search system 200 is depicted in accordance with example embodiments of the present disclosure. The multi-modal search system 200 can process image data 202 (e.g., data derived from the image input 102) and audio data 204 (data derived from the speech input 104) to generate search results 214. The image data 202 can be transmitted by a user device of a user to the system 200 via a network. In some examples, the image data 202 can be an image of one or more objects. Alternatively, the image data can be a plurality of images, where each image includes one or more objects. The audio data 204 can be an audio input associated with the image data 202. The audio data 204 can be captured by a microphone of the user device and transmitted to the system 200 via a network. In some examples, the camera and microphone can be components of the same user device (e.g., a mobile phone of the user). The audio data 204 is an audio input that can be transcribed into a plurality of words.

[0049] System 200 can use one or more machine learning models to process image data 202 to generate visual features 206 (e.g., visual objects, image embeddings 114). For example, image data 202 can be input into an image classification machine learning model to determine visual features 206 (e.g., objects in an image, image embeddings). Additionally, system 200 can process audio data 204 to generate multiple words 208. A natural language processing model can be used to transcribe audio data 204 to determine multiple words associated with audio data 204. Subsequently, visual features 206 and multiple words 208 are input into query refinement 210 to generate multiple search terms 212. For example, the machine learning model may include a replacement model that replaces words in multiple words 208 with updated words derived from visual features 206. Furthermore, system 200 may determine search results 214 based on multiple search terms and provide search results 214 as output. Search results 214 may include image results 216, web results 216, and generated results 220. Search results 214 can be based on the categories of audio data 204 and / or image data 202, as previously stated in FIG. 1 As discussed in the article.

[0050] The image data 202, audio data 204, and / or historical data from the user can be used through query refinement 210. FIG. 2 Information derived from (not shown) the data is used to generate (e.g., determine) multiple search terms. Additionally and / or alternatively, multiple search terms may be generated by a machine learning model using information derived from the user's image data 202, audio data 204, and / or historical data. For example, historical data may describe past interactions of the user when using one or more specific words. In some implementations, the user and / or multiple users may refine the search results to images when using one or more specific words. Alternatively and / or additionally, one or more specific words may often be used to describe an image (e.g., in image caption text). One or more specific words may be identified as associated with visual feature 206 based on common associations with the image and / or image features. In some implementations, the natural language meaning of words or phrases may be used to determine that one or more specific words are associated with a visual intent.

[0051] Additionally and / or alternatively, the machine-learned model can be used to determine one or more words of the plurality of words 208. The one or more words of the plurality of words can be associated with the visual features 206. The machine-learned model can transcribe the audio data to textual data, parse the textual data, classify the parsed textual data, and generate the plurality of words 208 associated with the image data 202 and / or the visual features 206. Alternatively and / or additionally, the machine-learned model can include a natural language processing model that can process the audio data 204 in its entirety and / or in various syntactically determined segments to generate the plurality of words 208.

[0052] Example Use Cases

[0053] According to some implementations, the multi-modal search system 200 can be used for multiple use cases. The type of use case can include a refinement query, a perspective query, or an information-seeking query. In some instances, the search results 214 (e.g., image results 216, web results 218, generated results 220) can depend on the type of use case. The type of use case can be determined based on the audio data 204 and / or the image data 202. Additionally, since the system 100, 200 is capable of receiving speech input 104 and / or audio data 204, the user is able to have a more conversational dialogue with the system 100, 200, rather than having the user simply type in keywords.

[0054] In a first category of use cases, the system can make refinement queries, as illustrated in diagram 300 in FIG. 3A In refinement queries, the user can provide additional contextual information that is not readily apparent from the image alone, and the search results can be customized based on the contextual information. In some instances, the contextual information can be a brand or a department. For example, as illustrated in diagram 300 in FIG. 3AAs shown in example 302, the system can receive an image 304 of a bracelet and an audio input 306 stating find similar bracelets from jewelry brand A. In example 302, the system can determine that the context information is that the bracelet is from “Brand A,” which can not be apparent from the image 304 of the bracelet. Similarly, in example 310, the system can receive an image 312 of a top and an audio input 314 stating find similar tops on marketplace B. In example 310, the system can determine that the context information is that the user wants to purchase similar tops on “marketplace B.” In example 330, the system can receive an image 322 of a jacket and an audio input 324 stating show me more women’s versions of this jacket. In example 330, the context information can be determined to be that the jackets returned in the search results should be from the “women’s” department. As described herein, the context information can be an embedding (e.g., text embedding 112) used by a machine-learned model during an image search (e.g., image search 110) in addition to image embeddings (e.g., image embeddings 114) obtained from the received images 304, 312, 322.

[0055] In a second category of use cases, the system can make perspective queries, as shown in illustration 340. FIG. 3B In a perspective query, a user can describe a user preference (e.g., like, dislike) associated with an object in an image. In example 342, the user preference can be an attribute 344 of the object (e.g., collar of a shirt). In example 346, the user preference can be a particular pattern 348 of the object. In example 350, the user preference can be a color 352 of the object in the image. As described in these examples, the user preference can include a visual attribute, color perspective, or color refinement. In example 346, the system can determine the pattern in the image to generate an image embedding 114 associated with this particular pattern. Additionally, a text embedding 112 (e.g., 346) can be “flowerpot,” and both the image embedding 114 and the text embedding 112 can be input into an image search 110 to obtain image results 116.

[0056] In a third category of use cases, the system can make information-seeking queries, such as “I want to do or ask something more with this image and want to express that.” Within the information-seeking use case, there can be multiple subcategories, including answer-seeking, tutorial-seeking, opinion-seeking, action-seeking, and local-seeking.

[0057] In FIG. 3C , the system can make answer-seeking queries of the information-seeking use case, as shown in examples 354, 356, and 358. In answer-seeking queries, the user can request an answer to a long-tail question about an object, such as “how much sunlight does this need?” In example 354, the particular plant depicted in the image can replace the word “this” in the audio input, such that the search query input to the machine-learned model can be “how much sunlight does plant A need?” In example 354, the system can make a web search 116 to determine the answer to the question “how much sunlight does plant A need?”

[0058] In FIG. 3D , the system can make tutorial-seeking queries of the information-seeking use case, as shown in examples 360, 362, and 364. In tutorial-seeking queries, when the user wants to do something to an object and the user is seeking a tutorial to do that thing, the system can provide a tutorial to do a task associated with the object in the image data. In example 360, the user can ask how to fix the object in the image. In this example, the system can determine what the object in the image is and what the problem with the object in the image is, and then make a web search 116 based on this determination.

[0059] In FIG. 3E , the system can make opinion-seeking queries of the information-seeking use case, as shown in examples 366, 368, and 370. For example, when the user is, the system can provide opinions about an object. In example 366, the user can upload a picture of a set of clothes and ask if the set of clothes can be worn to a formal event. In this example, the system can determine the type and visual features of the set of clothes in the image (e.g., image embeddings), and make an image search 110 and / or a web search 116 to determine the answer to the question.

[0060] In FIG. 3GIn some implementations, the system can perform an action search query for an information foraging use case, as shown in examples 372, 374, and 376. For example, a user can submit a query request to the multi-modal search system to perform an action on an image. In response to the request, the system can provide search results associated with an object detected in the image. In example 372, the user can request the system to summarize an image of a page in a book. The system, using the LLM 118, can generate a summary of the page in the book and output the summary as a search result.

[0061] In FIG. 3H, the system can perform a location search query for an information foraging use case, as shown in examples 378, 380, and 382. For example, a user can request local information about an object in an image. In some instances, the image can include metadata, such as temporal information and location information. In example 378, the voice input can be “where can I buy this at a store near me?” In response to the request, the system can determine location data (e.g., location of the object, location of the user, location where the image was taken) based on the metadata of the image. The system can provide web results 118 for the query by using the web search 116 based on the location data, the image input 102, and the audio input 104.

[0062] Example embodiments of the present disclosure will be discussed in further detail with reference to the accompanying drawings.

[0063] Example Method

[0064] FIG. 4 A flow diagram of an example method performed in accordance with example embodiments of the present disclosure is depicted. Although the method is depicted as a sequence of steps, the method of the present disclosure is not limited to the particular order or arrangement of steps shown. Individual steps of the method 400 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure. FIG. 4 For purposes of illustration and discussion, the steps are depicted in a particular order, but the methods of the present disclosure are not limited to the particular order or arrangement shown. Individual steps of the method 400 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0065] The method 400 can be performed by a computing system, such as the multi-modal search system 100, the multi-modal search system 200, the server computing system 730, the sensor processing system 60, or the output determination system 80 in FIG. 1 The method 400 can be performed by a computing system, such as the multi-modal search system 100, the multi-modal search system 200, the server computing system 730, the sensor processing system 60, or the output determination system 80 in

[0066] At 402, a computing system (e.g., the multi-modal search system 100, 200, the server computing system 730) can receive image data captured by a camera of a user device. The image data can be an image input 102 and / or an audio input 104 in FIG. 1 At 402, a computing system (e.g., the multi-modal search system 100, 200, the server computing system 730) can receive image data captured by a camera of a user device. The image data can be an image input 102 and / or an audio input 104 in FIG. 2image data 202 in FIG. 1. The image data can be derived from an image captured by a camera of the user device. The image data can also include metadata associated with the image, such as location data, time data, and / or personal user data. In some instances, the user device at 402 can be the user computing system 702 in FIG. 2 or the user computing system 52 in FIG. 3. FIG. 7A FIG. 7B

[0067] At 404, the computing system can receive audio data associated with the image data. The audio data can be captured by a microphone of the user device. The audio data can be FIG. 1 FIG. 2 FIGS. 3A-3G

[0068] At 406, the computing system can process the image data to generate visual features. In some instances, the image data can be processed by a machine-learned model to extract visual features (e.g., image embeddings 114), as described in FIGS. 1 and 2. In some instances, the query refinement 108 in FIG. 1, FIG. 1 FIG. 2 FIG. 1 FIG. 2 FIG. 7A FIG. 7B

[0069] In some implementations, one or more embedding models can be utilized to process image data to generate one or more visual features (e.g., image embeddings 114). For example, one or more embedding models can be utilized to process one or more images to generate one or more image embeddings 114 in an embedding space. The one or more image embeddings can be associated with one or more visual features of the one or more images. In some implementations, the one or more embedding models can be configured to process multi-modal data to generate multi-modal embeddings. The one or more embeddings can be utilized for classification, search, and / or learning embedding space distributions.

[0070] ​​​​​​​​​​​At 408, the computing system can process the audio data to generate a plurality of words. In some instances, the audio data can be processed by a natural language processing (NLP) model to generate the plurality of words, as described in FIG. 1 and FIG. 2 Alternatively or additionally, the computing system can process the audio data to generate an input audio signature. For example, the computing system can process the audio data of a clogged dishwasher to generate an input audio signature associated with the noise of the clogged dishwasher. In some instances, the machine-learned model 108 in FIG. 1 the machine-learned model 210 in FIG. 2 the machine-learned model 720 and the machine-learned model 740 in FIG. 7A the sensor processing system 60 and the output determination system 80 in FIG. 7B may operate 408. Additionally, NLP models can utilize a set of statistical techniques to identify parts of speech, entities, sentiment, and other aspects of text. Moreover, NLP models can rely on machine learning to automatically learn rules by analyzing a set of examples (i.e., a large corpus) and make statistical inferences on the audio data in order to generate the plurality of words. The machine-learned model (e.g., NLP model) can generate a text embedding based on the audio data and / or the plurality of words.

[0071] At 410, the computing system can generate a plurality of search terms based on the plurality of words and the visual features using one or more machine-learned models. In some instances, the plurality of search terms can be a combination of a text embedding and an image embedding input into an image search (e.g., image search 110) to generate image results. In some implementations, the plurality of search terms can be generated by replacing a word in the plurality of words with an updated word. The updated word can be derived from the visual features. For example, as previously mentioned, a user can take a picture of a rosemary plant and subsequently ask “what can I make with this?” The system can determine that the object in the image is a rosemary plant at 406 and replace the term “this” with “rosemary” at 410.

[0072] In some instances, the machine-learned model 108 in FIG. 1 the machine-learned model 210 in FIG. 2 the machine-learned model 720 and the machine-learned model 740 in FIG. 7A the sensor processing system 60 and the output determination system 80 in FIG. 7B may operate 410.

[0073] At 412, the computing system can determine one or more search results associated with the plurality of search terms. In some implementations, the search results can be determined by the category of the audio input. For example, when the audio input is classified by the machine-learned model as a refinement query or a perspective query, the search results can include image results determined from an image search. In another example, when the audio input is classified by the machine-learned model as an information-seeking query, an opinion-seeking query, an answer-seeking query, a tutorial-seeking query, or a location-seeking query, the search results can include web results determined from a web search. In yet another example, when the audio input is classified by the machine-learned model as an action-seeking query, the search results can include generation results 120 determined from an LLM. FIGS. 3A-3G Different use cases are provided based on the category of the audio input.

[0074] In some implementations, the computing system can further process the audio data at 408 to generate an input audio signature. Additionally, the one or more search results determined at 412 can be based on the plurality of search terms and the input audio signature. For example, the computing system can determine that the audio signature (e.g., the input audio signature) is similar or matches a known audio signature of a known audio file stored in an audio signature database. The audio signature database can be stored in the memory 734 of the server computing system 730 and / or the memory 754 of the third-party computing system 750. The system can access the known audio file (e.g., the plurality of known audio signatures) from the audio database (e.g., the audio signature database) based on the visual features extracted from the image data. For example, the extracted visual features can be a brand and model of a dishwasher or a brand and model of a vehicle, and the plurality of known audio signatures can be associated with known audio data of that particular brand and model. By accessing the known audio signature based on only the extracted visual features, the search can be performed faster, and in turn, the search results can be presented to the user quickly. The input audio signature can be compared to the known audio signature to determine a comparison score (e.g., a similarity value), and when the comparison score exceeds a threshold value, the system can determine a matching audio signature that is similar to the input audio signature. The threshold value can be dynamically updated by the machine-learned model based on user interactions with the search results. In another embodiment, the system can access the plurality of known audio signatures from the audio signature database based on the visual features from the generated visual features, and select a matching audio signature from the known audio signatures when the comparison score of the matching audio signature exceeds a threshold value. A match score of the matching audio signature can be computed by comparing the input audio signature to the matching audio signature. In yet another example, the system can compute a comparison score for each of the plurality of known audio signatures, and select the known audio signature with the highest comparison score as the matching audio signature.

[0075] For example, in a clogged dishwasher use case, the system can process audio input associated with the gurgling sound to generate an input audio signature. The system can determine a brand and model of the dishwasher based on extracted features of the image data. Additionally, the system can access known audio files associated with the extracted features of the image data (e.g., the brand and model of the dishwasher) from an audio database. By extracting and comparing only audio files of the particular brand and model of the dishwasher, the search can be performed quickly, and the results can be presented in real-time. Subsequently, known signatures of the known audio files associated with the particular brand and model of the dishwasher are compared to the input audio signature to compute a similarity value for each known signature. When the similarity value exceeds a threshold, the system can determine that the input audio signature is similar to a first known signature associated with a first known audio file. In this example, the first known audio file can be associated with a clogged drain. Accordingly, the system can determine one or more search results based on the plurality of search terms and the input audio signature at 412, where in this example, the dishwasher is indicated to be clogged. The search results can include a video tutorial on how to fix the clogged dishwasher or an augmented reality instruction for fixing the clogged dishwasher.

[0076] In some implementations, determining the one or more search results can include processing the plurality of search terms using one or more image search machine learning models to obtain one or more images. The one or more search results provided as output can include the one or more images. In some instances, the plurality of words can be text embeddings and the visual features can be image embeddings. For example, the text embeddings and the image embeddings can be combined and input into an image search machine learning model to generate the one or more images.

[0077] In some implementations, determining the one or more search results can include processing the plurality of search terms using a web search engine to obtain web results. The one or more search results provided as output can include the web results. In some instances, the visual features can include object names and the plurality of search terms are keywords derived from the plurality of words and the object names. In some instances, the visual features can include location data and the plurality of search terms are keywords derived from the plurality of words and the location data.

[0078] In some implementations, determining the one or more search results can include processing the plurality of search terms using one or more large language models (LLMs) to obtain generated results. The one or more search results provided as output can include the generated results.

[0079] In some implementations, the audio data can include contextual information associated with the object in the image data. Additionally, the one or more search results can be determined based on the contextual information as well. For example, the contextual information can be a brand name of the object. In another example, the contextual information can be a website that sells the object. In some instances, the contextual information can be a brand name requested by the user and / or a website requested by the user.

[0080] In some implementations, the audio data can include attributes associated with the object in the image data. Additionally, the one or more search results can be determined based on the attributes as well. For example, the attributes can be a color and / or a pattern associated with the object.

[0081] At 414, the computing system can provide the one or more search results as output.

[0082] In some implementations, the plurality of search terms can be generated by replacing a word in the plurality of words with an updated word. The updated word can be derived from the visual features.

[0083] In some implementations, the output can be determined per category of the audio input. For example, when the audio input is classified by the machine-learned model as a refinement query or a perspective query, the output can include image results determined from an image search. In another example, when the audio input is classified by the machine-learned model as an information-seeking query, an opinion-seeking query, an answer-seeking query, a tutorial-seeking query, or a location-seeking query, the output can include web results determined from a web search. In yet another example, when the audio input is classified by the machine-learned model as an action-seeking query, the output can include generated results 120 determined from an LLM.

[0084] In some implementations, the one or more search results at 412 can be web results, image results, and / or generated results. Similarly, the output at 414 can be image results, web results, and / or generated results. The one or more search results and / or the output are based on the category of the audio data, as described previously in the use cases in FIGS. 3A-3G

[0085] In some implementations, the method can include causing the output to be rendered on a graphical user interface. The graphical user interface can be rendered on a display of the user device.

[0086] ​In some implementations, the method can include causing presentation of one or more search results on a display of a user device. Additionally, in response to the presentation, the method can include receiving updated image data captured by a camera of the user device. Moreover, the method can include detecting, using one or more machine-learned models, updated visual features in the updated image data. Further, the method can include determining one or more updated search results associated with the plurality of search terms and the updated visual features. Subsequently, the method can include providing the one or more updated search results as updated output.

[0087] In some implementations, the method can include causing presentation of one or more search results on a display of a user device. In response to the presentation, the method can include receiving updated audio data captured by a capture of the user device. Additionally, the method can include processing, using one or more machine-learned models, the updated audio data to generate updated words. Moreover, the method can include determining one or more updated search results associated with the plurality of search terms and the updated words. Further, the method can include providing the one or more updated search results as updated output.

[0088] FIG. 5 A flow diagram depicting an example method for subsequent multimodal searching in response to receiving updated image data in accordance with example embodiments of the present disclosure is depicted. Although the method 500 is depicted as a series of steps occurring in a specific order, the method of the present disclosure is not limited to the specifically illustrated order or arrangement. Various steps of the method 500 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure. FIG. 5 For the purpose of illustration and discussion, steps are depicted in a particular order, but the method of the present disclosure is not limited to the particular order or arrangement specifically illustrated. Various steps of the method 500 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0089] The method 500 can be performed by a computing system such as the multimodal search system 100, the multimodal search system 200, the server computing system 730, the sensor processing system 60, or the output determination system 80 in FIG. 1 The method 500 can be performed by a computing system such as the multimodal search system 100, the multimodal search system 200, the server computing system 730, the sensor processing system 60, or the output determination system 80 in

[0090] At 502, the computing system can cause presentation of one or more search results on a display of a user device.

[0091] At 504, in response to the presentation, the system can receive updated image data captured by a camera of the user device.

[0092] At 506, the system can detect, using one or more machine-learned models, updated visual features in the updated image data. In some instances, FIG. 1 the query refinement 108 in the multimodal search system 100, FIG. 2 the query refinement 210 in the multimodal search system 200, FIG. 7Athe machine-learned model 720 and the machine-learned model 740 in the system 700, FIG. 7B The sensor processing system 60 and the output determination system 80 in the system 700 can perform operation 506.

[0093] At 508, the system can determine one or more updated search results associated with the plurality of search terms and the updated visual feature.

[0094] At 510, the system can provide the one or more updated search results as updated output.

[0095] FIG. 6 A flow diagram depicting an example method for conducting a subsequent multimodal search in response to receiving updated audio data in accordance with example embodiments of the present disclosure is depicted. Although the method 600 is depicted as a series of acts, it is contemplated that the method 600 can be performed in a different order, or that some acts can be omitted, adapted, or combined. FIG. 6 The method 600 is depicted for purposes of illustration and discussion. The method of the present disclosure is not limited to the particularly illustrated order or arrangement. Individual steps of the method 600 can be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0096] The method 600 can be performed by a computing system such as the multimodal search system 100, the multimodal search system 200, the server computing system 730, the sensor processing system 60, or the output determination system 80 in the system 700. FIG. 1 The method 600 can be performed by a computing system such as the multimodal search system 100, the multimodal search system 200, the server computing system 730, the sensor processing system 60, or the output determination system 80 in the system 700.

[0097] At 602, the computing system can cause one or more search results to be presented on a display of a user device.

[0098] At 604, in response to the presentation, the computing system can receive updated audio data captured by a capture of the user device.

[0099] At 606, the computing system can process the updated audio data using one or more machine-learned models to generate updated words. In some instances, FIG. 1 the query refinement 108 in the system 100, FIG. 2 the query refinement 210 in the system 200, FIG. 7A the machine-learned model 720 and the machine-learned model 740 in the system 700, FIG. 7B The sensor processing system 60 and the output determination system 80 in the system 700 can perform operation 606.

[0100] At 608, the computing system can determine one or more updated search results associated with the plurality of search terms and the updated words.

[0101] At 610, the computing system can provide the one or more updated search results as updated output.

[0102] FIG. 7AA block diagram of an example computing system 100 that performs multimodal search in accordance with example embodiments of the present disclosure is depicted. The system 700 includes a user computing system 702, a server computing system 730, and / or a third computing system 750 communicatively coupled over a network 780.

[0103] The user computing system 702 can include any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0104] The user computing system 702 includes one or more processors 712 and a memory 714. The one or more processors 712 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 714 can include one or more non-transitory computer-readable storage media, such as

[0105] In some implementations, the user computing system 702 can store or include one or more machine-learned models 720. For example, the machine-learned models 720 can be or can otherwise include various machine-learned models, such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including nonlinear models and / or linear models. The neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. The machine-learned models 720 can be FIG. 1 examples of query refinement 108 in FIG. 2 query refinement 210 in and / or machine-learned models used in methods 400, 500, and 600.

[0106] In some implementations, one or more machine-learned models 720 can be received from the server computing system 730 over the network 780, stored in the user computing device memory 714, and then used or otherwise implemented by the one or more processors 712. In some implementations, the user computing system 702 can implement multiple parallel instances of a single machine-learned model 720 (e.g., to perform parallel machine-learned model processing across multiple instances of input data and / or detected features).

[0107] More particularly, the one or more machine-learned models 720 can include one or more detection models, one or more classification models, one or more segmentation models, one or more enhancement models, one or more generative models, one or more natural language processing models, one or more optical character recognition models, and / or one or more other machine-learned models. The one or more machine-learned models 720 can include one or more transformer models. The one or more machine-learned models 720 can include one or more neural radiance fields models, one or more diffusion models, and / or one or more autoregressive language models.

[0108] The one or more machine-learned models 720 can be used to detect one or more object features. The detected object features can be classified and / or embedded. The classification and / or embedding can then be used to conduct a search to determine one or more search results. Alternatively and / or additionally, the one or more detected features can be used to determine that an indicator (e.g., a user interface element indicating the detected feature) is to be provided to indicate that the feature has been detected. The user can then select the indicator to cause the feature to be classified, embedded, and / or searched. In some implementations, the classification, embedding, and / or searching can be conducted prior to the selection of the indicator.

[0109] In some implementations, the one or more machine-learned models 720 can process image data, textual data, audio data, and / or latent encoded data to generate output data that can include image data, textual data, audio data, and / or latent encoded data. The one or more machine-learned models 720 can conduct optical character recognition, natural language processing, image classification, object classification, textual classification, audio classification, context determination, action prediction, image correction, image enhancement, textual enhancement, sentiment analysis, object detection, error detection, repair, video stabilization, audio correction, audio enhancement, and / or data segmentation (e.g., mask-based segmentation).

[0110] Additionally or alternatively, the one or more machine-learned models 740 can be included in or otherwise stored and implemented by a server computing system 730 that communicates with the user computing system 702 according to a client-server relationship. For example, the machine-learned models 740 can be implemented by the server computing system 740 as part of a web service (e.g., a lenser service, a visual search service, an image processing service, an environmental computing service, and / or an overlay application service). Thus, the one or more models 720 can be stored and implemented at the user computing system 702 and / or the one or more models 740 can be stored and implemented at the server computing system 730.

[0111] The user computing system 702 can also include one or more user input components 722 that receive user input (e.g., image input 102, audio input 104). For example, the user input component 722 can be a touch-sensitive component (e.g., a touch-sensitive display screen or trackpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can be used to implement a virtual keyboard. Other example user input components include a microphone, a conventional keyboard, or other means by which a user can provide user input.

[0112] In some implementations, the user computing system can store and / or provide one or more user interfaces 724 that can be associated with one or more applications. The one or more user interfaces 724 can be configured to receive input and / or provide data to be displayed (e.g., image data, textual data, audio data, one or more user interface elements, an augmented reality experience, a virtual reality experience, and / or other data to be displayed). The user interfaces 724 can be associated with one or more other computing systems (e.g., server computing system 730 and / or third-party computing system 750). The user interfaces 724 can include a viewfinder interface, a search interface, a generative model interface, a social media interface, and / or a media content library interface.

[0113] The user computing system 702 can include and / or receive data from (e.g., image data 202, audio data 204) one or more sensors 726. The one or more sensors 726 can be housed in a housing assembly that houses one or more processors 712, memory 714, and / or one or more hardware components that can store and / or cause one or more software packages to execute. The one or more sensors 726 can include one or more image sensors (e.g., cameras), one or more lidar sensors, one or more audio sensors (e.g., microphones), one or more inertial sensors (e.g., inertial measurement units), one or more biometric sensors (e.g., heart rate sensors, pulse sensors, retinal sensors, and / or fingerprint sensors), one or more infrared sensors, one or more location sensors (e.g., GPS), one or more touch sensors (e.g., conductive touch sensors and / or mechanical touch sensors), and / or one or more other sensors. The one or more sensors can be used to obtain data associated with a user’s environment (e.g., an image of the user’s environment, a recording of the environment, and / or a location of the user).

[0114] The user computing system 702 can include and / or be part of a user computing device 704. The user computing device 704 can include a mobile computing device (e.g., a smartphone or tablet), a desktop computer, a laptop computer, a smart wearable, and / or a smart appliance. Additionally and / or alternatively, the user computing system can obtain data from and / or utilize one or more user computing devices 704 to generate data. For example, a camera of a smartphone can be used to capture image data describing an environment, and / or an overlay application of the user computing device 704 can be used to track and / or process data provided to a user. Similarly, one or more sensors associated with a smart wearable can be used to obtain data about a user and / or about an environment of the user (e.g., image data can be obtained with a camera housed in smart glasses of the user). Additionally and / or alternatively, data can be obtained and uploaded from other user devices that can be dedicated to data obtaining or generation.

[0115] The server computing system 730 includes one or more processors 732 and memory 734. The one or more processors 732 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 734 can include one or more non-transitory computer-readable media, such as, for example, RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 734 can store data 736 and instructions 738 that are executed by the processor 732 to cause the server computing system 730 to operate.

[0116] In some implementations, the server computing system 730 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 730 includes multiple server computing devices, such server computing devices can operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0117] As described above, the server computing system 730 can store or otherwise include one or more machine-learned models 740. For example, the models 740 can be or can otherwise include various machine-learned models. Example machine-learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed-forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. See, e.g., FIG. 7B Example models 740 are discussed. The machine-learned models 740 can be FIG. 1 query refinement 108,FIG. 2 Examples of query refinement 210 in the method 400, 500, and / or 600, and / or the machine-learned model used in the method 400, 500, and / or 600.

[0118] Additionally and / or alternatively, the server computing system 730 can include and / or be communicatively coupled with a search engine 742, which can be used to crawl one or more databases (and / or resources). The search engine 742 can process data from the user computing system 702, the server computing system 730, and / or the third-party computing system 750 to determine one or more search results associated with the input data. The search engine 742 can conduct term-based searches, tag-based searches, Boolean-based searches, image searches, embedding-based searches (e.g., nearest neighbor searches), multi-modal searches, and / or one or more other search techniques.

[0119] The server computing system 730 can store and / or provide one or more user interfaces 744 for obtaining input data and / or providing output data to one or more users. The one or more user interfaces 744 can include one or more user interface elements, which can include input fields, navigation tools, content chips, selectable tiles, widgets, data display carousels, dynamic animations, information pop-ups, image enhancements, text-to-speech, speech-to-text, augmented reality, virtual reality, feedback loops, and / or other interface elements.

[0120] The user computing system 702 and / or the server computing system 730 can train the models 720 and / or 740 via interaction with the third-party computing system 750 communicatively coupled with over the network 780. The third-party computing system 750 can be separate from the server computing system 730 or can be part of the server computing system 730. Alternatively and / or additionally, the third-party computing system 750 can be associated with one or more web resources, one or more web platforms, one or more other users, and / or one or more contexts.

[0121] The third-party computing system 750 can include one or more processors 752 and memory 754. The one or more processors 752 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 754 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc. and combinations thereof. The memory 754 can store data 756 and instructions 758 that are executed by the processor 752 to cause the third-party computing system 750 to operate. In some implementations, the third-party computing system 750 includes or is otherwise implemented by one or more server computing devices.

[0122] The network 780 can be any type of communications network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and can include any number of wired or wireless links. In general, communication over the network 780 can be carried out using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL), via any type of wired and / or wireless connection.

[0123] The machine-learned models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0124] In some implementations, the input to the machine-learned models of the present disclosure can be image data (e.g., image data 202). The machine-learned models can process the image data to generate an output. As an example, the machine-learned models can process the image data to generate an image recognition output (e.g., a recognition of the image data, a latent embedding of the image data, an encoded representation of the image data, a hash of the image data, etc.). As another example, the machine-learned models can process the image data to generate an image segmentation output. As another example, the machine-learned models can process the image data to generate an image classification output. As another example, the machine-learned models can process the image data to generate an image data modification output (e.g., an alteration of the image data, etc.). As another example, the machine-learned models can process the image data to generate an encoded image data output (e.g., an encoded representation and / or compressed representation of the image data, etc.). As another example, the machine-learned models can process the image data to generate an upscaled image data output. As another example, the machine-learned models can process the image data to generate a prediction output.

[0125] In some implementations, the input to a machine-learned model of the present disclosure can be text or natural language data. The machine-learned model can process the text or natural language data to generate an output. As an example, the machine-learned model can process natural language data to generate a language encoding output. As another example, the machine-learned model can process text or natural language data to generate a latent text embedding output. As another example, the machine-learned model can process text or natural language data to generate a translation output. As another example, the machine-learned model can process text or natural language data to generate a classification output. As another example, the machine-learned model can process text or natural language data to generate a text segmentation output. As another example, the machine-learned model can process text or natural language data to generate a semantic intent output. As another example, the machine-learned model can process text or natural language data to generate upgraded text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language, etc.). As another example, the machine-learned model can process text or natural language data to generate a prediction output.

[0126] In some implementations, the input to a machine-learned model of the present disclosure can be speech data (e.g., audio data 204). The machine-learned model can process the speech data to generate an output. As an example, the machine-learned model can process speech data to generate a speech recognition output. As another example, the machine-learned model can process speech data to generate a speech translation output. As another example, the machine-learned model can process speech data to generate a latent embedding output. As another example, the machine-learned model can process speech data to generate encoded speech output (e.g., an encoded representation and / or a compressed representation of the speech data, etc.). As another example, the machine-learned model can process speech data to generate upgraded speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine-learned model can process speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine-learned model can process speech data to generate a prediction output.

[0127] In some implementations, an input to a machine-learned model of the present disclosure can be sensor data (e.g., image data 202, audio data 204). The machine-learned model can process the sensor data to generate an output. As an example, the machine-learned model can process the sensor data to generate a recognition output. As another example, the machine-learned model can process the sensor data to generate a prediction output. As another example, the machine-learned model can process the sensor data to generate a classification output. As another example, the machine-learned model can process the sensor data to generate a segmentation output. As another example, the machine-learned model can process the sensor data to generate a segmentation output. As another example, the machine-learned model can process the sensor data to generate a visualization output. As another example, the machine-learned model can process the sensor data to generate a diagnosis output. As another example, the machine-learned model can process the sensor data to generate a detection output.

[0128] In some cases, the input includes visual data (e.g., image data 202) and the task is a computer vision task. In some cases, the input includes pixel data for one or more images and the task is an image processing task. For example, the image processing task can be image classification, in which the output is a set of scores, each score corresponding to a different object class and representing a likelihood that the one or more images depict an object belonging to the object class. The image processing task can be object detection, in which the image processing output identifies one or more regions in the one or more images and, for each region, identifies a likelihood that the region depicts an object of interest. As another example, the image processing task can be image segmentation, in which the image processing output defines, for each pixel in the one or more images, a respective likelihood of each of a predetermined set of classes. For example, the set of classes can be foreground and background. As another example, the set of classes can be object classes. As another example, the image processing task can be depth estimation, in which the image processing output defines, for each pixel in the one or more images, a respective depth value. As another example, the image processing task can be motion estimation, in which the network input includes multiple images and the image processing output defines, for each pixel of one of the input images, a motion of a scene depicted at the pixel between the images in the network input.

[0129] A user computing system can include multiple applications (e.g., application 1 through application N). Each application can include its own respective machine learning library and machine-learned models. For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like.

[0130] Each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to that application.

[0131] The user computing system 702 can include multiple applications (e.g., application 1 through application N). Each application communicates with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a browser application, and the like. In some implementations, each application can use an API (e.g., a public API across all applications) to communicate with the central intelligence layer (and the models stored therein).

[0132] The central intelligence layer can include multiple machine-learned models. For example, a respective machine-learned model (e.g., model) can be provided for each application, and the respective machine-learned models are managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included within or otherwise implemented by the operating system of the computing system 700.

[0133] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized data store for the computing system 700. The central device data layer can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0134] FIG. 7BA block diagram of an example computing system 50 that conducts multimodal searches in accordance with example embodiments of the present disclosure is depicted. In particular, the example computing system 50 can include one or more computing devices 52 that can be used to obtain and / or generate one or more data sets that can be processed by a sensor processing system 60 and / or an output determination system 80 to feedback to a user, which can provide information about features in one or more obtained data sets. The one or more data sets can include image data, textual data, audio data, multimodal data, latent encoded data, and the like. The one or more data sets can be obtained via one or more sensors associated with the one or more computing devices 52 (e.g., one or more sensors in the computing devices 52). Additionally and / or alternatively, the one or more data sets can be stored data and / or retrieved data (e.g., data retrieved from a web resource). For example, a user can interact with an image, text, and / or other content item. The interaction with the content item can then be used to generate one or more determinations.

[0135] The one or more computing devices 52 can obtain and / or generate the one or more data sets based on image capture, sensor tracking, data store retrieval, content download (e.g., downloading an image or other content item from a web resource via the internet), and / or via one or more other techniques. The one or more data sets can be processed with the sensor processing system 60. The sensor processing system 60 can use one or more machine-learned models, one or more search engines, and / or one or more other processing techniques to conduct one or more processing techniques. The one or more processing techniques can be conducted in any combination and / or individually. The one or more processing techniques can be conducted serially and / or in parallel. In particular, the one or more data sets can be processed with a context determination block 62 that can determine a context associated with the one or more content items. The context determination block 62 can identify and / or process metadata, user profile data (e.g., preferences, user search history, user browsing history, user purchase history, and / or user input data), prior interaction data, global trend data, location data, temporal data, and / or other data to determine a particular context associated with the user. The context can be associated with an event, a determined trend, a particular action, a particular type of data, a particular environment, and / or another context associated with the user and / or the retrieved or obtained data.

[0136] The sensor processing system 60 can include an image pre-processing block 64. The image pre-processing block 64 can be used to adjust one or more values of an obtained and / or received image to prepare the image for processing by one or more machine-learned models and / or one or more search engines 74. The image pre-processing block 64 can resize the image, adjust saturation values, adjust resolution, strip and / or add metadata, and / or perform one or more other operations.

[0137] In some implementations, the sensor processing system 60 can include one or more machine-learned models, which can include detection models 66, segmentation models 68, classification models 70, embedding models 72, and / or one or more other machine-learned models. For example, the sensor processing system 60 can include one or more detection models 66, which can be used to detect particular features in a processed dataset. In particular, one or more detection models 66 can be utilized to process one or more images to generate one or more bounding boxes associated with detected features in the one or more images.

[0138] Additionally and / or alternatively, one or more segmentation models 68 can be used to segment one or more portions of a dataset from the one or more datasets. For example, one or more segmentation models 68 can utilize one or more segmentation masks (e.g., one or more segmentation masks generated manually and / or based on one or more bounding boxes) to segment a portion of an image, a portion of an audio file, and / or a portion of text. Segmentation can include isolating one or more detected objects and / or removing one or more detected objects from an image.

[0139] One or more classification models 70 can be used to process image data, text data, audio data, latent encoded data, multi-modal data, and / or other data to generate one or more classifications. The one or more classification models 70 can include one or more image classification models, one or more object classification models, one or more text classification models, one or more audio classification models, and / or one or more other classification models. The one or more classification models 70 can process data to determine one or more classifications.

[0140] In some implementations, one or more embedding models 72 can be utilized to process data to generate one or more embeddings (e.g., text embeddings 112, image embeddings 114). For example, one or more embedding models 72 can be utilized to process one or more images to generate one or more image embeddings (e.g., image embeddings 114) in an embedding space. The one or more image embeddings can be associated with one or more image features of the one or more images. In some implementations, one or more embedding models 72 can be configured to process multi-modal data to generate multi-modal embeddings. The one or more embeddings can be utilized for classification, search, and / or learning embedding space distributions.

[0141] The sensor processing system 60 can include one or more search engines 74 that can be utilized to conduct one or more searches. The one or more search engines 74 can crawl one or more databases (e.g., one or more local databases, one or more global databases, one or more private databases, one or more public databases, one or more specialized databases, and / or one or more general-purpose databases) to determine one or more search results. The one or more search engines 74 can conduct feature matching, text-based searching, embedding-based searching (e.g., k-nearest neighbor searching), metadata-based searching, multi-modal searching, web resource searching, image searching, text searching, and / or application searching.

[0142] Additionally and / or alternatively, the sensor processing system 60 can include one or more multi-modal processing blocks 76 that can be utilized to assist in processing multi-modal data. The one or more multi-modal processing blocks 76 can include generating multi-modal queries and / or multi-modal embeddings to be processed by one or more machine-learned models and / or one or more search engines 74.

[0143] The output of the sensor processing system 60 can then be processed to determine one or more outputs to be provided to a user utilizing an output determination system 80. The output determination system 80 can include heuristic-based determinations, machine-learned model-based determinations, user-selected based determinations, and / or context-based determinations.

[0144] The output determination system 80 can determine how and / or where to provide one or more search results in the search results interface 82. Additionally and / or alternatively, the output determination system 80 can determine how and / or where to provide one or more machine-learned model outputs in the machine-learned model output interface 84. In some implementations, one or more search results and / or one or more machine-learned model outputs can be provided for display via one or more user interface elements. The one or more user interface elements can be overlaid on displayed data. For example, one or more detection indicators can be overlaid on a detected object in a viewfinder. The one or more user interface elements can be selectable to conduct one or more additional searches and / or one or more additional machine-learned model processes. In some implementations, user interface elements can be provided as application-specific user interface elements and / or can be provided uniformly across different applications. The one or more user interface elements can include pop-up displays, interface overlays, interface tiles and / or tiles, carousel interfaces, audio feedback, animations, interactive widgets, and / or other user interface elements.

[0145] Additionally and / or alternatively, data associated with the output of the sensor processing system 60 can be used to generate and / or provide an augmented reality experience and / or a virtual reality experience 86. For example, one or more obtained data sets can be processed to generate one or more augmented reality rendering assets and / or one or more virtual reality rendering assets, which can then be used to provide an augmented reality experience and / or a virtual reality experience 86 to a user. An augmented reality experience can render information associated with an environment into the respective environment. Alternatively and / or additionally, objects related to the processed data sets can be rendered into a user’s environment and / or a virtual environment. Rendering data set generation can include training one or more neural radiance field models to learn three-dimensional representations of one or more objects.

[0146] In some implementations, one or more action prompts 88 can be determined based on the output of the sensor processing system 60. For example, a search prompt, a purchase prompt, a generation prompt, a reservation prompt, a call prompt, a redirection prompt, and / or one or more other prompts can be determined to be associated with the output of the sensor processing system 60. The one or more action prompts 88 can then be provided to a user via one or more selectable user interface elements. In response to a selection of the one or more selectable user interface elements, a respective action of the respective action prompt can be conducted (e.g., a search can be conducted, a purchase application programming interface can be utilized, and / or another application can be opened).

[0147] In some implementations, one or more generative models 90 can be utilized to process one or more data sets and / or outputs of the sensor processing system 60 to generate model-generated content items, which can then be provided to a user. The generation can be prompted based on a user selection and / or can be performed automatically (e.g., automatically based on one or more conditions, which can be associated with a threshold amount of unidentified search results).

[0148] The output determination system 80 can utilize a data augmentation block 92 to process one or more data sets and / or outputs of the sensor processing system 60 to generate augmented data. For example, one or more images can be processed with the data augmentation block 92 to generate one or more augmented images. Data augmentation can include data correction, data cropping, removal of one or more features, addition of one or more features, resolution adjustment, lighting adjustment, saturation adjustment, and / or other augmentations.

[0149] In some implementations, one or more data sets and / or outputs of the sensor processing system 60 can be stored based on a data storage block 94 determination.

[0150] The output of the output determination system 80 can then be provided to a user via one or more output components of the user computing device 52. For example, one or more user interface elements associated with one or more outputs can be provided for display via a visual display of the user computing device 52.

[0151] The process can be performed iteratively and / or continuously. One or more user inputs for providing user interface elements can adjust and / or influence the continuous processing loop.

[0152] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems and actions taken by and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and

[0153] While the subject matter has been described in detail with respect to various specific embodiments of the subject matter, it should be understood that the various examples were meant to be illustrative only and not limiting to the scope of the disclosure. Changes, modifications and equivalents to the examples described above can be made by those of ordinary skill in the art, with the foregoing description being understood as being by way of example only. Thus, the subject disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. For example, features illustrated or described as part of one embodiment, can be used with another embodiment to yield still a further embodiment. Thus, it is intended that the subject disclosure encompass such modifications, changes and / or additions as are within the scope of the disclosure.

[0154] ADDITIONAL DISCLOSURE

[0155] The technology discussed herein makes reference to servers, databases, software applications and other computer-based systems and actions taken by and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and

[0156] While the subject matter has been described in detail with respect to various specific embodiments of the subject matter, it should be understood that the various examples were meant to be illustrative only and not limiting to the scope of the disclosure. Changes, modifications and equivalents to the examples described above can be made by those of ordinary skill in the art, with the foregoing description being understood as being by way of example only. Thus, the subject disclosure is not intended to be limited to the examples described herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. For example, features illustrated or described as part of one embodiment, can be used with another embodiment to yield still a further embodiment. Thus, it is intended that the subject disclosure encompass such modifications, changes and / or additions as are within the scope of the disclosure.

Claims

1. A computer-implemented method for multimodal search, the method comprising: Image data captured by a camera on a user device is received by a computing system including one or more processors; Receive audio data associated with the image data, the audio data being captured by the microphone of the user device; The image data is processed to generate visual features; The audio data is processed to generate multiple words; Multiple search terms are generated based on the multiple words and the visual features, wherein the multiple search terms are generated by replacing words in the multiple words with updated words derived from the visual features; Determine one or more search results associated with the plurality of search terms; as well as Provide one or more search results as output.

2. The method of claim 1, further comprising: Process the audio data to generate an input audio signature; and The determination of the one or more search results is also based on the input audio signature.

3. The method of claim 2, wherein determining the one or more search results comprises: Access multiple known audio signatures from an audio signature database based on visual features derived from generated visual features; as well as Select a matching audio signature from the known audio signatures, wherein the comparison score of the matching audio signature exceeds a threshold, and the matching score of the matching audio signature is calculated by comparing the input audio signature with the matching audio signature.

4. The method of claim 1, further comprising: The output is displayed on a graphical user interface, which is shown on the display of the user device.

5. The method of claim 1, wherein the audio data is a refined query or a perspective query, and wherein determining the one or more search results includes: Process the multiple search terms to obtain one or more images; The one or more search results mentioned therein include the one or more images.

6. The method of claim 5, wherein generating the plurality of search terms includes generating one or more text embeddings and one or more image embeddings.

7. The method of claim 6, wherein the text embedding and the image embedding are combined and input into an image search machine learning model to obtain the one or more images.

8. The method of claim 1, wherein the audio data is an information search query, wherein determining the one or more search results includes: Use a web search engine to process the multiple search terms to obtain web results; The one or more search results mentioned above include the web results.

9. The method of claim 8, wherein the visual feature includes an object name, and wherein the plurality of search terms are keywords derived from the plurality of words and the object name.

10. The method of claim 8, wherein the visual features include location data, and wherein the plurality of search terms are keywords derived from the plurality of words and the location data.

11. The method of claim 1, wherein the audio data is an action search query, wherein determining the one or more search results includes: One or more large language models (LLMs) are used to process the multiple search terms to obtain the generated results; The generated results are included in one or more of the search results.

12. The method of claim 1, further comprising: To display the one or more search results on the user device's display; In response to the presentation, updated image data captured by the camera of the user device is received; One or more machine learning models are used to detect updated visual features in the updated image data; Determine one or more updated search results associated with the plurality of search terms and the updated visual features; as well as Provide one or more updated search results as updated output.

13. The method of claim 1, further comprising: To display the one or more search results on the user device's display; In response to the presentation, receive the captured and updated audio data from the user device. The updated audio data is processed using one or more machine learning models to generate updated words; Determine one or more updated search results associated with the plurality of search terms and the updated words; as well as Provide one or more updated search results as updated output.

14. The method of claim 1, wherein the audio data includes contextual information associated with objects in the image data, and wherein the one or more search results are further determined based on the contextual information.

15. The method of claim 14, wherein the context information is a brand name requested by the user.

16. The method of claim 14, wherein the context information is a website requested by the user.

17. The method of claim 1, wherein the audio data includes attributes associated with objects in the image data, wherein one or more search results are further determined based on the attributes, and wherein the attributes are colors or patterns associated with the objects.

18. The method of claim 1, wherein the output is an image result, a web result, or a generated result, and wherein the output is based on the category of the audio data.

19. A computing system, the system comprising: One or more processors; as well as One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media collectively storing instructions that, when executed by the one or more processors, cause the computing system to operate, the operations including: Image data captured by a camera on a user device is received by a computing system including one or more processors; Receive audio data associated with the image data, the audio data being captured by the microphone of the user device; The image data is processed to generate visual features; The audio data is processed to generate multiple words; Multiple search terms are generated using one or more machine learning models based on the plurality of words and the visual features, wherein the plurality of search terms are generated by replacing words in the plurality of words with updated words derived from the visual features; Determine one or more search results associated with the plurality of search terms; and Provide one or more search results as output.

20. One or more non-transitory computer-readable media, the one or more non-transitory computer-readable media collectively storing instructions, the instructions causing the one or more computing devices to operate when executed by the one or more computing devices, the operations including: Receive image data captured by the user device's camera; Receive audio data associated with the image data, the audio data being captured by the microphone of the user device; The image data is processed to generate visual features; The audio data is processed to generate multiple words; Multiple search terms are generated using one or more machine learning models based on the plurality of words and the visual features, wherein the plurality of search terms are generated by replacing words in the plurality of words with updated words derived from the visual features; Determine one or more search results associated with the plurality of search terms; as well as Provide one or more search results as output.

Images