Application prediction based on visual search determination

Abstract

A visual search in an operating system of a computing device may process providing content for display and providing additional information about the content. The computing device may include an operating system including a visual search interface that retrieves display data associated with content currently provided for display and processes the display data. The visual search interface may generate display data based on current content provided for display, process the display data using one or more on-device machine learning models, and provide additional information to the user. The visual search interface may send data associated with the display data to perform additional data processing tasks. Application suggestions may be determined and provided based on the visual search data.

Description

[0001] Related Applications

[0002] This application is based upon and claims priority to U.S. Non-Provisional Patent Application No. 18 / 486,663, filed on October 13, 2023, which claims priority to and the benefit of U.S. Provisional Patent Application No. 63 / 584,539, filed on September 22, 2023. Applicants claim priority to and the benefit of each of such applications, and all of such applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to application prediction based on visual search results. More specifically, the present disclosure relates to a visual search interface in an operating system of a computing device that determines application suggestions based on determined visual search data. Background Art

[0004] Understanding the world at large can be difficult. Whether an individual is trying to understand what an object is in front of them, trying to determine where to find the object, and / or trying to determine where an image on the Internet was captured, conducting a text search alone can be difficult. Specifically, users may have difficulty determining which words to use. Additionally, the words may not be descriptive and / or rich enough to produce the desired results.

[0005] In addition, when the data is visual and / or niche, it may be difficult to obtain additional information associated with the information provided for display across different applications and / or media files. Therefore, it may be difficult for a user to attempt to construct a search query to search for the additional information. In some cases, a user may capture a screenshot and use the screenshot as a query image. However, this search may result in irrelevant search results associated with items that the user is not interested in. In addition, screenshot capture and / or screenshot cropping may rely on several user inputs provided, which may still fail to provide the desired results.

[0006] In addition, the content requested by the user may not be readily available and / or understandable to the user based on the user not knowing where to search, based on the storage location of the content, and / or based on the content not existing. The user may request search results based on an imaginary concept, but there is no clear way to express the imaginary concept. Summary of the invention

[0007] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0008] An example aspect of the present disclosure relates to a computing system. The system may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include obtaining display data. The display data may describe content currently presented for display in a first application on a user computing device. The operations may include processing at least a portion of the display data to generate visual search data. The visual search data may include one or more visual search results. The one or more visual search results may be associated with detected features in the display data. The operations may include determining that a specific second application on the computing device is associated with the visual search data, and providing application suggestions associated with the specific second application based on the visual search data.

[0009] Another example aspect of the present disclosure relates to a computer-implemented method. The method may include acquiring display data by a computing system including one or more processors. The display data may describe content currently presented for display on a user computing device. The method may include processing at least a portion of the display data by the computing system to generate visual search data. The visual search data may include one or more visual search results. The one or more visual search results may be associated with detected features in the display data. The method may include processing the visual search data by the computing system to determine that a second application is associated with the one or more visual search results, and providing, by the computing system, an application suggestion associated with the second application based on the visual search data.

[0010] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. The operations may include obtaining display data. The display data may describe content presented for display on a user computing device. The operations may include processing the display data using one or more on-device machine learning models to generate segmented portions of the display data. The segmented portions of the display data may include data describing a feature set of content presented for display. The operations may include sending the segmented portions of the display data to a server computing system and receiving visual search data from the server computing system. The visual search data may include one or more search results. The one or more search results may be associated with detected features in the segmented portions of the display data. The operations may include processing the visual search data to determine a plurality of candidate second applications associated with the one or more search results. The operations may include obtaining a selection of a particular application suggestion to send at least a portion of the visual search data to a particular second application of the plurality of candidate second applications. The operations may include obtaining a model-generated content item based on the selection of the particular application suggestion. The model-generated content item may have been generated using a generative model based on a portion of the visual search data. The operations may include providing the model-generated content item to a particular second application.

[0011] Another example aspect of the present disclosure relates to a computing device. The device may include a visual display. The visual display may display a plurality of pixels. The plurality of pixels may be configured to display content associated with one or more applications. The device may include an operating system. The operating system may include a visual search interface. The visual search interface may be at the operating system level. The visual search interface may obtain display data associated with content currently provided for display by the visual display, and may process the display data using one or more on-device machine learning models. The device may include a wireless network component. The wireless network component may include a communication interface for communicating with one or more other computing devices. The device may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing device to perform operations.

[0012] Another example aspect of the present disclosure relates to a mobile computing device. The device may include a visual display. The visual display may display a plurality of pixels. The plurality of pixels may be configured to display content associated with one or more applications. The device may include an operating system. The operating system may include an operating system-level visual search interface. The visual search interface may include a display capture component. The display capture component may obtain display data associated with content currently provided for display by the visual display. The visual search interface may include an object detection model. The object detection model may process the display data to determine that one or more objects are depicted. The visual search interface may include a segmentation model. The segmentation model may segment an area depicting one or more objects to generate image fragments. The visual search interface may include a server interface. The server interface may send the image fragments to a server computing system. The device may include a wireless network component. The wireless network component may include a communication interface for communicating with one or more other computing devices. The device may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing device to perform operations.

[0013] Another example aspect of the present disclosure relates to a computing system for visual search. The system may include a visual display. The visual display may display a plurality of pixels. The plurality of pixels may be configured to display content associated with one or more applications. The system may include a computing device. The computing device may be communicatively connected to the visual display. The computing device may include an operating system, a network component, one or more processors, and one or more non-transitory computer-readable media that store instructions together, and the instructions cause the computing device to perform operations when executed by one or more processors. The operating system may include an operating system-level visual search application. The visual search application may include an overlay interface and one or more on-device machine learning models. The overlay interface may acquire display data associated with content currently provided for display by the visual display in response to receiving user input. One or more on-device machine learning models may have been trained to process image data to generate one or more machine learning outputs based on detected features in the display data. The network component may include a communication interface for communicating with one or more other computing devices.

[0014] Another example aspect of the present disclosure relates to a computing system. The system may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations. The operations may include obtaining a prompt at a computing device. The prompt may describe a request for information from one or more applications on the computing device. The operations may include processing the prompt to determine a plurality of content items associated with the one or more applications. The plurality of content items may be determined by accessing data associated with the one or more applications on the computing device. The operations may include processing the plurality of content items using a machine learning model to generate a structured output. The structured output may include information from the plurality of content items extracted in a structured data format. The operations may include providing a structured output at the computing device for display as a response to the prompt.

[0015] Another example aspect of the present disclosure relates to a computer-implemented method. The method may include obtaining a prompt by a computing system including one or more processors. The prompt may describe a request for information from one or more applications on the computing system. The method may include processing the prompt by the computing system to determine a plurality of content items associated with the one or more applications. The plurality of content items may be determined by accessing data associated with the one or more applications on the computing system. The method may include processing the plurality of content items by the computing system using a machine learning model to generate a structured output. The structured output may include information from the plurality of content items extracted in a structured data format. The structured output may include formatting that is different from the native format of the plurality of content items. The method may include providing, by the computing system, a structured output for display as a response to the prompt.

[0016] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations. The operations may include obtaining a prompt at a computing device. The prompt may describe a request for information from one or more applications on the computing device. The operations may include processing the prompt to determine a plurality of content items associated with the one or more applications. The plurality of content items may be determined by accessing data associated with the one or more applications on the computing device. The plurality of content items may include one or more multimodal content items. The operations may include processing the plurality of content items using a machine learning model to generate a structured output. The structured output may include information from the plurality of content items extracted in a structured data format. The structured output may include multimodal data. The operations may include providing a structured output at a computing device for display as a response to the prompt.

[0017] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0018] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0020] Figure 1 Depicted is a block diagram of an example visual search system according to an example embodiment of the present disclosure.

[0021] Figure 2 Depicted is a block diagram of an example visual search in an operating system according to an example embodiment of the present disclosure.

[0022] Figure 3 Depicted is a flow chart of an example method for performing display data processing according to an example embodiment of the present disclosure.

[0023] FIG. 4A to FIG. 4D Depicted is a diagram of an example visual search interface according to an example embodiment of the present disclosure.

[0024] FIG. 5A to FIG. 5D Depicted is a diagram of an example data transfer interface according to an example embodiment of the present disclosure.

[0025] FIG. 6A to FIG. 6E Depicted is a diagram of an example data call interface according to an example embodiment of the present disclosure.

[0026] FIG. 7A to FIG. 7B Depicted is a diagram showing a data processing interface displayed on an example device according to an example embodiment of the present disclosure.

[0027] Figure 8 Depicted is a flow chart of an example method for performing a data push according to an example embodiment of the present disclosure.

[0028] Fig. 9 Depicted is a block diagram of an example application suggestion system according to an example embodiment of the present disclosure.

[0029] Fig.10 Depicted is a flow chart of an example method for performing a data call according to an example embodiment of the present disclosure.

[0030] Fig.11 Depicted is a block diagram of an example data aggregation system according to an example embodiment of the present disclosure.

[0031] Fig. 12A Depicted is a block diagram of an example computing system that performs visual search at the operating system level according to an example embodiment of the present disclosure.

[0032] Fig. 12B Depicted is a block diagram of an example computing system that performs visual search at the operating system level according to an example embodiment of the present disclosure.

[0033] Reference numerals that are repeated across multiple drawings are intended to identify like features in the various implementations. DETAILED DESCRIPTION

[0034] In general, the present disclosure relates to systems and methods for visual search at an operating system level. Specifically, the systems and methods disclosed herein can utilize a visual search interface in an operating system of a computing device. The visual search interface can generate display data describing content provided for display on a computing device, and can process the display content to determine information associated with the displayed content. The visual search interface may include a display capture component for generating display data, one or more on-device machine learning models for processing the display data, and / or a transmission component for interfacing with one or more other computing systems.

[0035] Visual search in an operating system may include an interface at the operating system level that a user may utilize to process visual data across applications executed by a computing device. The visual search interface may be invoked by user input, which may include voice commands, touch gestures, and / or one or more other user inputs. Visual search in an operating system may be included in a mobile computing device (e.g., a smartphone, a tablet computer, and / or a smart wearable device), a smart television, a smart appliance, and / or a desktop computing device. In some implementations, the visual search interface may be implemented as an extension and / or overlay interface of a web browser.

[0036] When the data is visual, niche, and / or not selectable in its current native form, it may be difficult to obtain additional information associated with providing information for display across different applications and / or media files. Therefore, it may be difficult for a user to attempt to construct a search query to search for the additional information. In some cases, a user may capture a screenshot and use the screenshot as a query image. However, this search may result in irrelevant search results associated with items that the user is not interested in. In addition, screenshot capture and / or screenshot cropping may rely on several user inputs being provided, which may still fail to provide the desired results.

[0037] Visual search can be performed across different applications using an overlay visual search application at the operating system level, which may include social media applications, browser applications, media content viewing applications, map applications, and / or viewfinder applications. Visual search can be implemented by the kernel of the operating system installed on the computing device. The operating system can obtain and / or process data received from one or more applications, and then send the data to one or more server computing systems to perform one or more artificial intelligence technologies for object classification, object recognition, optical character recognition, image captioning, image to text summary, text summary, query suggestion, and / or web search based on image and / or text processing. The overlay interface can generate display data, detect objects, and provide detection indicators in a single interface, and then send data for further processing based on user selection.

[0038] Visual search in an operating system may be included in a computing device to provide a user with a ready-made interface for accessing multiple artificial intelligence processing systems for object classification, image captioning, image-to-text summarization, response generation, web search, and / or one or more other artificial intelligence technologies. Smartphone and smart wearable device manufacturers may typically implement visual search in an operating system to leverage the utility of machine learning models and / or search engines across different applications. Visual search in an operating system may then be utilized to determine auxiliary applications associated with visual search data to provide suggestions for sending (or sharing) visual search data across applications on the device.

[0039] The systems and methods disclosed herein can utilize a visual search application in an operating system to provide an overlay visual search interface that can interface with multiple different applications on a computing device without incurring the computational costs and / or privacy issues of traditional visual search techniques. For example, a visual search interface can generate display data based on content currently and / or previously provided for display, and the display data can be processed to perform object detection, optical character recognition, segmentation, and / or other techniques on a computing device without incurring upload and / or download costs for interfacing with a server computing system. Additionally and / or alternatively, display data can be generated and temporarily stored during a visual search process, and then deleted to save storage space and free up resources for future visual search instances. Data generation and processing on the device can reduce the data sent to the server computing system and can improve privacy.

[0040] The visual search interface may include and / or utilize multiple on-device machine learning models. The on-device machine learning models may include object detection models, optical character recognition models, segmentation models, language models, visual language models, embedding models, input determination models (e.g., gesture recognition models), speech to text models, enhancement models, suggestion models, and / or other machine learning models. The on-device machine learning models may be used for on-device processing. Additionally and / or alternatively, a portion and / or all of the display data may be sent to a server computing system to perform additional processing tasks, which may include search result determination using a search engine, content generation using a generative model, and / or other processing tasks.

[0041] The visual search data generated and / or determined based on processing on the device and / or on the server can provide the user with additional information that was previously unavailable to the user (and / or traditionally more cumbersome and computationally expensive to obtain). The visual search data can be further processed to generate application suggestions to interact with and / or utilize the additional information. The application suggestions can be based on the data type associated with the determined visual search results and / or based on the topic, task, and / or entity associated with the visual search results. The application suggestions can be selected to navigate to one or more applications on the computing device and / or to send the visual search data to one or more applications on the computing device. The transmission can be performed at the operating system level and can be facilitated by one or more application programming interfaces. Model-generated content items can be generated based on the visual search data and / or based on the selected application.

[0042] It may be difficult for the user to apply the additional knowledge to other tasks, such as notifying others and / or taking action based on the additional information (e.g., generating a list, messaging others, composing a social media post, and / or interacting with displayed data). An operating system-level overlay visual search application can be used to perform visual searches across different applications, and visual search data can be sent to other applications on the device. The operating system can obtain and / or process data received from one or more applications, and then process the data to perform one or more artificial intelligence techniques to generate an output, which can then be processed to suggest that a second application send the visual search data to operatively use the visual search data.

[0043] Additionally and / or alternatively, the visual search interface in the operating system can be configured to obtain prompt input from the user to aggregate data from multiple different applications on a computing device and / or the web. The prompt can be processed to determine that one or more applications on the computing device are associated with the subject, task, and / or content type associated with the request for the prompt. Then an application call can be generated and executed based on the application determination. The application call can access one or more specific applications, search for related content items, and obtain the content items associated with the prompt. The obtained content items can be provided for display. Alternatively and / or additionally, a generative model can be used to process the content items to generate a structured output, which includes information from the content items in response to the prompt, and the information can be formatted in an easily understandable format, which can include graphics, stories, articles, images, poems, web pages, widgets, games, and / or other data formats.

[0044] The visual search interface may include an audio search interface, a multimodal search interface, and / or other data processing interfaces implemented at the operating system level to process data associated with a variety of different data types. Thus, a user may invoke an overlay interface to process image data, video data, audio data, text data, statistical data, potentially encoded data, and / or multimodal data across a variety of different applications.

[0045] The system and method of the present disclosure provide a variety of technical effects and benefits. As an example, the system and method can provide visual search across multiple different surfaces provided by a computing device. Specifically, the system and method disclosed herein can utilize the visual search interface in the operating system of the computing device to provide an overlay interface for data processing across multiple different surfaces (e.g., multiple different applications). The visual search interface can provide an overlay interface at the operating system level, which can obtain and process data from multiple different applications, which can include generating and processing screenshots of currently (and / or previously) displayed content. The visual search interface can include an on-device machine learning model that can perform object detection, optical character recognition, segmentation, query suggestions, action suggestions, and / or other data processing tasks on the device without sending data to a server computing system. The on-device machine learning model can provide privacy and provide data processing services even when network access is limited and / or unavailable. The visual search interface can be implemented in the kernel of the operating system. In addition and / or alternatively, the visual search kernel may include an interface (e.g., an application programming interface) for communicating with a server computing system to perform one or more additional data processing tasks (e.g., search engine processing, generative model media content generation, etc.).

[0046] Another technical advantage of the system and method of the present disclosure is that it is possible to utilize a visual search system including one or more communication interfaces to send data to and obtain data from multiple different applications. For example, the system and method disclosed herein may include an application programming interface and / or other communication interfaces for performing data packet generation and transmission and / or data calls. The system and method can process display data, generate one or more data processing outputs, and send data to auxiliary applications to perform one or more actions. In addition and / or alternatively, data can be obtained from one or more auxiliary applications to generate one or more additional information content items. The system at the operating system level can utilize a communication interface to provide seamless data use across different applications, which can be used to send data packets to other users and / or generate (and / or aggregate) information for users of computing devices. The implementation of the operating system level can be used to reduce the upload and download instances and costs of multiple different processing tasks. In addition and / or alternatively, temporary files and / or embedding can be generated and processed to reduce storage usage and increase privacy budgets.

[0047] Another example of technical effects and advantages relates to improved operational efficiency and improved computing system functionality. For example, the systems and methods disclosed herein can utilize a visual search interface in an operating system to reduce the input and operations required to perform specific data processing tasks across different applications. The reduction in input and operations can reduce the computing resources used to perform visual searches, feature detection, and / or queries and / or action suggestions based on processing display data. In addition and / or alternatively, the visual processing system at the operating system level can utilize a communication interface to send data and / or obtain data from auxiliary applications, which can reduce the user's manual navigation, storage, and / or selection.

[0048] Referring now to the drawings, example embodiments of the present disclosure will be discussed in further detail.

[0049] Figure 1 Depicted is a block diagram of an example visual search interface in operating system 10 according to an example embodiment of the present disclosure. Specifically, visual search interface 14 may be implemented at the operating system level to provide visual search interface 14 across applications and throughout the operating system of a computing device. Figure 1 A block diagram is depicted showing that the visual search interface 14 can utilize the computing resources of the computing device hardware 12 to provide visual search and other data processing techniques across multiple applications, which may include a first application 16, a second application 18, a third application, and / or an nth application 22.

[0050] For example, the visual search interface 14 may include a display capture component, one or more on-device machine learning models, and a transmission component that can utilize the hardware 12 of the computing device to perform display capture, object detection, optical character recognition, image segmentation, image enhancement, and / or data transmission. The visual search interface 14 may provide an overlay interface that can be accessed and utilized regardless of the application currently being used and / or displayed. The visual search interface 14 may be implemented in the kernel of an operating system.

[0051] In some implementations, the first application 16 may be a social media application, the second application 18 may be a web browser application, the third application 20 may be a media gallery application, and / or the nth application 22 may be a game application. The visual search interface 14 may be used to obtain and process data displayed in the social media application to identify locations, detect and search for objects for shopping tasks, and / or one or more other tasks. Additionally and / or alternatively, the visual search interface 14 may be used to process web information displayed in a viewing window of the second application 18 to generate image annotations, provide suggested searches, provide additional information, and / or suggest actions. The visual search interface 14 may be used to detect and search data viewed in a media gallery viewing window of the third application 20. In some implementations, the visual search interface 14 may be used to detect and search data associated with a game of the nth application 22 to obtain tutorials, determine progress, and / or find additional information about the game and / or game features.

[0052] The systems and methods disclosed herein may include an operating system-level visual search interface. Specifically, an operating system may include a kernel that utilizes machine learning models, interfaces, and / or components on multiple devices to provide a visual search interface for applications and virtual environments accessed across a computing device.

[0053] Figure 2 A block diagram of an example visual search interface system 200 is depicted in accordance with an example embodiment of the present disclosure. Specifically, a user computing device 210 may include a visual search interface 216 in an operating system 214 that may utilize resources of hardware 218 to provide visual search across multiple different applications 212. The visual search interface 216 may communicate with a server computing system 230 to obtain search results and / or perform one or more other processing tasks.

[0054] The user computing device 210 may include a visual display. The visual display may display a plurality of pixels. A plurality of pixels may be configured to display content associated with one or more applications 212. The visual display may include an organic light emitting diode display, a liquid crystal display, an active matrix organic light emitting diode display, and / or other types of displays. In some implementations, the user computing device 210 may include one or more additional output components. The one or more additional output components may include a tactile feedback component, one or more speakers, an auxiliary visual display (e.g., a second display on a projector and / or an adjacent side of the user computing device 210), and / or other output components.

[0055] The user computing device 210 may include an operating system 214 that includes a visual search interface 216. The visual search interface 216 may include an operating system-level visual search interface. The kernel may obtain display data associated with content currently provided for display by a visual display of the user computing device 210, and may send the display data and / or data associated with the display data (e.g., one or more machine learning model outputs) to the server computing system 230.

[0056] The visual search interface 216 may include one or more machine learning models stored on the user computing device 210. The one or more machine learning models may have been trained to detect features in the image data. The one or more on-device machine learning models may have been trained to process the image data to generate one or more machine learning outputs based on the detected features in the display data. The user computing device 210 may store multiple on-device machine learning models. The multiple on-device machine learning models may be used to perform object recognition, optical character recognition, input recognition, query suggestion, and / or image segmentation.

[0057] The visual search interface 216 may include an overlay interface. The overlay interface may obtain display data associated with content currently provided for display by the visual display in response to receiving user input. The visual search interface 216 may include a transmission component. The transmission component may send data describing the display data and one or more machine learning outputs to a server computing system.

[0058] The visual search interface 216 may include a display capture component. The display capture component may acquire display data associated with content currently provided for display by a visual display. The display capture component may generate a screenshot, which may then be processed by one or more machine learning models. Alternatively and / or in addition, a data packet may be generated based on the content provided for display.

[0059] Additionally and / or alternatively, the visual search interface 216 can include an object detection model. The object detection model can process the display data to determine that one or more objects are depicted. The object detection model can be trained to identify features that describe one or more objects associated with one or more object classes. In some implementations, the object detection model can process the screenshot (and / or a script describing the displayed content) to generate one or more bounding boxes associated with the locations of one or more detected objects in the screenshot.

[0060] The visual search interface 216 may include an optical character recognition model. The optical character recognition model may process the display data to determine features describing the text and may classify (e.g., transcribe) the text. The optical character recognition model may generate text data based on the image data. The optical character recognition model may detect scripts in the display data. The scripts may be transcribed and / or translated. Different machine learning models may be used for different content types, different languages, different locations, and / or other different context types.

[0061] In some implementations, the visual search interface 216 may include a segmentation model. The segmentation model may segment areas depicting one or more objects to generate image fragments. The segmentation model may have been trained to generate a segmentation mask describing the outline of the depicted object. The segmentation model may determine the outline pixels of the detected object, and then the outline pixels may be used to generate one or more indicators of the position and outline of the detected object. In some implementations, the segmentation model may be trained to parse the detected text to isolate the text from the display data. The segmentation model may determine to segment the text from other text in the display data based on semantics and / or entities. The segmentation mask may be used to provide alignment indicators and / or segmentation, which may assist in input determination.

[0062] In some implementations, the visual search interface 216 may include one or more classification models. The one or more classification models may process the display data to generate one or more classifications. The one or more classifications may include image classifications, object classifications, scene classifications, and / or one or more other classifications.

[0063] Additionally and / or alternatively, the visual search interface 216 may include a machine learning region of interest model. The machine learning region of interest model may have been trained to predict the region of the image for which the user is requesting a search. The machine learning region of interest model may have been trained to determine the significance of an object depicted in an image based on size, location, and / or other features in the image. In some implementations, the machine learning region of interest model may have been trained to update one or more predictions based on processing one or more user inputs. One or more user interface elements may be provided based on determining an object, text, and / or region of interest.

[0064] The visual search interface 216 may include a suggestion model. The suggestion model may process the display data to determine one or more query suggestions. Alternatively and / or in addition, the machine learning suggestion model may process the output of at least one of the object detection model or the segmentation model to generate one or more query suggestions. The one or more query suggestions may include a query to be sent to the server computing system 230. The query may include a multimodal query that includes a portion of the display data and a text snippet. The display data may be processed using one or more on-device machine learning models to generate the text snippet. The suggestion model may process the display data to determine one or more action suggestions. One or more action suggestions may be provided as selectable graphical user interface elements. In some implementations, one or more action suggestions may be selectable to navigate to a second application and perform one or more model-determined actions within the second application. The second application may be different from the first application associated with the display data. Query suggestions and / or action suggestions may be determined based on one or more detected objects and / or based on one or more entity classifications. The suggestion may be based on determining that the display data and / or visual search data are associated with a specific topic, a specific entity, and / or a specific task. An entity may be associated with an individual, a group, a company, a country, and / or a product.

[0065] Additionally and / or alternatively, the visual search interface 216 can include a server interface. The server interface can send data associated with the display data to a server computing system. The server interface can send a query to the server computing system to perform a search based on the query.

[0066] The user computing device 210 may include a wireless network component. The wireless network component may include a communication interface for communicating with one or more other computing devices. The user computing device 210 may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing device to perform operations.

[0067] Figure 2 An example visual search interface system 200 may be depicted that includes a user computing device 210 that communicates with one or more server computing systems 230 to perform one or more processing tasks across a plurality of different applications 212. The user computing device 210 may include hardware 218, an operating system 214, and a plurality of applications 212.

[0068] Hardware 218 may include the physical components of the user computing device 210, which may include a central processing unit, a graphics processing unit, random access memory, speakers, a sound card, computer data storage devices, input components, a physical display component (e.g., a visual display), and / or other hardware components.

[0069] The operating system 214 may include software for managing the computer resources of the hardware 218, and may be used to manage and operate multiple applications 212 and / or other computer software running on the user computing device 210. The operating system 214 may include a visual search interface 216, which may be used as an overlay visual search interface at the operating system level. The visual search interface 216 may include multiple machine learning models, heuristics, and / or deterministic functions to provide data processing services across multiple different applications 212. The data processing services may include image classification, object detection, object classification, image segmentation, data enhancement, data annotation, visual search, optical character recognition, input recognition, query prediction, action prediction, and / or other data processing tasks.

[0070] The visual search interface 216 can obtain display data associated with content currently provided for display, process the display data to generate one or more processing outputs, generate one or more graphical user interface elements that provide additional information to the user, send one or more processing outputs and / or display data to the server computing system 230, receive data from the server computing system 230, and provide data for display. The visual search interface 216 may include a display capture component that can generate screenshots, parse the displayed data, and / or generate a data packet describing the displayed content. The display data can be processed using one or more machine learning models to perform object detection and / or optical character recognition. A mask can be generated for each detected object, which can be used to indicate to the user the objects identified in the displayed content. In addition and / or alternatively, the identified text and / or objects can be processed to determine entities associated with the content, which can then be annotated in the display interface. The display data, detected objects, and / or detected text can be processed to provide one or more suggestions (e.g., one or more query suggestions and / or one or more action suggestions).

[0071] The visual search interface 216 may include, provide, and / or generate a plurality of different user interface elements that may provide additional information, options, and / or indicators to the user. The user interface elements may include indicators of detected objects and / or text that may be selected to perform one or more additional actions that may include sending the selected data for processing by the search engine and / or the generative model. In addition, the user interface elements may provide the user with the option of gesture selection. In some implementations, selectable suggestions may be provided that may be selected to perform a search (e.g., a search with a suggested query) and / or one or more other actions (e.g., sending an email, opening a map application, color correction, autofocus, and / or data enhancement).

[0072] The visual search interface 216 can obtain data from multiple different applications 212 and can send data from multiple different applications to provide an overlay interface for determining and providing additional information to the user and providing compiled and sendable data.

[0073] The visual search interface 216 may include an input understanding model. The input understanding model may be trained to determine the relevance and / or significance of multiple different features in the display data. The relevance and / or significance may be determined based on object and / or character size, position, and / or cohesion with other objects and / or characters in the display data. Additionally and / or alternatively, the input understanding model may be trained and / or adjusted based on previous user interactions. For example, the input understanding model may be adjusted based on previously viewed data to adjust the significance and / or relevance based on recently viewed content. Additionally and / or alternatively, the input understanding model may be trained on previous inputs and / or gestures to understand the deviation from the true value when receiving input from the user. The training may configure the model to understand which element is being selected and / or when a gesture is received. The input determination model may be personalized for a specific user based on previous user interactions. Alternatively and / or additionally, the input determination model may be unified for multiple users. The input determination model may be trained to determine whether a gesture is associated with a call to the visual search interface or an interaction with a displayed application. Additionally and / or alternatively, the input determination model can be trained to determine when the input is a gesture for selecting a particular object for search and / or when another input is received. In some implementations, the input understanding model can generate a polygon associated with the user input and determine an overlap between the polygon and the detected object. Objects that overlap with the polygon can be determined to be selected. The input understanding model can utilize heuristics, deterministic functions, and / or learned weights.

[0074] The visual search interface 216 can communicate with the server computing system 230 via a network to provide a plurality of additional processing services. The server computing system 230 can include one or more generation models 232, one or more object detection models 234, one or more segmentation models 236, one or more classification models 238, one or more embedding models 240, one or more semantic analysis models 242, and / or one or more search engines 244.

[0075] One or more generation models 232 can be used to process display data and / or one or more processing outputs to generate natural language output (e.g., natural language output including additional information about the display data and / or entities associated with the data depicted in the displayed content), generate images, and / or other model-generated media content items. For example, one or more web resources can be accessed and processed to generate summaries of specific topics. One or more object detection models 234 can be used to perform object detection in display data. One or more segmentation models 236 can be used to segment objects and / or text fragments from the displayed content. One or more classification models 238 can be used to perform object classification, image classification, entity classification, format classification, sentiment classification, and / or other classification tasks. One or more embedding models 240 can be used to embed part and / or all of the display data. Embedding can then be used to search for similar objects and / or text, classification, grouping, and / or compression. Semantic analysis model 242 can be used to process display data to generate semantic outputs describing understanding of display data and / or one or more other semantic outputs regarding topic understanding, scene understanding, focus, pattern recognition, application understanding.

[0076] One or more search engines 244 can process display data, parts of display data, and / or one or more machine learning model outputs to determine one or more search results. One or more search results may include web pages, images, text, videos, and / or other data. Search results may be determined based on feature mapping, feature matching, embedded search, metadata search, tag search, clustering, and / or other search techniques. Search results may be determined based on query intent classification, search result classification, and / or entity classification. The output of the model and / or search results may be sent back to the user computing device to be provided to the user through one or more user interface elements generated and provided by the visual search interface 216.

[0077] Figure 3 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Figure 3The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The various steps of method 300 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0078] At 302, the computing system can obtain input data. The input data can describe a request to open an overlay visual search interface. The input data can describe a user selection and / or a user gesture. For example, a user can provide an oblique pull gesture. The overlay visual search interface can be opened across multiple surfaces of a computing device, which can include multiple different applications. The overlay visual search interface can be provided regardless of the application and / or data provided for display.

[0079] At 304, the computing system may generate display data. The display data may describe content currently provided for display by the user computing device. In some implementations, generating the display data may include generating a screenshot. The display data may describe the screenshot and / or a data packet associated with rendering the content for display.

[0080] At 306, the computing system may process the display data using one or more on-device machine learning models to generate one or more machine learning model outputs. The one or more on-device machine learning models may include an object detection model, an optical character recognition model, a segmentation model, a region of interest model, a suggestion model, and / or one or more classification models. The one or more machine learning model outputs may include one or more bounding boxes, one or more text strings, one or more segmentation masks, one or more region of interest values ​​and / or annotations, one or more suggestions (e.g., one or more query suggestions and / or one or more action suggestions), and / or one or more classifications (e.g., one or more object classifications, one or more image classifications, one or more entity classifications, and / or one or more other classifications).

[0081] In some implementations, the display data may be processed using an object detection model to generate a plurality of bounding boxes associated with a plurality of detected objects. The display data and / or the plurality of bounding boxes may then be processed using a segmentation model to generate a plurality of segmentation masks associated with the contours of the plurality of detected objects. The plurality of segmentation masks may be used to generate a user interface indicator that indicates the detected objects and the outlines of the detected objects. In some implementations, the display data may be processed using one or more classification models to generate one or more object classifications for objects depicted in the displayed content. Additionally and / or alternatively, the display data may be processed using an optical character recognition model to generate text data describing the text in the displayed content. The display data and / or text data may be processed to determine one or more entities associated with the text and / or objects in the displayed content. One or more user interface elements may be generated and provided to provide an indication of the determined entity to the user.

[0082] In some implementations, display data, segmented image data, bounding boxes, text data, classification data, and / or metadata may be processed using a suggestion model to generate one or more suggestions. The one or more suggestions may include one or more query suggestions and / or one or more action suggestions. The one or more query suggestions may describe queries suggested based on detected features in the display data, and may include a multimodal query that includes at least a portion of the display data and a generated text string. The one or more action suggestions may be associated with a suggested processing task, which may include sending data to another application, platform, and / or computing system.

[0083] At 308, the computing system can generate one or more selectable user interface elements based on the one or more machine learning model outputs. The selectable user interface elements can include detected object annotations, preliminary classifications, suggested queries, and / or suggested actions.

[0084] At 310, the computing system may send data associated with the display data to the server computing system. The data associated with the display data may include at least a portion of the display data, a segmented portion of the displayed content, a display data embedding, one or more bounding boxes, a multimodal query including at least a portion of the display data and a generated text query, and / or one or more machine learning model outputs. The server computing system may include one or more search engines, one or more generative models (e.g., a large language model, an image-to-text model, a text-to-image model, a visual language model, and / or other generative models), one or more classification models, and / or one or more enhancement models. The server computing system may process the data associated with the display data to determine one or more search results, generate one or more model-generated media content items, and / or one or more server outputs.

[0085] In some implementations, data associated with the display data can be associated with user input selecting a particular selectable user interface element. For example, a portion of the display data can be segmented and sent to the server computing system based on the user input. Alternatively and / or in addition, the user input can select a query suggestion, and a query associated with the suggestion can be sent to the server computing system.

[0086] At 312, in response to sending data associated with the display data to the server computing system, the computing system may receive additional information associated with the display data from the server computing system. The additional information may include one or more search results, one or more model-generated outputs, augmented reality renderings, updated suggestions, object annotations, and / or other information. The additional information may then be provided to the user for display. In some implementations, the additional information may be provided for display with at least a portion of the displayed content.

[0087] FIG. 4A to FIG. 4D Depicted is an illustration of an example visual search interface according to an example embodiment of the present disclosure. Specifically, Figure 4A An illustration depicting a visual search interface being opened and used to perform display data processing and annotation.

[0088] For example, a user may be viewing a content item in a first application 402, which may include a first application interface. The first application 402 may include a social media application that displays one or more content items. The content items may be social media posts published by the user and / or entities that the user follows on a particular social media platform.

[0089] The computing device may provide content items in the first application 402 and may receive input 404 to open a visual search interface. Input 404 may include a gesture input (e.g., a swipe from a corner of the interface to the middle). The visual search interface may be an overlay interface implemented at the operating system level of the computing device.

[0090] Display data describing content items in the first application 402 can be generated and processed based on the visual search interface being opened. An input screen 406 can then be provided for display to indicate to the user that a visual search interface is being provided. The input screen 406 may include a filter (e.g., a pixy dust filter) above the display data, input instructions 408 for how to provide input, query suggestions 410 based on preliminary processing of the display data, and / or a query input box 412.

[0091] The filter may include coloring of the displayed content. The input instructions 408 may include text, icons, and / or animations that instruct the user how to select portions of the displayed content for searching. For example, a user interface element may indicate that an object and / or display area may be selected using a circle gesture. The query suggestions 410 may include query suggestions determined based on processing all display data (e.g., an entire screenshot of the displayed content). Alternatively and / or in addition, the query suggestions 410 may be determined based on on-device object detection, on-device object segmentation, on-device optical character recognition, on-device classification, context data processing, and / or other processing techniques. The query suggestions 410 may be provided on a scrollable carousel and may be selectable to perform a search. In some implementations, a language model may be used to generate natural language query suggestions. The query input box 412 may be configured to receive text input, image input, audio input, video input, and / or other input, and then process to perform a search locally and / or on the web.

[0092] Figure 4B An illustration of object selection and processing in a visual search interface is depicted. For example, a circular gesture 414 can be received that selects a particular object depicted in displayed content. The visual search interface can process an area associated with the gesture to determine the object and / or set of objects selected by the circular gesture 414. The visual search interface can process the area using one or more on-device machine learning models (e.g., an object detection model and a segmentation model) to identify the shape of the object. A graphical indicator 416 of the object and its corresponding shape can be provided for display.

[0093] Pixels describing an object can be segmented and searched. In some implementations, a generative model (e.g., a visual language model and / or a large language model) can be used to process the image fragment to generate a model-generated response 418 to the query. The model-generated response 418 can include a natural language response that summarizes one or more web resources determined to be associated with the segmented object. In addition and / or alternatively, the segmented image can be processed to determine one or more visual search results 420. One or more visual search results 420 can be determined based on classification label matching, embedding search, feature matching, clustering, and / or image matching. One or more visual search results 420 can include product listings, articles, and / or other web resources. Visual matches 422 can be provided for one or more visual search results 420, which include images depicting objects that match the segmented object. The search result interface can include a variety of different types of search results and can be displayed in a variety of different formats in a variety of different panels.

[0094] In some implementations, subsequent query suggestions can be determined and provided for display in a suggestion panel adjacent to the query input box 412 .

[0095] Figure 4C An example diagram of an example subsequent search in a visual search interface is depicted. For example, a subsequent query suggestion 424 can be selected and processed. When determining 428 and providing a subsequent visual search result for display, a processed subsequent query 426 can be provided for display. The subsequent visual search result can include a model-generated response 432 to the processed subsequent query 426. The model-generated response 432 can include a natural language response generated using a generative model (e.g., a large language model). The model-generated response 432 can be generated based on and / or provided by one or more subsequent visual search results 434 in response to the processed subsequent query 426. In some implementations, additional subsequent search results 436 can be determined and provided for display.

[0096] Query suggestions 438 may be updated again to further reflect subsequent predictions. Query suggestions 438 may be provided via a subsequent input box 440 .

[0097] Figure 4CAn illustration of text input retrieval and processing using a visual search interface is depicted. For example, a graphical keyboard interface 444 can be utilized to receive subsequent text input 442 (e.g., "Are there any other shapes available?"). The input can be obtained and provided as an updated query 446, which can include the input text and thumbnails depicting image segments. The updated query 446 can be processed to determine a plurality of updated search results 448. The plurality of updated search results 448 can be formatted by processing the web resource search results using a generative model to include natural language sentences, a uniform structure and style, and / or user interface elements determined by the model.

[0098] In some implementations, the visual search interface may include action suggestions and one or more query suggestions in the suggestion panel. Action suggestions may be determined and provided based on the multiple updated search results 448. The action suggestions may include viewing one or more products in the user environment using an augmented reality experience. The one or more products may be associated with the search results. The action suggestions may include interfacing with and / or navigating to another application on the computing device.

[0099] FIG. 5A to FIG. 5D Depicted is a diagram of an example data transfer interface according to an example embodiment of the present disclosure. Figure 5A Depicts a diagram of display data generation and processing using an overlay visual search interface.

[0100] For example, a user may be viewing a web page 502 in a browser application. The user may provide input to utilize a visual search interface. The visual search interface may then obtain gesture input 504, which may select a portion of a file in the web page 502. The input screen may be provided with a plurality of preliminary query suggestions 506, which may be determined by performing optical character recognition on the web page, parsing the text, and predicting candidate queries that the user may request. Additionally and / or alternatively, the input screen may include a query input box 508 for receiving text and / or other input from the user to be processed with the displayed data.

[0101] The selected text 510 may be indicated by one or more user interface elements (e.g., highlighting, selective filters, etc.). The selected text 510 may be processed to determine one or more search results 514. The selected text 510 and / or the one or more search results 514 may be processed using a generative model to generate a model-generated response 512 to the query. The model-generated response 512 may be a summary of at least a portion of the one or more search results 514. The search results interface may include a plurality of different search result types (e.g., model-generated responses, web search results, map search results, etc.). The search results interface may be provided with a plurality of updated query suggestions 516.

[0102] A search result may be selected to view a web page 518 associated with the search result. A web page 518 may be provided to display multiple application suggestions 520. Multiple application suggestions 520 may be determined based on processing the web page 518, the selected text 510, and / or the content of the search result interface to determine a predicted action associated with the processed data. For example, a topic, entity, and / or task may be determined to be associated with the processed data. One or more actions may be determined to be associated with a topic, entity, and / or task. An application associated with an action may be determined to be located on the device. Multiple application suggestions 520 may then be determined and application icons and action suggestions may be provided to the multiple application suggestions.

[0103] Figure 5B An illustration of an example application data push using a visual search interface is depicted. For example, a text suggestion may be selected to be created. The created text suggestion may be a specific application suggestion from a plurality of application suggestions 520. The text messaging application of the computing device may then be opened. The text writing interface may include a sent and received message viewing panel 522. An overlay interface 524 may be provided to assist in writing messages. Overlay interface 524 may depict model-generated prompts (and / or user input prompts), which may be processed to generate model-generated messages 526. Model-generated prompts may be generated based on visual search data and / or selections of specific application suggestions. Model-generated messages 526 may be sent via web page 518 along with data packet 528.

[0104] The “Insert” user interface element 530 may be selected to insert the model-generated message 526 and data packet 528 into a text messaging application (e.g., into an input text box 532 of a messaging application), which may be supplemented by input to a graphical keyboard interface 534. The model-generated message 526 and data packet 528 may then be sent as text 536 to a second user.

[0105] Figure 5CAn illustration of an example visual search of an order confirmation page 538 is depicted. The order confirmation page 538 may be processed to determine action suggestions and one or more query suggestions 540. A specific query suggestion (e.g., "What furniture matches this?") in the one or more query suggestions 540 may be selected. The selected query suggestion and a screenshot of the order confirmation page 538 may be utilized as a multimodal query 542. The multimodal query 542 may be processed using a search engine and / or one or more machine learning models. A plurality of search results 544 may be determined based on the multimodal query 542. In some implementations, the plurality of search results 544 may be formatted and / or enhanced based on processing web resource data using a generative model. The plurality of search results 544 may be associated with furniture that matches the color, style, and / or aesthetic of a recently ordered lamp. In some implementations, the plurality of search results 544 may be formatted so that only one piece of furniture from a particular category may be provided for each category. The search result products may be filtered and / or determined based on user location, user budget, user preferences, and / or other contextual data.

[0106] The plurality of search results 544 may be provided for display using a suggestion carousel 546 that includes an application suggestion and one or more updated query suggestions. The application suggestion and the one or more updated query suggestions may be determined based on the plurality of search results 544. The application suggestion may be selectable to navigate to an augmented reality application that may be used to render one or more search result products into a user environment.

[0107] Figure 5D An illustration of an example data transfer to an application on a computing device is depicted. For example, a subset 548 of multiple search results 544 can be selected. Based on the selected subset 548, a suggestion panel 550 can be updated to include multiple different application suggestions. An email application suggestion can be selected, which can cause an email application 552 to be opened through an overlay message composition interface. Prompts can be generated based on the selected subset 548 and / or based on one or more user inputs. The prompts can be processed to generate a model-generated message 554, which can then be added to a draft email message 556 in the email application 552. The draft email message 556 can be sent together with a data packet 558 describing the selected subset 548. The data packet 558 can include a model-generated content item that includes details associated with the selected subset 548.

[0108] FIG. 6A to FIG. 6E Depicted is a diagram of an example data call interface according to an example embodiment of the present disclosure. Fig. 6A Depicted is a diagram of cue generation and processing.

[0109] For example, a data call interface 602 may be opened and provided for display. The data call interface 602 may be part of a visual search interface implemented by an operating system of a computing device. A graphical keyboard interface 606 may be used to obtain input from a user to generate (or write) a prompt 604. The prompt 604 may include a request for information from one or more specific applications. The prompt may be processed to determine the specific application. The specific application may then be accessed 608 and searched for based on the application call generated according to the prompt 604. A status response may be provided when data is obtained from one or more specific applications.

[0110] A plurality of content items may be obtained from one or more specific applications based on the prompt 604. The plurality of content items may be processed using a machine learning model to generate a structured output 610 that provides information from the plurality of content items in an organized format. A free-form input box 612 may be provided to obtain subsequent input to enhance, supplement, and / or perform an action based on the structured output 610 (e.g., perform a search based on the structured output 610).

[0111] Figure 6B An illustration of a structured output enhancement is depicted. A user may be budget-conscious, and the structured output 610 may indicate that the wish list generated by the current model is over budget. The user may select a specific item on the wish list to replace to (a) meet the budget and / or (b) replace the item with a different product based on one or more user preferences. The system may process the selection of the "suggest couch" option to determine products of a specific product type that match the user's style, aesthetic, price range, and / or other preferences. Multiple product alternatives may then be provided for display in a carousel interface 614 for the user to view and select to enhance the structured output. Multiple product alternatives may be obtained from one or more applications and / or from the web. The user may select a specific alternative, and the specific alternative may be processed to generate an enhanced structured output 616 that updates at least a portion of the structured output 610.

[0112] Figure 6C A diagram of subsequent prompt generation and processing is depicted. A user may provide voice command input 618, which may be transcribed to generate a second prompt 620. The second prompt 620 and the enhanced structured output 616 may be processed to generate a graphical representation 622. The graphical representation 622 may include a map graphic with one or more location indicators bearing one or more products from the enhanced structured output 616. Additionally and / or alternatively, an image 624 and / or other content items may be provided for display with the graphical representation 622.

[0113] Fig.6DAn illustration of calendar invitation generation is depicted. For example, a user may provide a third prompt 626, which may be processed 628 by the system to generate a calendar invitation 634. The calendar invitation 634 may include information associated with the third prompt 626 and the graphical representation 622. The calendar invitation 634 may be displayed with a model-generated natural language response 632 to the third prompt 626. The calendar invitation 634 may include a title, a graphical representation 622 with a suggested route, a date, one or more locations, and / or a proposed itinerary 636.

[0114] Fig. 6E An illustration of a calendar invitation transmission to a calendar application is depicted. The calendar invitation 634 may include an option to add an event 638 to the calendar. The calendar invitation 634 may then be added and then viewed 640 in the calendar application on the device.

[0115] FIG. 7A to FIG. 7B A diagram showing a data processing interface on an example device according to an example embodiment of the present disclosure is depicted. For example, Fig. 7A An illustration of an overlay interface opened and used to process a displayed document 702 is depicted. The displayed document 702 may include a product manual, a textbook, and / or another content item. A user may provide input to open the overlay interface, which may open an input screen 704. The input screen 704 may include filters for displayed data, one or more action suggestions, one or more query suggestions, and a query input box.

[0116] The user can select an action suggestion to open an action interface (e.g., translation interface 706). The action interface can be interacted with to perform one or more processing techniques, which can include translation, object detection, optical character recognition, data enhancement, object segmentation, classification, annotation, parsing, and / or other data processing techniques.

[0117] For the translation interface 706, a language option can be provided. In some implementations, the language can be automatically determined based on determining the language of the displayed document 702 and determining the native language of the user (e.g., based on user preferences and / or settings). The translation interface 706 can include a text-to-speech option, a copy option, one or more query suggestions, and / or a query input box.

[0118] The translation may be performed based on the user input to generate a translated document in the desired language 708. Alternatively and / or in addition, other document enhancements (e.g., formatting adjustments) may be performed. The translation may be performed using one or more translation models, which may be stored on the device.

[0119] Figure 7BAn illustration of query suggestions for a selected (e.g., selected at 710) translation document is depicted. The query suggestions can be processed using the translation document 708 to determine visual search results 712 that are responsive to the multimodal query. The visual search results 712 can include a model-generated response that summarizes one or more web resources that are responsive to the query. In some implementations, the visual search results 712 can include images, articles, videos, and / or other data. Updated query suggestions including predictions of subsequent queries can be provided for the visual search results 712.

[0120] A visual search interface in an operating system of a computing device may be utilized to perform application suggestions based on visual search. For example, visual search data may be processed to predict actions that may be of interest to a user based on the visual search data. Action predictions may be based on user-specific data, entity-action correlations, global historical data, and / or other data. Action predictions may be used to determine one or more applications on a computing device that may perform an action. The visual search interface in the operating system may then provide options for navigating to and / or interfacing with an application to perform the suggested action.

[0121] Figure 8 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Figure 8 The steps performed in a particular order are depicted for purposes of illustration and discussion, but the method of the present disclosure is not limited to the order or arrangement specifically shown. Various steps of method 800 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0122] At 802, the computing system may acquire display data. The display data may describe content currently presented for display in a first application on a user computing device. Acquiring the display data may include generating a screenshot. The screenshot may describe a plurality of pixels provided for display. In some implementations, the display data may be generated using a visual search application in an operating system. The visual search application may include an overlay application that is compatible with generating and processing screenshots across multiple different applications on the user computing device. In some implementations, the display data may be acquired and processed based on user input requesting a visual search overlay application. The display data may describe a plurality of pixels previously displayed before receiving a visual search interface request. The display data may depict a first application interface with one or more content items (e.g., a social media interface with one or more social media posts in a social media application, an email interface with one or more messages in an email application, a news application interface with one or more news articles in a news application, etc.). The display data may include metadata associated with the context (e.g., time, application currently provided for display, display duration, and / or historical data). In some implementations, the display data may include one or more images, text data, audio data, one or more embeddings, potential representation data, and / or encrypted data.

[0123] At 804, the computing system may process at least a portion of the display data to generate visual search data. The visual search data may include one or more visual search results. The one or more visual search results may be associated with detected features in the display data. The display data may be processed using one or more machine learning models to generate one or more outputs associated with the detected features. For example, the display data (e.g., one or more images of the display data) may be processed using an object detection model to generate one or more bounding boxes associated with the position of the detected object in the captured display. The one or more bounding boxes and the display data may be processed using a segmentation model to generate a mask for each detected object, thereby segmenting the object from one or more images of the display data and / or generating a detailed outline of the object indicating the object boundary. In some implementations, the segmented object may be processed using a search engine and / or one or more additional machine learning models to generate visual search data. The search engine may determine one or more visual search results based on detected features in the image fragment, embedded search (e.g., embedded neighbor determination), one or more object classifications, one or more image classifications, application classifications, and / or multimodal search (e.g., search based on image fragments and text data (e.g., input text, metadata, text tags, etc.)). In some implementations, the display data may be processed using an optical character recognition model to identify text in one or more images of the display data. The text may be utilized to condition the search.

[0124] In some implementations, one or more visual search results may include reverse image search results. One or more visual search results may be determined based on the detected features. One or more visual search results may include images similar to one or more images of the display data, may include objects similar to detected objects in the display data, may include interfaces similar to detected user interface features in the display data, determined subtitle data, determined classifications, and / or other search result data. The visual search data may include the output of one or more classification models, one or more enhancement models, and / or one or more generated visual language models. For example, the display data may be processed using a machine learning visual language model to generate predicted subtitles for the display data.

[0125] In some implementations, processing at least a portion of the display data to generate visual search data may include: processing the display data using one or more on-device machine learning models to generate segmented portions of the display data. The segmented portions of the display data may include data describing a set of features of content presented for display. The computing system may send the segmented portions of the display data to the server computing system and receive the visual search data from the server computing system. The visual search data may include one or more search results. The one or more search results may be associated with detected features in the segmented portions of the display data. The visual search data may include one or more search results and a knowledge panel generated by the model. In some implementations, the knowledge panel generated by the model may include a summary of a topic associated with the segmented portion of the display data. The summary may be generated by processing the web resource data using a language model. For example, one or more visual search results may be determined based on the segmented portions. Content items (e.g., articles, images, videos, audio, blogs, and / or social media posts) associated with one or more visual search results may be processed using a generated language model (e.g., an autoregressive language model that may include a large language model) to generate a summary in a natural language format. The one or more on-device machine learning models may include an object detection model and a segmentation model stored on the user computing device.

[0126] Alternatively and / or in addition, processing portions of the display data to generate visual search data may include processing the display data using an object detection model to determine a segmented portion of the display data depicting one or more objects and generating the display data. The segmented portion may include one or more objects. Processing portions of the display data to generate visual search data may include processing the segmented portion of the display data to generate visual search data. The object detection model may generate one or more bounding boxes. The one or more bounding boxes may describe the location of one or more objects within the content currently presented for display. In some implementations, generating the segmented portion of the display data may include processing the display data and the one or more bounding boxes using a segmentation model to generate the segmented portion of the display data. The object detection model and the segmentation model may be machine learning models. The object detection model and the segmentation model may be stored on a user computing device. In some implementations, processing portions of the display data to generate visual search data may be performed on the device.

[0127] At 806, the computing system may determine that a particular second application on the computing device is associated with the visual search data. For example, the computing system may process the visual search data using a machine learning suggestion model to determine that the second application is associated with one or more visual search results. The second application may be different from the first application that depicts the content processed to generate the display data. The first application and the second application may be different from the overlay application that performs the display data generation and processing. The machine learning suggestion model may be trained to identify topics and / or entities associated with the visual search data. The identified entities and / or topics may then be used to determine actions associated with a given entity and / or topic. The action may include delivering a message to another user, opening a map application, purchasing a product, viewing an augmented reality and / or virtual reality asset, adding to an annotation, adding to a gallery database, and / or other actions. Based on the determined action, the application on the device may be determined to be associated with the visual search data based on the action being able to be performed by the application. The machine learning suggestion model may be trained to process the visual search data, determine a topic and / or entity classification, and then determine whether the classification is associated with one or more applications on the device. The machine learning suggestion model may be trained to generate natural language suggestions and / or multimodal suggestions (e.g., icons and text) indicating applications and proposed actions. The application suggestion may include a data package and / or a prompt, which, if selected, may be sent to the second application.

[0128] In some implementations, the computing system may determine that multiple candidate second applications are associated with the visual search data, and may provide multiple application suggestions for display in a suggestion panel. The suggestion panel may include multiple application suggestions and one or more query suggestions. The one or more query suggestions may be determined based on the display data and / or the one or more visual search results.

[0129] At 808, the computing system can provide application suggestions associated with the particular second application based on the visual search data. The application suggestions can be provided with an icon indicator of the application and action suggestions. The application suggestions can be provided to be displayed with one or more visual search results.

[0130] In some implementations, the computing system may receive a selection of an application suggestion and send data to a second application based on the selection. For example, the computing system may obtain a selection of an application suggestion to send at least a portion of the visual search data to a specific second application, and generate a model-generated content item (e.g., a visual search summary, a content item summary, an image caption, an enhanced image, a generated table, etc.) based on the selection of the application suggestion. The model-generated content item may be generated using a generative model (e.g., a generative language model, a generative image model, etc.) based on a portion of the visual search data. The computing system may provide the model-generated content item to a specific second application. In some implementations, the generative model may include a generative language model that generates a natural language output based on features of the processed input data. The first application and the specific second application associated with the content provided for display when generating display data may be different. The specific second application may include a messaging application, and the model-generated content item may include a model-written message to a second user. The model-generated content item may be generated using a generative language model. Alternatively and / or in addition, the model-generated content item may include a model-generated list of visual search results selected by multiple users. The model-generated list may be generated using a generative language model that organizes the plurality of user-selected visual search results and generates a natural language output for each of the plurality of user-selected visual search results. Providing the model-generated content item to a particular second application may include sending the model-generated content item to the second application via an application programming interface.

[0131] In some implementations, obtaining a selection of application suggestions to send at least a portion of the visual search data to a second application may include: determining multiple application transfer actions associated with the visual search data. Multiple application transfer actions may be associated with multiple candidate second applications to send data associated with the visual search data. Obtaining a selection of application suggestions to send at least a portion of the visual search data to a second application may include: providing multiple selectable options based on the multiple application transfer actions. Multiple selectable options may be associated with multiple application transfer actions. Multiple selectable options may include application suggestions. Multiple application transfer actions may include a specific second application. Additionally and / or alternatively, obtaining a selection of application suggestions to send at least a portion of the visual search data to a second application may include receiving a selection of application suggestions. The application suggestion may be associated with a specific second application.

[0132] In some implementations, generating a model-generated content item based on a selection of an option may include: processing visual search data and data associated with a particular second application to determine a suggested prompt, receiving input selecting the suggested prompt, and processing the suggested prompt and the visual search data using a generative model to generate a model-generated content item. The model-generated content item may then be sent to the second application.

[0133] Additionally and / or alternatively, the computing system can determine a plurality of application suggestions. For example, the computing system can process the visual search data to determine a plurality of candidate second applications associated with one or more search results, obtain a selection for a particular application suggestion to send at least a portion of the visual search data to a particular second application from the plurality of candidate second applications, obtain a model-generated content item based on the selection for the particular application suggestion, and provide the model-generated content item to the particular second application. The model-generated content item may have been generated using a generative model based on a portion of the visual search data.

[0134] Fig. 9 A block diagram of an example application suggestion system 900 is depicted, in accordance with an example embodiment of the present disclosure. Specifically, the application suggestion system 900 may process image data 902 to determine and / or generate visual search data 906 , which may then be processed to determine one or more application suggestions 914 .

[0135] For example, image data 902 may be obtained. Image data 902 may describe content previously provided for display by a computing device. Image data 902 may include one or more images and may describe one or more objects. Image data 902 may describe a previously displayed application, which may include an application interface and one or more content items.

[0136] The image data may be processed to perform a visual search 904, thereby generating visual search data 906. The visual search 904 may include object detection, optical character recognition, image segmentation, object classification, generative model processing, and / or search engine processing. The visual search 904 may include processing the image data 902 using text data 908 and / or context data 910 to determine one or more visual search results, which may be associated with one or more web resources. The text data 908 may include user input text, predicted text, selected text suggestions, extracted text, and / or text tags. The context data 910 may include metadata. In some implementations, the context data 910 may be associated with time, location, search history, browsing history, application history, user profile data, personalized models, and / or other contexts.

[0137] Visual search data 906 may describe one or more visual search results associated with image data 902. The one or more visual search results may include images, text, audio, video, and / or other search result data. Visual search data 906 may include one or more object classifications and / or one or more image classifications. Visual search data 906 may include a model-generated response that may be generated by processing one or more web resources associated with the one or more visual search results to generate a natural language response to the image query.

[0138] The visual search data 906 may be processed using a suggestion model 912 to determine one or more application suggestions 914 and / or one or more query suggestions 916. The one or more query suggestions 916 may include suggested follow-up queries determined based on the content of the one or more visual search results and / or based on topics and / or subtopics associated with the image data 902 and / or the one or more search results. The one or more application suggestions 914 may include applications on a computing device determined to be associated with the image data 902 based on the visual search data 906. For example, the visual search data 906 may be processed to determine one or more topics, entities, and / or tasks associated with the one or more visual search results. Based on the one or more determined topics, entities, and / or tasks, an application associated with the one or more visual search results may be determined.

[0139] In some implementations, one or more of the application suggestions 914 can be selected to send at least a portion of the visual search data 906 to a second application. Additionally and / or alternatively, the visual search data 906 and / or one or more application suggestions 914 can be processed using a generative model to generate one or more model-generated content items 918 to send to the second application. The model-generated content items 918 are generated by processing the application suggestions 914 and / or the visual search data 906 using a prompt generation model 920 to generate a prompt, and then processing the prompt using the generative model to generate the model-generated content items 918. The model-generated content items 918 can describe a summary and / or representation of at least a portion of the visual search data 906, and can be configured and / or formatted based on a particular second application.

[0140] The visual search interface in the operating system can be used to interface with one or more applications on the computing device to aggregate data for the user. The aggregated data can be processed using one or more machine learning models to generate an output that organizes the data in a format that conveys information in an easily understandable manner.

[0141] Fig.10 Depicted is a flowchart of an example method performed in accordance with an example embodiment of the present disclosure. Fig.10 For the purpose of illustration and discussion, the steps performed in a specific order are depicted, but the method of the present disclosure is not limited to the specific order or arrangement shown. The various steps of method 1000 can be omitted, rearranged, combined or adjusted in various ways without departing from the scope of the present disclosure.

[0142] At 1002, a computing system may obtain a prompt. The prompt may be obtained by a computing device. The prompt may describe a request for information from one or more applications on a computing system and / or a specific computing device. The prompt may be obtained through an overlay interface. The overlay interface may be provided by an operating system. In some implementations, the prompt may include a multimodal prompt. The multimodal prompt may include text data and image data. The prompt may include an application call indicating a specific application to be accessed and from which data is to be obtained for information retrieval. Alternatively and / or in addition, one or more applications to be accessed and searched may be determined by identifying a subject, task, and / or entity associated with the request. In some implementations, the prompt may include text data, image data, audio data, potentially encoded data, and / or context data. The prompt may include image data and text data describing a task performed using an image (e.g., searching for images with a chair in a specific social media application, a specific messaging application, and / or a specific image storage application). The prompt may describe application calls to multiple applications to aggregate information associated with a specific topic (e.g., room makeover, clothing wish list, travel itinerary, story ideas, etc.).

[0143] At 1004, the computing system may process the prompt to determine a plurality of content items associated with one or more applications. The plurality of content items may be determined by accessing data associated with one or more applications on the computing device. In some implementations, the plurality of content items may include one or more multimodal content items (e.g., an email with text and one or more images, a product list with images and text, and / or a video list with videos and subtitles). The plurality of content items may be obtained from a plurality of applications on the computing device. One or more first content items may be obtained from a first application, and one or more second content items may be obtained from a second application. The first application and the second application may be different from the application from which the prompt is obtained. The one or more applications may include one or more messaging applications. In some implementations, the plurality of content items may include a plurality of messages determined to be associated with the prompt. One or more applications may be determined based on processing the prompt using an application interface model, which may determine one or more specific applications associated with the prompt. The application interface model may process the prompt to generate an application call, which may be used to interface with one or more specific applications to access and obtain the plurality of content items. The application call may be performed using one or more application programming interfaces. The one or more application programming interfaces may be implemented by an operating system of the computing device.

[0144] At 1006, the computing system may process the plurality of content items using a machine learning model to generate a structured output. The structured output may include information from the plurality of content items extracted in a structured data format (e.g., natural language output (e.g., articles, stories, poems, etc.), information graphics (e.g., tables, Venn diagrams, etc.), and / or media content items (e.g., videos, images, etc.)). The structured output may include formatting that is different from the native format of the plurality of content items. The structured output may include multimodal data. The machine learning model may include a generative model (e.g., a generative language model, a generative text-to-image model, a generative visual language model, and / or a generative graphics model).

[0145] In some implementations, the computing system may determine multiple objects associated with multiple content items and obtain multiple object details associated with the multiple objects. A structured output may be generated based on the multiple objects and the multiple object details. Additionally and / or alternatively, the structured output may include a graphical representation. The graphical representation may include object data and detail data. The object data may identify multiple objects. The detail data may describe multiple object details. In some implementations, the structured output may include multiple object images associated with the multiple objects. The structured output may include text describing the multiple object details.

[0146] At 1008, the computing system may provide a structured output for display as a response to the prompt. The structured output may be provided via an overlay interface. The structured output may be provided for display at a computing device. The structured output may be provided for display with the prompt, and the structured output may include one or more options for storing, sending, and / or enhancing the structured output.

[0147] In some implementations, the computing system may obtain a second prompt at a computing device. The second prompt may describe a subsequent request to obtain additional information associated with the structured content. The computing system may process the second prompt and the structured output to determine additional content in response to the subsequent request. The additional content may be determined based on determining that the structured output is associated with one or more entities and determining that the additional content is associated with the one or more entities. In some implementations, processing the second prompt and the structured output to determine the additional content may include determining that one or more second applications are associated with the second prompt and obtaining the additional content by interfacing with the one or more second applications. The additional content may include additional details about the content of the structured output, and the additional details may include location data for products listed in the model generation table.

[0148] Additionally and / or alternatively, the computing system may generate a second structured output based on the additional content. Generating the second structured output based on the additional content may include processing the additional content to generate a graphical representation associated with the additional content. A plurality of content items may be associated with a plurality of different products. The structured output may include a table. In some implementations, the table may include a structured representation of the details of a plurality of different products. The additional content may include one or more locations associated with a plurality of different products. The second structured output may include a graphical map of one or more indicators of the one or more locations.

[0149] In some implementations, the computing system may provide a second structured output at the computing device for display as a response to the second prompt. The second structured output may be displayed with the structured output and / or may replace the display location of the structured output. The second structured output may be provided for display with the prompt, and the second structured output may include one or more options for storing, sending, and / or enhancing the second structured output.

[0150] In some implementations, the computing system may determine that the structured output is associated with one or more second applications. The computing system may generate application suggestions based on the one or more second applications, obtain selections of the application suggestions, and generate a data package. The data package may describe the structured output. The computing system may provide the data package to the one or more second applications.

[0151] In some implementations, the computing system may obtain input data. The input data may describe a selection to input the structured output to the second application. The computing system may provide the structured output to the second application in response to the selection.

[0152] Alternatively and / or in addition, the computing system may obtain an enhanced input. The enhanced input may describe a request to adjust the structured output. The computing system may process the enhanced input and the structured output to generate an enhanced structured output. The enhanced structured output may include a structured output having one or more enhanced portions. Processing the enhanced input and the structured output to generate the enhanced structured output may include obtaining revision data based on the enhanced input and replacing a subset of the structured output with the revision data to generate the enhanced structured output. The revision data may include manually entered data, data obtained from the web, and / or data obtained from one or more applications on the computing device.

[0153] Fig.11 A block diagram of an example data aggregation system 1100 is depicted according to an example embodiment of the present disclosure. Specifically, the data aggregation system 1100 can process prompts 1102, perform application calls 1106 to obtain content items from one or more applications, and generate structured output 1112 based on the content items.

[0154] For example, a prompt 1102 may be obtained. The prompt 1102 may include a text string describing a request for information. The prompt 1102 may include an indication of a specific application from which to obtain data and / or may be an open request to be processed by the data aggregation system 1100 to determine which applications to extract data from for data aggregation. In some implementations, the prompt 1102 may include a multimodal prompt (e.g., text data and image data, audio data and image data, embedded data and text data, metadata and image data, etc.).

[0155] Prompt 1102 may be processed using application determination box 1104 to determine one or more applications to access and search to obtain one or more content items. An application may be determined based on determining that a prompt is associated with one or more topics, tasks, and / or entities associated with one or more specific applications. Alternatively and / or in addition, prompt 1102 may be parsed to determine that a request is associated with a specific application (e.g., an explicit request and / or an implicit request). One or more applications may include messaging applications (e.g., email, text, group chat, etc.), work management applications, storage applications (e.g., document management applications, media content item gallery applications, etc.), browser applications, search applications, note applications, streaming media applications, and / or other applications.

[0156] An application call 1106 may then be generated and executed based on the application determination. The application call 1106 may be facilitated by an overlay interface implemented in an operating system. In some implementations, the application call 1106 may be performed by an application programming interface and / or one or more other application interfacing systems. A content item determination 1108 may be performed based on the application call 1106. The content item determination 1108 may be used to determine that a plurality of content items of one or more applications are associated with the prompt 1102. The content item determination 1108 may include a keyword search, an embedded search, an image search, data parsing, a metadata search, and the like.

[0157] The one or more content items may then be processed using the generative model 1110 to generate a structured output 1112 including information from the one or more content items. The structured output 1112 may include natural language output, graphical representations, model-generated media content items, code, and / or other data. In some implementations, the format of the structured output 1112 may be based on the request of the prompt 1102. Alternatively and / or in addition, the format of the structured output 1112 may be based on a task, theme, and / or entity associated with the prompt 1102 and / or the content item.

[0158] In some implementations, the structured output 1112 may be provided with one or more action suggestions. For example, an enhancement option, a new prompt option, and / or a structured output interaction option may be provided to the user for selection. The enhancement option may be associated with an option to enhance at least a portion of the structured output 1112, which may include adding new information based on manual user input, another application call, web search, and / or other data acquisition. The enhancement may include format changes, style changes, data deletions, and / or content expansions (e.g., generating a long-form version of the structured output 1112 based on additional generation model 1110 processing). The new prompt option may include processing the structured output 1112 and / or a second prompt using the data aggregation system 1100 to generate a second structured output. The structured output interaction option may include storing the structured output 1112, sending the structured output 1112 to one or more applications and / or one or more users, and / or interacting with a user interface element of the structured output 1112.

[0159] Fig. 12A A block diagram of an example computing system 100 performing a visual search is depicted according to an example embodiment of the present disclosure. System 100 includes a user computing system 102, a server computing system 130, and / or a third computing system 150 communicatively coupled via a network 180.

[0160] The user computing system 102 may include any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0161] The user computing system 102 includes one or more processors 112 and a memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 that are executed by the processor 112 to cause the user computing system 102 to perform operations.

[0162] In some implementations, the user computing system 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks.

[0163] In some implementations, one or more machine learning models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing system 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., to perform parallel machine learning model processing across multiple instances of input data and / or detected features).

[0164] More specifically, the one or more machine learning models 120 may include one or more detection models, one or more classification models, one or more segmentation models, one or more enhancement models, one or more generation models, one or more natural language processing models, one or more optical character recognition models, and / or one or more other machine learning models. The one or more machine learning models 120 may include one or more transformer models. The one or more machine learning models 120 may include one or more neural radiation field models, one or more diffusion models, and / or one or more autoregressive language models.

[0165] One or more machine learning models 120 can be used to detect one or more object features. The detected object features can be classified and / or embedded. The classification and / or embedding can then be used to perform a search to determine one or more search results. Alternatively and / or in addition, one or more detected features can be used to determine an indicator to be provided (e.g., a user interface element indicating the detected feature) to indicate that the feature has been detected. The user can then select the indicator to perform feature classification, embedding, and / or searching. In some implementations, the classification, embedding, and / or searching can be performed before selecting the indicator.

[0166] In some implementations, the one or more machine learning models 120 may process image data, text data, audio data, and / or potentially coded data to generate output data, which may include image data, text data, audio data, and / or potentially coded data. The one or more machine learning models 120 may perform optical character recognition, natural language processing, image classification, object classification, text classification, audio classification, context determination, action prediction, image correction, image enhancement, text enhancement, sentiment analysis, object detection, error detection, restoration, video stabilization, audio correction, audio enhancement, and / or data segmentation (e.g., mask-based segmentation).

[0167] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing system 102 according to a client-server relationship. For example, the machine learning model 140 may be implemented by the server computing system 130 as part of a web service (e.g., a viewfinder service, a visual search service, an image processing service, an environmental computing service, and / or an overlay application service). Thus, one or more models 120 may be stored and implemented at the user computing system 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0168] The user computing system 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other means by which a user may provide user input.

[0169] In some implementations, the user computing system may store and / or provide one or more user interfaces 124, which may be associated with one or more applications. One or more user interfaces 124 may be configured to receive input and / or provide data for display (e.g., image data, text data, audio data, one or more user interface elements, augmented reality experience, virtual reality experience, and / or other data for display). The user interface 124 may be associated with one or more other computing systems (e.g., server computing system 130 and / or third-party computing system 150). The user interface 124 may include a viewfinder interface, a search interface, a generative model interface, a social media interface, and / or a media content gallery interface.

[0170] The user computing system 102 may include one or more sensors 126 and / or receive data from one or more sensors. The one or more sensors 126 may be housed in a housing assembly that houses one or more processors 112, memory 114, and / or one or more hardware components that may store one or more software packages and / or cause one or more software packages to be executed. The one or more sensors 126 may include one or more image sensors (e.g., cameras), one or more lidar sensors, one or more audio sensors (e.g., microphones), one or more inertial sensors (e.g., inertial measurement units), one or more biosensors (e.g., heart rate sensors, pulse sensors, retinal sensors, and / or fingerprint sensors), one or more infrared sensors, one or more position sensors (e.g., GPS), one or more touch sensors (e.g., conductive touch sensors and / or mechanical touch sensors), and / or one or more other sensors. One or more sensors may be utilized to obtain data associated with a user environment (e.g., an image of the user environment, a record of the environment, and / or the user's location).

[0171] The user computing system 102 may include and / or be part of a user computing device 104. The user computing device 104 may include a mobile computing device (e.g., a smartphone or tablet computer), a desktop computer, a laptop computer, a smart wearable device, and / or a smart appliance. Additionally and / or alternatively, the user computing system may acquire data from one or more user computing devices 104 and / or generate data using the one or more user computing devices. For example, a smartphone camera may be used to capture image data describing an environment, and / or an overlay application of the user computing device 104 may be used to track and / or process data provided to the user. Similarly, one or more sensors associated with a smart wearable device may be used to acquire data about the user and / or about the user's environment (e.g., a camera housed in a user's smart glasses may be used to acquire image data). Additionally and / or alternatively, data may be acquired and uploaded from other user devices that may be specifically used for data acquisition or generation.

[0172] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 executed by the processor 132 to cause the server computing system 130 to perform operations.

[0173] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0174] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the model 140 may be or may otherwise include a variety of machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Fig. 12B Discuss example model 140 .

[0175] Additionally and / or alternatively, server computing system 130 may include and / or be communicatively connected to a search engine 142 that may be used to crawl one or more databases (and / or resources). Search engine 142 may process data from user computing system 102, server computing system 130, and / or third-party computing system 150 to determine one or more search results associated with the input data. Search engine 142 may perform term-based searches, tag-based searches, Boolean-based searches, image searches, embedding-based searches (e.g., nearest neighbor searches), multimodal searches, and / or one or more other search techniques.

[0176] The server computing system 130 may store and / or provide one or more user interfaces 144 to obtain input data and / or provide output data to one or more users. The one or more user interfaces 144 may include one or more user interface elements, which may include input fields, navigation tools, content widgets, selectable tiles, widgets, data display carousels, dynamic animations, information pop-ups, image enhancements, text-to-speech, speech-to-text, augmented reality, virtual reality, feedback loops, and / or other interface elements.

[0177] User computing system 102 and / or server computing system 130 may train models 120 and / or 140 through interaction with a third-party computing system 150, which is communicatively coupled via network 180. Third-party computing system 150 may be separate from server computing system 130 or may be a part of server computing system 130. Alternatively and / or in addition, third-party computing system 150 may be associated with one or more web resources, one or more web platforms, one or more other users, and / or one or more contexts.

[0178] The third-party computing system 150 may include one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the third-party computing system 150 to perform operations. In some implementations, the third-party computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0179] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be conducted over any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0180] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0181] In some implementations, the input of the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of the image data, potential embedding of the image data, encoded representation of the image data, hash of the image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change in the image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning model may process the image data to generate an amplified image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0182] In some implementations, the input of the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a conversion output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data with higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0183] In some implementations, the input of the machine learning model of the present disclosure may be speech data. The machine learning model may process the speech data to generate an output. As an example, the machine learning model may process the speech data to generate a speech recognition output. As another example, the machine learning model may process the speech data to generate a speech conversion output. As another example, the machine learning model may process the speech data to generate a potential embedding output. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an amplified speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a predicted output.

[0184] In some implementations, the input of the machine learning model of the present disclosure may be sensor data. The machine learning model may process the sensor data to generate an output. As an example, the machine learning model may process the sensor data to generate a recognition output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a classification output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a visualization output. As another example, the machine learning model may process the sensor data to generate a diagnostic output. As another example, the machine learning model may process the sensor data to generate a detection output.

[0185] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data for one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the possibility that one or more images depict an object belonging to an object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the possibility that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, a corresponding possibility for each category in a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be an object class. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel in one of the input images, the motion of the scene depicted at the pixel between the images in the network input.

[0186] The user computing system may include multiple applications (e.g., applications 1 to N). Each application may include its own corresponding machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0187] Each application can communicate with multiple other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0188] The user computing system 102 may include multiple applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0189] The central intelligence layer can include multiple machine learning models. For example, a corresponding machine learning model (e.g., model) can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included in or otherwise implemented by an operating system of the computing system 100.

[0190] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized data repository for the computing system 100. The central device data layer can communicate with many other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0191] Fig. 12BA block diagram of an example computing system 50 for performing visual search according to an example embodiment of the present disclosure is depicted. Specifically, the example computing system 50 may include one or more computing devices 52, which may be used to acquire and / or generate one or more data sets, which may be processed by a sensor processing system 60 and / or an output determination system 80 for feedback to a user, who may provide information about features in one or more acquired data sets. The one or more data sets may include image data, text data, audio data, multimodal data, potentially encoded data, etc. The one or more data sets may be acquired by one or more sensors associated with the one or more computing devices 52 (e.g., one or more sensors in the computing devices 52). Additionally and / or alternatively, the one or more data sets may be stored data and / or retrieved data (e.g., data retrieved from a web resource). For example, a user may interact with images, text, and / or other content items. One or more determinations may then be generated using the interaction with the content items.

[0192] One or more computing devices 52 may acquire and / or generate one or more data sets based on image capture, sensor tracking, data storage retrieval, content download (e.g., downloading images or other content items from web resources via the Internet), and / or through one or more other technologies. One or more data sets may be processed using a sensor processing system 60. The sensor processing system 60 may perform one or more processing techniques using one or more machine learning models, one or more search engines, and / or one or more other processing techniques. The one or more processing techniques may be performed in any combination and / or individually. The one or more processing techniques may be performed serially and / or in parallel. Specifically, one or more data sets may be processed using a context determination block 62, which may determine a context associated with one or more content items. The context determination block 62 may identify and / or process metadata, user profile data (e.g., preferences, user search history, user browsing history, user purchase history, and / or user input data), previous interaction data, global trend data, location data, time data, and / or other data to determine a specific context associated with a user. A context may be associated with an event, a determined trend, a particular action, a particular type of data, a particular environment, and / or another context associated with a user and / or the retrieved or acquired data.

[0193] The sensor processing system 60 may include an image pre-processing block 64. The image pre-processing block 64 may be used to adjust one or more values ​​of an acquired and / or received image to prepare the image to be processed by one or more machine learning models and / or one or more search engines 74. The image pre-processing block 64 may resize the image, adjust saturation values, adjust resolution, strip and / or add metadata, and / or perform one or more other operations.

[0194] In some implementations, the sensor processing system 60 may include one or more machine learning models, which may include a detection model 66, a segmentation model 68, a classification model 70, an embedding model 72, and / or one or more other machine learning models. For example, the sensor processing system 60 may include one or more detection models 66, which may be used to detect specific features in the processed dataset. Specifically, one or more detection models 66 may be used to process one or more images to generate one or more bounding boxes associated with detected features in the one or more images.

[0195] Additionally and / or alternatively, one or more portions of the data set may be segmented from the one or more data sets using one or more segmentation models 68. For example, the one or more segmentation models 68 may utilize one or more segmentation masks (e.g., one or more segmentation masks generated manually and / or based on one or more bounding boxes) to segment a portion of an image, a portion of an audio file, and / or a portion of text. Segmentation may include isolating one or more detected objects and / or removing one or more detected objects from an image.

[0196] One or more classification models 70 can be used to process image data, text data, audio data, potential coding data, multimodal data, and / or other data to generate one or more classifications. One or more classification models 70 may include one or more image classification models, one or more object classification models, one or more text classification models, one or more audio classification models, and / or one or more other classification models. One or more classification models 70 can process data to determine one or more classifications.

[0197] In some implementations, one or more embedding models 72 may be used to process data to generate one or more embeddings. For example, one or more embedding models 72 may be used to process one or more images to generate one or more image embeddings in an embedding space. The one or more image embeddings may be associated with one or more image features of the one or more images. In some implementations, the one or more embedding models 72 may be configured to process multimodal data to generate multimodal embeddings. The one or more embeddings may be used to classify, search, and / or learn an embedding space distribution.

[0198] The sensor processing system 60 may include one or more search engines 74 that may be used to perform one or more searches. The one or more search engines 74 may crawl one or more databases (e.g., one or more local databases, one or more global databases, one or more private databases, one or more public databases, one or more dedicated databases, and / or one or more general databases) to determine one or more search results. The one or more search engines 74 may perform feature matching, text-based searches, embedding-based searches (e.g., k-nearest neighbor searches), metadata-based searches, multimodal searches, web resource searches, image searches, text searches, and / or application searches.

[0199] Additionally and / or alternatively, the sensor processing system 60 may include one or more multimodal processing blocks 76 that may be used to assist in processing multimodal data. The one or more multimodal processing blocks 76 may include generating multimodal queries and / or multimodal embeddings for processing by one or more machine learning models and / or one or more search engines 74.

[0200] The output of the sensor processing system 60 may then be processed using an output determination system 80 to determine one or more outputs to provide to a user. The output determination system 80 may include a heuristic-based determination, a machine learning model-based determination, a user-selected determination, and / or a context-based determination.

[0201] The output determination system 80 can determine how and / or where to provide one or more search results in the search results interface 82. In addition and / or alternatively, the output determination system 80 can determine how and / or where to provide one or more machine learning model outputs in the machine learning model output interface 84. In some implementations, one or more search results and / or one or more machine learning model outputs can be provided for display through one or more user interface elements. One or more user interface elements can be superimposed on the displayed data. For example, one or more detection indicators can be superimposed on the detected object in the viewfinder. One or more user interface elements can be selected to perform one or more additional searches and / or one or more additional machine learning model processes. In some implementations, user interface elements can be provided as specialized user interface elements for specific applications and / or can be provided uniformly in different applications. One or more user interface elements may include pop-up displays, interface overlays, interface tiles and / or small pieces, carousel interfaces, audio feedback, animations, interactive widgets, and / or other user interface elements.

[0202] Additionally and / or alternatively, data associated with the output of the sensor processing system 60 may be utilized to generate and / or provide an augmented reality experience and / or a virtual reality experience 86. For example, one or more acquired data sets may be processed to generate one or more augmented reality rendering assets and / or one or more virtual reality rendering assets, which may then be used to provide an augmented reality experience and / or a virtual reality experience 86 to a user. The augmented reality experience may render information associated with an environment into the corresponding environment. Alternatively and / or additionally, objects associated with the processed data sets may be rendered into the user environment and / or the virtual environment. Rendering data set generation may include training one or more neural radiance field models to learn a three-dimensional representation of one or more objects.

[0203] In some implementations, one or more action prompts 88 can be determined based on the output of the sensor processing system 60. For example, a search prompt, a purchase prompt, a generate prompt, a reservation prompt, a call prompt, a redirect prompt, and / or one or more other prompts can be determined to be associated with the output of the sensor processing system 60. The one or more action prompts 88 can then be provided to the user via one or more selectable user interface elements. In response to selection of the one or more selectable user interface elements, a corresponding action of the corresponding action prompt can be performed (e.g., a search can be performed, a purchase application interface can be utilized, and / or another application can be opened).

[0204] In some implementations, one or more generative models 90 may be used to process one or more data sets and / or output of the sensor processing system 60 to generate model-generated content items that may then be provided to the user. The generation may be prompted based on user selections and / or may be performed automatically (e.g., automatically based on one or more conditions that may be associated with a threshold amount of search results that are not represented).

[0205] One or more generative models 90 may include a language model (e.g., a large language model and / or a visual language model), an image generation model (e.g., a text-to-image generation model and / or an image enhancement model), an audio generation model, a video generation model, a graphics generation model, and / or other data generation models (e.g., other content generation models). One or more generative models 90 may include one or more Transformer models, one or more convolutional neural networks, one or more recurrent neural networks, one or more feedforward neural networks, one or more generative adversarial networks, one or more self-attention models, one or more embedding models, one or more encoders, one or more decoders, and / or one or more other models. In some implementations, one or more generative models 90 may include one or more autoregressive models (e.g., a machine learning model trained to generate predicted values ​​based on prior behavior data) and / or one or more diffusion models (e.g., a machine learning model trained to generate predicted data based on generating and processing distribution data associated with input data).

[0206] One or more generative models 90 can be trained to process input data and generate model-generated content items, which can include multiple predicted words, pixels, signals, and / or other data. The content items generated by the model can include novel content items that are different from any existing works. One or more generative models 90 can utilize learned representations, sequences, and / or probability distributions to generate content items, which can include phrases, storylines, settings, objects, characters, beats, lyrics, and / or other aspects that are not included in existing content items.

[0207] The one or more generative models 90 may include a visual language model. The visual language model may be trained, tuned, and / or configured to process image data and / or text data to generate natural language output. The visual language model may utilize a pre-trained large language model (e.g., a large autoregressive language model) and one or more encoders (e.g., one or more image encoders and / or one or more text encoders) to provide detailed natural language output that simulates natural language written by humans.

[0208] The visual language model can be used for zero-shot image classification, few-shot image classification, image captioning, multimodal query refinement, multimodal question answering, and / or can be tuned and / or trained for multiple different tasks. The visual language model can perform visual question answering, image caption generation, feature detection (e.g., content monitoring (e.g., for inappropriate content)), object detection, scene recognition, and / or other tasks.

[0209] The visual language model can utilize a pre-trained language model and then tune it to achieve multimodality. The training and / or tuning of the visual language model can include image-text matching, masked language modeling, multimodal fusion of cross attention, contrastive learning, prefix language model training, and / or other training techniques. For example, the visual language model can be trained to process images to generate predicted text similar to real-value text data (e.g., real-value subtitles of images). In some implementations, the visual language model can be trained to replace the masked word units of the natural language template with text words describing the features depicted in the input image. Alternatively and / or in addition, training, tuning, and / or model reasoning can include multi-layer connections of visual and text embedding features. In some implementations, the visual language model can be trained and / or tuned by jointly learning image embedding and text embedding generation, which can include training and / or tuning the system to map the embedding to a joint feature embedding space that maps text features and image features to a shared embedding space. Joint training can include parallel embedding of image-text pairs and / or can include triple training. In some implementations, images can be utilized and / or processed as prefixes to a language model.

[0210] One or more generation models 90 may be stored on the device and / or may be stored on the server computing system. In some implementations, one or more generation models 90 may perform on-device processing to determine the suggested search, the suggested action, and / or the suggested prompt. One or more generation models 90 may include one or more compact visual language models, which may include fewer parameters than the visual language model stored and operated by the server computing system. The compact visual language model may be trained by refining training. In some implementations, the visual language model may process display data to generate suggestions. The display data may include a single image describing a screenshot and / or may include image data, metadata, and / or other data describing a period of time before the currently displayed content (e.g., applications, images, videos, messages, and / or other content viewed in the past 30 seconds). The user computing device may generate and store a rolling buffer window (e.g., 30 seconds) of data describing the content displayed during the buffering period. Once the time has passed, the data may be deleted. The rolling buffer window data may be used to determine a context, which may be used for query, content, action, and / or prompt suggestions.

[0211] The output determination system 80 may use a data enhancement block 92 to process one or more data sets and / or the output of the sensor processing system 60 to generate enhanced data. For example, one or more images may be processed using the data enhancement block 92 to generate one or more enhanced images. Data enhancement may include data correction, data cropping, removal of one or more features, addition of one or more features, resolution adjustment, lighting adjustment, saturation adjustment, and / or other enhancements.

[0212] In some implementations, one or more data sets and / or outputs of the sensor processing system 60 may be stored based on a determination by the data storage block 94 .

[0213] The output of the output determination system 80 may then be provided to the user via one or more output components of the user computing device 52. For example, one or more user interface elements associated with the one or more outputs may be provided for display via a visual display of the user computing device 52.

[0214] The process may be performed iteratively and / or continuously.One or more user inputs to provided user interface elements may adjust and / or influence the continuous processing loop.

[0215] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from these systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and partitions of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0216] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, rather than limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, can easily produce changes, modifications, and equivalents of these embodiments. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the subject matter that will be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment can be used together with another embodiment to produce yet another embodiment. Therefore, the present disclosure is intended to cover these changes, changes, and equivalents.

Claims

1. A computing system, the system comprising: one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: obtaining display data, wherein the display data describes content currently presented for display in a first application on a user computing device; processing at least a portion of the display data to generate visual search data, wherein the visual search data includes one or more visual search results, wherein the one or more visual search results are associated with a detected feature in the display data; determining that a particular second application on the computing device is associated with the visual search data; and An application suggestion associated with the particular second application is provided based on the visual search data.

2. The system of claim 1, wherein the operations further comprise: obtaining a selection of the application suggestion to send at least a portion of the visual search data to the particular second application; generating a model-generated content item based on the selection of the application suggestion, wherein the model-generated content item is generated using a generative model based on the portion of the visual search data; as well as The content item generated by the model is provided to the specific second application.

3. The system of claim 2, wherein obtaining the selection of the application suggestion to send at least the portion of the visual search data to the second application comprises: determining a plurality of application transfer actions associated with the visual search data, wherein the plurality of application transfer actions are associated with a plurality of candidate second applications for transmitting data associated with the visual search data; providing a plurality of selectable options based on the plurality of application transfer actions, wherein the plurality of selectable options are associated with the plurality of application transfer actions, wherein the plurality of selectable options include the application suggestions, and wherein the plurality of application transfer actions include the particular second application; and A selection of the application suggestion is received, wherein the application suggestion is associated with the particular second application.

4. The system of claim 2, wherein generating the model-generated content item based on the selection of the option comprises: processing the visual search data and data associated with the particular second application to determine a suggested prompt; receiving input selecting the suggested prompt; as well as The suggested hint and the visual search data are processed using the generative model to generate the model-generated content item.

5. The system of claim 2, wherein the generative model comprises a generative language model that generates natural language output based on processing features of input data.

6. The system of claim 1, wherein acquiring the display data comprises: A screenshot is generated, wherein the screenshot describes a number of pixels provided for display.

7. The system of claim 1 , wherein processing the portion of the display data to generate the visual search data comprises: processing the display data using an object detection model to determine one or more objects are depicted in the display data; generating a segmented portion of the display data, wherein the segmented portion includes the one or more objects; as well as The segmented portions of the display data are processed to generate the visual search data.

8. The system of claim 7, wherein the object detection model generates one or more bounding boxes, wherein the one or more bounding boxes describe locations of the one or more objects within the content currently presented for display.

9. The system of claim 8, wherein generating the segmented portion of the display data comprises: The display data and the one or more bounding boxes are processed using a segmentation model to generate the segmented portion of the display data.

10. A system as described in claim 9, wherein the object detection model and the segmentation model are machine learning models, wherein the object detection model and the segmentation model are stored on the user computing device, and wherein processing the portion of the display data to generate the visual search data is performed on the device.

11. A computer-implemented method, the method comprising: acquiring, by a computing system including one or more processors, display data describing content currently presented for display on a user computing device; processing, by the computing system, at least a portion of the display data to generate visual search data, wherein the visual search data includes one or more visual search results, wherein the one or more visual search results are associated with a detected feature in the display data; processing, by the computing system, the visual search data to determine that a second application is associated with the one or more visual search results; as well as Application suggestions associated with the second application are provided, by the computing system, based on the visual search data.

12. The method of claim 11, further comprising: obtaining, by the computing system, a selection of the application suggestion to send a portion of the visual search data to the second application; generating, by the computing system, a model-generated content item based on the selection of the suggestion, wherein the model-generated content item is generated using a generative model based on the portion of the visual search data; as well as The content items generated by the model and the portion of the visual search data are provided by the computing system to the second application, wherein the first application is different from a particular second application.

13. The method of claim 12, wherein the particular second application comprises a messaging application, and wherein the model-generated content item comprises a model-written message to a second user, wherein the model-generated content item is generated using a generative language model.

14. The method of claim 12, wherein the model-generated content items include a model-generated list that organizes a plurality of user-selected visual search results, and wherein the model-generated list is generated using a generative language model that organizes the plurality of user-selected visual search results and generates a natural language output for each of the plurality of user-selected visual search results.

15. A method as claimed in claim 11, wherein the display data is generated using a visual search application in the operating system, wherein the visual search application includes an overlay application that is compatible to generate and process screenshots across multiple different applications on the user computing device.

16. The method of claim 11, further comprising: determining a plurality of candidate second applications to be associated with the visual search data; as well as A plurality of application suggestions are provided for display in a suggestion panel, wherein the suggestion panel includes the plurality of application suggestions and one or more query suggestions.

17. One or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising: acquiring display data, wherein the display data describes content presented for display on a user computing device; processing the display data using one or more on-device machine learning models to generate segmented portions of the display data, wherein the segmented portions of the display data include data describing a set of features of the content presented for display; sending the segmented portions of the display data to a server computing system; receiving visual search data from the server computing system, wherein the visual search data includes one or more search results, wherein the one or more search results are associated with detected features in the segmented portion of the display data; processing the visual search data to determine a plurality of candidate second applications associated with the one or more search results; obtaining a selection of a specific application suggestion to send at least a portion of the visual search data to a specific second application among the plurality of candidate second applications; obtaining a model-generated content item based on the selection of the specific application suggestion, wherein the model-generated content item is generated using a generative model based on the portion of the visual search data; as well as The content item generated by the model is provided to the specific second application.

18. One or more non-transitory computer-readable media as described in claim 17, wherein the visual search data includes the one or more search results and a model-generated knowledge panel, wherein the model-generated knowledge panel includes a summary of topics associated with the segmented portion of the display data, and wherein the summary is generated by processing web resource data using a language model.

19. One or more non-transitory computer-readable media as described in claim 17, wherein the one or more on-device machine learning models include an object detection model and a segmentation model stored on the user computing device, and wherein the display data is obtained and processed based on user input requesting a visual search overlay application.

20. The one or more non-transitory computer-readable media of claim 17, wherein providing the model-generated content item to the particular second application comprises sending the model-generated content item to the second application through an application programming interface.

Images