Environmental multi-device framework for agent partners

Abstract

The invention relates to an environmental multi-device framework for agent partners. Systems and methods for generating and providing output in a multi-device system may include providing dynamic response generation and display using environment-based cue generation and generative model response generation. The systems and methods may obtain input data associated with one or more computing devices within an environment, may obtain environmental data describing a plurality of computing devices within the environment, and may generate hints based on the input data and the environmental data. The cues may be processed with a generative model to generate a model-generated output. The model generated output may then be sent to a particular computing device of the plurality of computing devices.

Description

Priority claim

[0001] This application is based upon and claims priority to U.S. non-provisional application No. 18 / 390,768, filed on December 20, 2023, which is incorporated herein by reference. Technical Field

[0002] The present disclosure generally relates to generating and providing output in a multi-device system. More specifically, the present disclosure relates to regulating output generation based on devices in an environment and determining which computing device provides the output. Background Art

[0003] Computing devices may be found throughout a user's living room, bedroom, den, office, and / or other environments. A multi-device environment may provide a user with multiple computing devices to interact with, which may include smart TVs, smart speakers, smart appliances, virtual assistant devices, tablet computers, smart wearable devices, smartphones, and / or other computing devices that may be provided throughout the user's environment. However, due to the lack of interconnectivity between devices, the capabilities of these devices may not be utilized. For example, a user may be performing a search on a smartphone, which may result in a video search result being played on the smartphone, while the smart TV is just a few feet away.

[0004] Understanding the world at large can be difficult. Whether an individual is trying to understand what the object in front of them is, trying to determine where else the object can be found, and / or trying to determine where an image on the Internet was captured, conducting a text search alone can be difficult. Specifically, users may have difficulty determining which words to use. Additionally, their words may not be descriptive and / or rich enough to produce the desired results. Summary of the invention

[0005] Aspects and advantages of embodiments of the present disclosure will be set forth in part in the following description, or may be learned from the description, or may be learned through practice of the embodiments.

[0006] An example aspect of the present disclosure relates to a computing system for determining an output device for providing a query response. The system may include one or more processors and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform an operation. The operation may include obtaining input data. The input data may include a query associated with a particular user. The operation may include obtaining environmental data. The environmental data may describe multiple computing devices in the user's environment. In some implementations, multiple computing devices may be associated with multiple different output components. The operation may include generating a prompt based on the input data and the environmental data. The prompt may include data describing the query and device information associated with at least a subset of multiple computing devices. The operation may include processing the prompt with a generative model to generate a model-generated output. The model-generated output may include a response to the query. In some implementations, the model-generated output may be generated to be provided by a specific computing device among multiple computing devices. The operation may include sending the model-generated output to a specific computing device.

[0007] In some implementations, the output generated by the model may be generated to be provided by a specific output component among a plurality of different output components. The generative model may generate output device instructions. The output device instructions may describe a specific computing device among a plurality of computing devices to provide the output generated by the model. The specific computing device may be associated with a specific output component. The output generated by the model may be sent to the specific computing device based on the output device instructions.

[0008] In some implementations, processing a prompt with a generative model to generate a model-generated output may include generating a plurality of model outputs. The plurality of model outputs may include a plurality of candidate responses. Sending the model-generated output to a particular computing device may include: sending a first model output of the plurality of model outputs to a first computing device of the plurality of computing devices; and sending a second model output of the plurality of model outputs to a second computing device of the plurality of computing devices. The first model output may include visual data for display via a visual display. The second model output may include audio data for playback via a speaker assembly. In some implementations, the first computing device may include a smart television, and wherein the second computing device includes a smart speaker.

[0009] In some implementations, generating a prompt based on the input data and the environment data may include: determining an environment-specific device configuration based on the environment data; obtaining a prompt template from a prompt library based on the environment-specific device configuration; and enhancing the prompt template based on the input data to generate the prompt. The environment-specific device configuration may describe a respective output type and a respective output quality of a plurality of computing devices. The prompt library may include a plurality of different prompt templates associated with a plurality of different device configurations.

[0010] In some implementations, multiple computing devices may be connected via a cloud computing system. Each of the multiple computing devices may be registered with a platform of the cloud computing system. Environmental data may be obtained using the cloud computing system. Model-generated output may be sent via the cloud computing system. In some implementations, multiple computing devices may be located close to each other computing device within the multiple computing devices. Multiple computing devices may be communicatively connected via a local network. Specific computing devices in the multiple computing devices may facilitate the acquisition of input data and the transmission of model-generated output.

[0011] In some implementations, the generative model may be communicatively connected to the search engine via an application programming interface. Processing the prompt with the generative model to generate the model-generated output may include: generating an application programming interface call based on the prompt; determining a plurality of search results with the search engine based on the application programming interface call; and processing the plurality of search results with the generative model to generate the model-generated output.

[0012] Another example aspect of the present disclosure relates to a computer-implemented method. The method may include obtaining input data by a computing system including one or more processors. The input data may include a query associated with a specific user. The method may include obtaining environmental data by the computing system. The environmental data may describe multiple computing devices in the environment of the user. The multiple computing devices may be associated with multiple different output components. The method may include generating a prompt by the computing system based on the input data and the environmental data. The prompt may include data describing the query and device information associated with at least a subset of the multiple computing devices. The method may include processing the prompt with a generative model by the computing system to generate a model-generated output and an output device instruction. The model-generated output may include a response to the query. In some implementations, the model-generated output may be generated to be provided by a specific output component among multiple different output components. The output device instruction may describe a specific computing device among multiple computing devices to provide the model-generated output. The specific computing device may be associated with a specific output component. The method may include sending the model-generated output to the specific computing device based on the output device instruction by the computing system.

[0013] In some implementations, a plurality of different output components may be associated with a plurality of corresponding output capabilities associated with a plurality of computing devices. Each of the plurality of corresponding output capabilities may describe an output type and quality available via a corresponding computing device. The output device instructions may include an application programming interface call to send the output generated by the model to a particular computing device. In some implementations, the plurality of different output components may include a speaker associated with a first device and a visual display associated with a second device. The method may include determining, by a computing system, that a particular output component is associated with the intent of the query. A prompt may be generated based on the association of the particular output component with the intent of the query.

[0014] In some implementations, the method may include determining, by the computing system and based on the environmental data, an output hierarchy based on specification information for a plurality of different output components. A prompt may be generated based on the output hierarchy and the query.

[0015] Another example aspect of the present disclosure relates to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause one or more computing devices to perform operations. The operations may include obtaining environmental data. The environmental data may describe multiple computing devices within an environment associated with a particular user. The operations may include processing the environmental data to determine a plurality of corresponding input capabilities and a plurality of corresponding output capabilities associated with the multiple computing devices. The plurality of corresponding input capabilities may be associated with candidate input types associated with the multiple computing devices. The plurality of corresponding output capabilities may be associated with candidate output types associated with the multiple computing devices. The operations may include generating a plurality of corresponding interfaces for the multiple computing devices based on the plurality of corresponding input capabilities and the plurality of corresponding output capabilities. Based on the fact that the multiple computing devices are based on the plurality of corresponding input capabilities and the plurality of corresponding output capabilities, the plurality of corresponding interfaces may be specifically used for the multiple computing devices. The operations may include providing a plurality of corresponding interfaces to the multiple computing devices.

[0016] In some implementations, the plurality of corresponding interfaces may include a plurality of device indicators indicating a plurality of computing devices within an environment associated with a particular user. The plurality of computing devices may be configured as a user-specific device ecosystem that is communicatively connected to receive input and provide output. Each of the plurality of corresponding interfaces may be configured to receive a particular input type and provide a particular output type based on a respective input capability and a respective output capability of a particular computing device in the plurality of computing devices.

[0017] In some implementations, the operation may include: obtaining user input via a first interface of a first computing device among multiple computing devices; processing the user input with a search engine to determine multiple search results; processing the multiple search results with a generative model to generate a model output; and providing the model output for display via a second interface of a second computing device among the multiple computing devices.

[0018] Other aspects of the disclosure relate to various systems, apparatuses, non-transitory computer-readable media, user interfaces, and electronic devices.

[0019] These and other features, aspects and advantages of various embodiments of the present disclosure will be better understood with reference to the following description and appended claims.The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the present disclosure and, together with the description, serve to explain the relevant principles. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] A detailed discussion of the embodiments for those skilled in the art is set forth in this specification with reference to the accompanying drawings, in which:

[0021] Figure 1 Depicted is a block diagram of an example generative model system according to an example embodiment of the present disclosure.

[0022] Figure 2 Depicted is a block diagram of an example response generation system according to an example embodiment of the present disclosure.

[0023] Figure 3 Depicted is a flow diagram of an example method of performing response generation according to an example embodiment of the present disclosure.

[0024] Figure 4 Depicted is a block diagram of an example multi-device management system according to an example embodiment of the present disclosure.

[0025] Figure 5 A block diagram of an example environment personalization system is depicted according to an example embodiment of the present disclosure.

[0026] Fig. 6A Depicted are diagrams of example interfaces according to example embodiments of the present disclosure.

[0027] Figure 6B Depicted is a diagram of an example image capture entry point in accordance with an example embodiment of the present disclosure.

[0028] Figure 6C A diagram of an example Smart TV entry point is depicted in accordance with an example embodiment of the present disclosure.

[0029] Fig.6D Depicted is a diagram of an example calendar interface according to an example embodiment of the present disclosure.

[0030] Fig. 6E Depicted is a diagram of an example video conferencing interface according to an example embodiment of the present disclosure.

[0031] Fig. 6F Depicted is a diagram of an example email interface according to an example embodiment of the present disclosure.

[0032] Figure 6G Depicted is a diagram of an example video player interface according to an example embodiment of the present disclosure.

[0033] Figure 7 Depicted is a flow diagram of an example method of performing output generation and routing according to an example embodiment of the present disclosure.

[0034] Figure 8 Depicted is a flow chart of an example method of performing interface generation according to an example embodiment of the present disclosure.

[0035] Fig.9A Depicted is a block diagram of an example computing system that performs multi-device output management according to an example embodiment of the present disclosure.

[0036] Fig. 9B Depicted is a block diagram of an example computing system that performs multi-device output management according to an example embodiment of the present disclosure.

[0037] Reference numerals that are repeated across multiple drawings are intended to identify like features in various implementations. DETAILED DESCRIPTION

[0038] In general, the present disclosure relates to a multi-device framework for managing input acquisition and output generation for an environment including multiple computing devices. Specifically, the systems and methods disclosed herein may utilize environment-based prompt generation and generative model processing to generate and / or provide outputs that can be generated based on computing devices available in the environment. For example, the multi-device framework may include a computing system that facilitates obtaining input data and environmental data associated with multiple computing devices, and generates one or more model-generated outputs that are configured to be provided by one or more specific computing devices in the multiple computing devices. The computing system can be used to utilize the diverse input and / or output capabilities of different computing devices within the environment, which may include automatically obtaining different types of input from different devices and / or providing different types of output via different devices (e.g., voice commands may be received via a smartphone, while output may be provided via a visual display of a television and / or speakers of a sound system).

[0039] Environmental data associated with multiple computing devices within an environment can be utilized for prompt generation, environmental understanding, and / or interface generation. Environmental data can be processed to generate prompts that can adjust the output generation of a generative model to be compatible with and / or optimized for display and / or playback in a user environment. Prompt generation may include obtaining a specific prompt template from a prompt library based on a computing device associated with the environment, and / or may include prompt generation by processing input data and / or environmental data with a machine learning model to generate prompts. Additionally and / or optionally, environmental data may be processed to determine which computing devices associated with the environment are used for a specific type of input acquisition and / or a specific type of output playback. In some implementations, environmental data may be processed to understand computing devices within the environment, and corresponding interfaces may be generated for multiple computing devices based on the determined input and / or output capabilities of the computing devices associated with the environment.

[0040] An ambient multi-device framework may be utilized to provide an immersive virtual assistant that may receive input from and provide output to multiple different computing devices. The ambient multi-device framework may obtain queries and / or prompts from a user and provide responses with relevant content types using computing devices in the environment that may provide the relevant content types at a higher quality than other devices in the environment. Determination of the computing devices may be based on a determined hierarchy that may be determined based on device specification information (and / or device capability information).

[0041] Computing devices may be found throughout a user's environment, whether that environment is at an office, at home, or elsewhere. Smart TVs, smart speakers, smart appliances, virtual assistant devices, tablet computers, smart wearables, smartphones, and / or other computing devices may be provided throughout a user's environment; however, the capabilities of these devices may not be utilized due to a lack of interconnectivity and / or lack of collaboration between the devices. Specifically, computing devices may not be able to determine when and / or how to interact with each other to obtain input and / or provide context-specific output.

[0042] The environment multi-device framework may include obtaining and / or generating information describing computing devices in the environment, including information associated with input and / or output capabilities. This information may then be used to generate prompts for a generative model (e.g., a large language model) that may adjust what data to obtain and / or generate for output and / or what computing device to utilize to perform output associated with a user query. In some implementations, the generative model may be fine-tuned (e.g., parameter-efficient fine-tuning and / or soft prompt tuning) for the multi-device framework, which may include tuning the generative model to handle prompts with environmental context and generating output based on both the query and the environmental context.

[0043] Specifically, different computing devices may have different components to capture different forms of input data (e.g., text input, voice commands, gesture input, etc.) and / or provide different types of output data to the user (e.g., visual display, audio playback, etc.). The environment multi-device framework can utilize capability information associated with computing devices in the environment to provide an immersive and multifaceted computing system that can obtain input and provide output using multiple different computing systems within the environment.

[0044] Smartphones, smart watches, smart speakers, smart TVs, smart assistant devices, and / or other computing devices are always around users; however, interconnectivity may be limited and the highest quality input and / or output data may not be efficiently utilized. For example, a user may enter a query via their smartphone, which may result in video and / or audio being provided on the smartphone in response, although a high-quality smart TV and / or smart speaker may be easily accessible and close to the user. Whether the content is entertaining, educational, or other in nature, a multi-device framework may be utilized to obtain additional forms of input and / or provide additional forms of output that may be of higher quality than a single-device system.

[0045] The systems and methods disclosed herein can be used to manage obtaining input from multiple devices and / or providing output via multiple devices. In some implementations, input can be received from smartphones and smart watches, and output responsive to the input can be provided via smart TVs and smart speakers. Additionally and / or optionally, the system and method can determine that a particular computing device within an environment has the highest processing power, and then the system and method can utilize the particular computing device to perform model reasoning. In some implementations, processing tasks can be split between multiple computing devices within an environment and / or performed by a server computing system.

[0046] The system and method of the present disclosure provide a variety of technical effects and benefits. As an example, the system and method can be used to provide an interconnected multi-device ecosystem. Specifically, the system and method disclosed herein can obtain input data from one or more devices within the ecosystem. The system and method can obtain environmental data describing the device within the environment, and then can generate prompts based on the input data and the environmental data, which can be processed with a generative model to generate a response to the input data generated based on the device in the environment. The response can then be sent to one or more specific devices within the environment. For example, text input can be obtained via a tablet computer, and output can be provided via a smart speaker.

[0047] Another example technical effect and benefit relates to improved computing efficiency and improved operation of computing systems. For example, a technical benefit of the systems and methods of the present disclosure is the ability to reduce the computing resources required to interact with multiple computing devices within an environment. Specifically, the multi-device framework can centralize the processing and flow of data, thereby reducing the duplication of processing instances between devices within the ecosystem.

[0048] The systems and methods of the present disclosure provide a variety of technical effects and benefits. As an example, the systems and methods may provide an interface generation system. The interface generation system may be used to generate environment-specific and device-specific interfaces that may be specifically generated based on the input capabilities and output capabilities available within a multi-device ecosystem.

[0049] Another technical benefit of the system and method of the present disclosure is the ability to provide an immersive multi-device environment using interface generation. Specifically, an interface can be generated and provided to provide a user with the ability to obtain multiple different input types from multiple different devices in an interconnected system and provide multiple different output types via multiple different devices in the interconnected system.

[0050] Referring now to the drawings, example embodiments of the present disclosure will be discussed in further detail.

[0051] Figure 1 A block diagram of an example generative model system 10 is depicted in accordance with an example embodiment of the present disclosure. In some implementations, the generative model system 10 is configured to receive and / or obtain a set of input data 12 describing a query and / or context, and as a result of receiving the input data 12, generate, determine, and / or provide a model-generated output 20 describing a response to the query. Thus, in some implementations, the generative model system 10 may include a generative model 18 operable to process a prompt 16 to generate a response to the query, the response being configured to be provided by a particular computing device in the environment.

[0052] Specifically, the generative model system 10 may obtain input data 12 and environmental data 14. The input data 12 may describe one or more inputs. The input data 12 may describe queries (e.g., "Who is playing today?", "How do I make focaccia?", "What was the song that played at the end of the movie of the year?"). The input data 12 may include directly entered input (e.g., a user typing in a graphical keyboard, capturing an image, selecting a graphical user interface element, providing a voice command, etc.) and / or contextual input (e.g., user search history, time of day, user habit data, user browsing history, application activity, currently opened applications, temperature, user location, and / or data obtained from other computing devices within the environment).

[0053] Environmental data 14 may describe multiple computing devices within an environment. The environment may be a room, a suite of rooms, proximity to a user, and / or a space including multiple computing devices in proximity to each other. Environmental data 14 may include specification information for each of the multiple computing devices within the environment. Alternatively and / or in addition, environmental data 14 may be descriptive identification data of the multiple computing devices (e.g., device names, device serial numbers, device tags, registration data, etc.), capability data of the multiple computing devices (e.g., input capabilities of the multiple computing devices, processing capabilities of the multiple computing devices, and / or output capabilities of the multiple computing devices), and / or other environmental data.

[0054] The generative model system 10 may process the input data 12 and the environmental data 14 to generate a prompt 16. The prompt 16 may include information associated with the query and a plurality of computing devices. The prompt 16 may include the query and an indicator of one or more candidate computing devices for providing an output. The prompt 16 may be generated by querying a prompt library based on the environmental data 14 to determine a prompt template. The prompt template may then be filled in based on the input data 12. Alternatively and / or additionally, the prompt 16 may be generated by processing the input data 12 and the environmental data 14 with a prompt generation model. The prompt generation model may include a machine learning model, which may include a generative model (e.g., an autoregressive language model, a diffusion model, a visual language model, and / or other generative models).

[0055] The generative model 18 may then process the prompt 16 to generate a model-generated output 20. The generative model 18 may include a generative language model (e.g., a large language model, a visual language model, and / or other language models), an image generation model (e.g., a text generation image generation model), and / or other generative models. The generative model 18 may include a transformer model, a convolutional model, a feedforward model, a recurrent model, a self-attention model, and / or other models.

[0056] The model-generated output 20 may include a predicted response to a query of the input data 12. The model-generated output 20 may be a novel generated output including multiple predicted features (e.g., predicted text characters, predicted audio signals, predicted pixels, etc.). The model-generated output 20 may be generated to be provided by a specific computing device among multiple computing devices. For example, the model-generated output 20 may be generated to have a specific size, quality, and / or content type based on a preferred computing device (having specific capabilities) and / or generative model 18 processing as determined based on a prompt.

[0057] The model-generated output 20 may then be provided to a particular computing device within the environment. Providing the model-generated output 20 to the particular computing device may include sending data, which may include generating application programming interface calls with the generative model and executing the API calls with one or more application programming interfaces. The particular computing device may then provide the model-generated output 20 to a user (e.g., via a visual display, a speaker, and / or other output component of the particular computing device).

[0058] Figure 2 A block diagram of an example response generation system 200 is depicted according to an example embodiment of the present disclosure. The response generation system 200 is similar to Figure 1 The generative model system 10 is different in that the response generation system 200 further includes a prompt library 224 and a search engine 226.

[0059] Specifically, the response generation system 200 may obtain input data 212 and environmental data 214. The input data 212 may include text data, image data, audio data, embedded data, signal data, search history data, browsing history data, application interaction data, latent coding data, multimodal data, global data, and / or other data. The input data 212 may describe one or more inputs, which may include obtaining input from multiple computing devices within the environment. The input data 212 may describe a query (e.g., "Who is playing today?", "How do I make focaccia?", "What was the song that played at the end of the movie of the year?"). The input data 212 may include directly entered input (e.g., a user typing in a graphical keyboard, capturing an image, selecting a graphical user interface element, providing a voice command, etc.) and / or contextual input (e.g., user search history, time of day, user habit data, user browsing history, application activity, currently open applications, temperature, user location, and / or data obtained from other computing devices within the environment). The input data 212 may include contextual data determined to be relevant to the query. Alternatively and / or additionally, the input data 12 may include a predictive query that may be generated based on a prediction of what the user may be interested in based on one or more determined user contexts (e.g., a determined interest in a particular football team, a series of searches may be utilized to determine a possible interest in horror movies, and / or a determination of a possible interest in purchasing a particular product based on current purchases of similar products). In some implementations, the response generation system 200 may suggest content to the user by generating the input data based on a personalized signal and / or a contextualized signal.

[0060] The environment data 214 may describe the plurality of computing devices 222 within the environment. The environment may be a room, a set of rooms, proximity to a user, and / or a space including a plurality of computing devices in close proximity to each other. The environment data 214 may include specification information for each of the plurality of computing devices 222 within the environment. Optionally and / or additionally, the environment data 214 may be descriptive identification data of the plurality of computing devices 222 (e.g., device name, device serial number, device tag, registration data, etc.), capability data of the plurality of computing devices 222 (e.g., input capabilities of the plurality of computing devices 222, processing capabilities of the plurality of computing devices 222, and / or output capabilities of the plurality of computing devices 222), and / or other environment data. The plurality of computing devices 222 may include smartphones, smart wearable devices (e.g., smart watches and / or smart glasses), smart speakers, smart TVs, laptop computers, desktop computers, smart home appliances (e.g., smart refrigerators, smart washing machines, smart dryers, and / or smart dispensers), virtual assistant devices (e.g., smart home panels, room-based assistants, and / or other assistant devices), tablet computers, and / or other computing devices. The plurality of computing devices 222 may include mobile computing devices and / or fixed computing devices. In some implementations, the plurality of computing devices 222 may include devices that are proximate to the user, devices registered with the user's mobile device, devices registered with the user's profile, devices connected to a particular Internet hub, devices registered with a particular assistant device, and / or devices having other associations.

[0061] The response generation system 200 may process the input data 212 and / or the environment data 214 to generate a prompt 216. The prompt 216 may include information associated with the query and a plurality of computing devices. The prompt 216 may include the query and an indicator of one or more candidate computing devices for providing an output. The prompt 216 may be generated by querying a prompt library 224 based on the environment data 214 to determine a prompt template. The query prompt library 224 may include determining a prompt template associated with an environment configuration associated with the user's environment. Optionally and / or additionally, the query prompt library 224 may include generating an embedding based on the environment data 214, and then determining a prompt template associated with the embedding based on a nearest neighbor search. The prompt template may then be filled in based on the input data 212. Optionally and / or additionally, the prompt 216 may be generated by processing the input data 212 and the environment data 214 with a prompt generation model. The prompt generation model may include a machine learning model, which may include a generative model (e.g., an autoregressive language model, a diffusion model, a visual language model, and / or other generative models). The prompt 216 may include text data, image data, audio data, potentially encoded data, multimodal data, embedded data, and / or other data. The prompt 216 may include a hard prompt (e.g., a text string and / or other data input) and / or a soft prompt (e.g., a learned parameter set). In some implementations, the prompt 216 may be a zero shot prompt and / or a few shot prompt. The prompt 216 may divide the user request into multiple tasks, which may be provided as multiple prompts.

[0062] The generative model 218 can then process the prompt 216 to generate a model-generated output 220. The generative model 218 can include a generative language model (e.g., a large language model, a visual language model, and / or other language models), an image generation model (e.g., a text generation image generation model), and / or other generative models. The generative model 218 can include a transformer model, a convolutional model, a feed-forward model, a recurrent model, a self-attention model, and / or other models.

[0063] In some implementations, the generative model 218 may be communicatively connected to one or more processing engines (e.g., a search engine, a rendering engine, and / or other engines) and / or one or more machine learning models (e.g., a classification model, an enhancement model, a segmentation model, a detection model, an embedding model, and / or other models). For example, the generative model 218 may communicate back and forth with the search engine 226 to obtain search results based on the input data 212, the environment data 214, and / or reasoning. The search engine 226 may output search results, which may then be processed by the generative model 218 to generate a summary of the search results and / or determine a response to the query based on information from the search results, which may then be utilized to generate the model-generated output 220. The model-generated output 220 may include text data, image data, audio data, latently encoded data, multimodal data, and / or other data.

[0064] The model-generated output 220 may include a predicted response to a query of the input data 212. The model-generated output 220 may be a novel generated output including a plurality of predicted features (e.g., predicted text characters, predicted audio signals, predicted pixels, etc.). The model-generated output 220 may be generated to be provided by a particular computing device among a plurality of computing devices. For example, the model-generated output 220 may be generated to have a particular size, quality, and / or content type based on a preferred computing device (having particular capabilities) as determined based on a prompt and / or generative model 218 processing.

[0065] Generative model 218 can process prompts 216 to generate output device instructions 230. Output device instructions 230 can include instructions for sending model-generated output 220 to a particular computing device 228 in plurality of computing devices 222. Output device instructions 230 can include application programming interface calls, registry files, and / or instructions for a particular computing device 228.

[0066] The model-generated output 220 may then be provided to a particular computing device within the environment based on and / or with the output device instructions 230. Providing the model-generated output 220 to the particular computing device may include sending data, which may include generating application programming interface calls with the generative model and executing API calls with one or more application programming interfaces. The particular computing device may then provide the model-generated output 220 to a user (e.g., via a visual display, a speaker, and / or other output component of the particular computing device).

[0067] Figure 3 A flowchart of an example method performed according to an example embodiment of the present disclosure is depicted. Figure 3The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The steps of method 300 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0068] At 302, a computing system may obtain input data. The input data may include a query associated with a particular user. The input data may include text data, audio data, image data, gesture data, latent coding data, multimodal data, and / or other data. The input data may be obtained from one or more computing devices in an environment and / or generated using the one or more computing devices. The input data may include prompts for a generative model. In some implementations, the input data may describe a question associated with one or more topics. The question may be associated with content viewed on one or more computing devices and / or features in the environment.

[0069] At 304, the computing system may obtain environmental data. The environmental data may describe multiple computing devices in the environment of the user. The multiple computing devices may be associated with multiple different output components. In some implementations, the multiple computing devices may be located close to each other computing device within the multiple computing devices. The multiple computing devices may be connected via network communication. In some implementations, the environmental data may describe the specifications of the multiple computing devices. For example, the environmental data may indicate components of the computing device that may include input components and / or output components. Additionally and / or optionally, the environmental data may include information describing the input capabilities and / or output capabilities of multiple different computing devices.

[0070] At 306, the computing system may generate a prompt based on the input data and the environmental data. The prompt may include data describing the query and device information associated with at least a subset of the plurality of computing devices. The prompt may include text data, image data, audio data, embedded data, statistics, graphical representation data, latent encoding data, semantic data, multimodal data, and / or other data. The prompt may be generated using a machine learning model based on deterministic functions, heuristics, and / or hybrid methods. The prompt may be generated by querying a prompt template library based on the environmental data and filling in a selected prompt template based on the input data.

[0071] In some implementations, generating a prompt based on the input data and the environment data may include: determining an environment-specific device configuration based on the environment data; obtaining a prompt template from a prompt library based on the environment-specific device configuration; and enhancing the prompt template based on the input data to generate the prompt. The environment-specific device configuration may describe a respective output type and a respective output quality of a plurality of computing devices. The prompt library may include a plurality of different prompt templates associated with a plurality of different device configurations.

[0072] At 308, the computing system may process the prompt with a generative model to generate a model-generated output. The model-generated output may include a response to the query. In some implementations, the model-generated output may be generated to be provided by a specific computing device among a plurality of computing devices. The model-generated output may be generated to be provided by a specific output component among a plurality of different output components. The generative model may generate output device instructions. The output device instructions may describe a specific computing device among a plurality of computing devices for providing the model-generated output. A specific computing device may be associated with a specific output component. For example, the model-generated output may be generated to be provided to a user via a corresponding output component of a specific computing device (e.g., a speaker of a smart surround system and / or a display screen of a smart television).

[0073] In some implementations, the generative model may be communicatively connected to the search engine via an application programming interface. Processing the prompt with the generative model to generate the model-generated output may include: generating an application programming interface call based on the prompt; determining a plurality of search results with the search engine based on the application programming interface call; and processing the plurality of search results with the generative model to generate the model-generated output.

[0074] At 310, the computing system may send the model-generated output to a specific computing device. The model-generated output may be sent to the specific computing device based on the output device instructions. Sending the model-generated output to the specific computing device may include executing an application programming interface call generated by the generative model. The model-generated output may be sent to a virtual assistant device in a user environment, and then the virtual assistant device may control the specific computing device to provide the model-generated output to the specific user.

[0075] In some implementations, multiple computing devices may be connected via a cloud computing system. Each of the multiple computing devices may be registered with a platform of the cloud computing system. Environmental data may be obtained using the cloud computing system. Model-generated output may be sent via the cloud computing system. Alternatively and / or additionally, the multiple computing devices may be communicatively connected via a local network. A particular computing device among the multiple computing devices may facilitate the acquisition of input data and the transmission of model-generated output.

[0076] In some implementations, processing a prompt with a generative model to generate a model-generated output may include generating a plurality of model outputs. The plurality of model outputs may include a plurality of candidate responses. Additionally and / or optionally, sending the model-generated output to a particular computing device may include: sending a first model output of the plurality of model outputs to a first computing device of the plurality of computing devices; and sending a second model output of the plurality of model outputs to a second computing device of the plurality of computing devices. The first model output may include visual data for display via a visual display. The second model output may include audio data for playback via a speaker assembly. In some implementations, the first computing device may include a smart TV. Additionally and / or optionally, the second computing device may include a smart speaker.

[0077] Figure 4 A block diagram of an example multi-device management system 400 is depicted according to an example embodiment of the present disclosure. Specifically, the multi-device management system 400 may include a computing system including multiple user computing devices and a server computing system 420. The multiple user computing devices and the server computing system 420 may be communicatively connected via a network 410.

[0078] The plurality of user computing devices may include a first computing device 402, a second computing device 404, a third computing device 406, and / or an nth computing device 408. The plurality of user computing devices may include a plurality of different computing devices, which may have different input, processing, and / or output capabilities. For example, the first computing device 402 may include a smartphone with an image sensor, an audio sensor, a touch sensor, a motion sensor, a speaker, a visual display, a tactile component, and / or a light. The second computing device 404 may include a smart wearable device (e.g., a smart watch), which may include a biometric sensor, a motion sensor, a touch sensor, a visual display, and / or a tactile component. The third computing device 406 may include a smart speaker, which may include a high-quality speaker and / or a Bluetooth transmitter. The nth computing device 408 may include a smart TV, which may include an infrared sensor receiver, a transmitter-receiver, a speaker, a visual display, and / or multiple input ports. The plurality of user computing devices may be connected to the network 410 via Ethernet, WiFi, and / or a Bluetooth connection with a companion device.

[0079] Multiple user computing devices may be associated with an environment based on device registration, location, and / or proximity.Environmental data may be generated based on multiple user computing devices, which may include obtaining signals from the multiple user computing devices.

[0080] The server computing system 420 may obtain input data and / or environmental data from multiple user computing devices via the network 410. The server computing system 420 may include multiple processing services for processing input data and / or environmental data. For example, the server computing system 420 may include one or more generative models 422, one or more search engines 424, one or more prompt generation models 426, one or more interface models 428, and / or one or more other models. One or more generative models 422 may be configured, trained, and / or tuned to process prompts associated with input data and / or environmental data to generate model-generated outputs that respond to input data and are configured to have specific content types based on environmental data. One or more search engines 424 may be communicatively connected to obtain search results that can be used to understand input data and / or environmental data and / or to respond to input data and / or environmental data. One or more prompt generation models 426 may be configured, trained, and / or tuned to process input data and / or environmental data, thereby generating prompts for one or more generative models 422. One or more interface models 428 may be configured, trained, and / or tuned to process environmental data associated with multiple user computing devices and generate multiple corresponding interfaces for the multiple user computing devices. The multiple corresponding interfaces may be generated based on the input and / or output capabilities of the devices within the environment.

[0081] Figure 5 A block diagram of an example ambient personalization system 500 is depicted according to an example embodiment of the present disclosure. Specifically, the ambient personalization system 500 can utilize information from multiple applications and / or platforms to provide personalized responses and / or personalized experiences. For example, data from a search assistant 502, a document assistant 504, an operating system assistant 506, a video player assistant 508, a browser assistant 510, and / or a chat interface assistant 512 is used to determine user context and / or generate queries and / or suggestions.

[0082] Search assistant 502 can be associated with a search application (and / or platform).Search assistant 502 can obtain current queries, session status, search history, trend data, and / or other search data.

[0083] Document assistant 504 can be associated with one or more document applications. Document assistant 504 can obtain data associated with the current document being viewed and / or edited, driver data (eg, stored document information), sharing permission data, and / or other document data.

[0084] Operating system assistant 506 can be associated with an operating system of one or more computing devices. Operating system assistant 506 can obtain data associated with content currently being provided for display, application data (e.g., app deep links, application programming interfaces (APIs), and / or usage data), and / or other operational data.

[0085] The video player assistant 508 can be associated with the video player application (and / or platform). The video player assistant 508 can be used to obtain video data of the currently displayed video, video saves, viewing history, concerned media providers, subscriptions, comment history and / or other video player data.

[0086] The browser assistant 510 can be associated with the browser application. The browser assistant 510 can obtain data associated with the current page, bookmark data, tab data, browsing history data and / or other browser data.

[0087] The chat interface assistant 512 may be associated with one or more chatbots. The chat interface assistant may obtain session history data, response history, input history, topics, links, and / or other chatbot data.

[0088] The multi-device management system 514 may obtain data from multiple applications and / or platforms and provide the data to one or more other systems, which may include a core model 516 , a grounded service model 518 , and / or a personalized model 520 .

[0089] For example, the core model 516 can be utilized for summarization, planning and reasoning and / or function calls. The grounded service model 518 can be utilized to utilize a library of tools, utilize an external connector application programming interface (API), access search results, and / or access context and / or memory. The core model 516 and / or the grounded service model 518 can interact with one or more other models and / or services via one or more cloud APIs. The output of the core model 516 and / or the grounded service model 518 can be provided back to the multi-device management system 514 and then to the personalized model 520.

[0090] The personalization model 520 can process the data to generate a personalized output that can include a prediction prompt (and / or a suggestion prompt). The personalized output can be a prompt response enhanced based on user data, interactions, and / or devices associated with user preferences.

[0091] Fig. 6AAn illustration of an example interface according to an example embodiment of the present disclosure is depicted. Specifically, the systems and methods disclosed herein may be utilized to generate and process environment data associated with a computing device to generate multiple corresponding interfaces for multiple computing devices within the environment. The multiple corresponding interfaces may take into account multiple other computing devices and may be confused and / or mixed with pre-existing interfaces. Fig. 6A Three example interfaces are depicted that may be associated with different devices in an environment and / or may be associated with different environments.

[0092] For example, first interface 602 may be associated with a first computing device (e.g., a mobile device) within an environment, second interface 604 may be associated with a second computing device (e.g., a smart watch) within the environment, and third interface 606 may be associated with a third computing device (e.g., a smart refrigerator) within the environment. Alternatively and / or additionally, the same computing device may have different interfaces based on being in different environments and / or based on different user contexts.

[0093] Figure 6B An illustration of an example image capture entry point according to an example embodiment of the present disclosure is depicted. Specifically, an entry point to a search interface and / or assistant interface utilizing multiple device input and / or output may be provided via a user interface element provided based on context.

[0094] For example, a user may be capturing an image 610 (e.g., capturing an image of a refrigerator). The image may be processed to determine (and / or identify) an object in the image (e.g., classify the object as an X brand, Y model refrigerator). Selectable user interface elements may then be provided in a viewfinder 612. The image and / or identification may then be processed to generate a response, which may include search results 614 associated with objects similar to the identified object. Other options for interacting with the response may also be provided. Other options may include an augmented reality experience, which may include rendering the object into an image of the user's home 616.

[0095] Figure 6C A diagram of an example smart TV entry point is depicted according to an example embodiment of the present disclosure. Specifically, an entry point may include a search entry point indicator that provides suggestions on one device and may be interacted with on another device.

[0096] For example, content provided for display on a first computing device 620 (e.g., a smart TV) may be determined to include features that may be of interest to the user when searching. Thus, a suggested entry point user interface element may be rendered over the content. The user may then select a selectable user interface element 622 on a second computing device (e.g., a mobile computing device) to view the model-generated output 624. The model-generated output may be provided via the first computing device 620, the second computing device, and / or the third computing device. The model-generated output 624 may include search results, generative model outputs, one or more renderings, one or more suggestions, and / or other options.

[0097] Fig.6D Depicted is an illustration of an example calendar interface according to an example embodiment of the present disclosure. Specifically, Fig.6D An example calendar interface is depicted that includes a search window that can be adjusted based on calendar data. For example, the search window can include a text input field and two suggested actions. The suggested actions can include finding a conference room 630 and / or suggesting setting aside time for focus time 632. The suggestions can be generated based on information from multiple applications and / or multiple computing devices. The search window can be provided on a device where the calendar application is open and / or can be provided on another device.

[0098] Fig. 6E A diagram of an example video conferencing interface 634 is depicted according to an example embodiment of the present disclosure. Specifically, Fig. 6E An example video conferencing interface 634 is depicted that includes a search window 636 that can be adjusted based on the meeting data. For example, the search window 636 can include a text input field and one or more suggested actions. The suggested actions can include taking a meeting record (e.g., transcribing the meeting and / or opening a recording application), setting a reminder, rescheduling the meeting, and / or obtaining a recording of a similar meeting. The suggestions can be generated based on information from multiple applications and / or multiple computing devices. The search window 636 can be provided on the device where the video conferencing application is open and / or can be provided on another device.

[0099] Fig. 6F Depicted is a diagram of an example email interface according to an example embodiment of the present disclosure. Specifically, Fig. 6FAn example email interface 638 is depicted that includes a search window 640 that can be adjusted based on the email data and / or the meeting data. For example, the search window 640 can include a text input field and one or more suggested actions. The suggested actions can include summarizing the meeting minutes and / or the manuscript, scheduling another meeting, obtaining information about the meeting topic, drafting a follow-up email based on the context associated with the meeting minutes, and / or drafting an email based on the email data and / or the meeting data. The suggestions can be generated based on information from multiple applications and / or multiple computing devices. The search window 640 can be provided on a device where the email application is open and / or can be provided on another device.

[0100] Figure 6G Depicted is a diagram of an example video player interface according to an example embodiment of the present disclosure. Specifically, Figure 6G An example video player interface 650 is depicted, which includes a search window 654 that can be adjusted based on video data and / or viewing history data. For example, the search window 654 may include a text input field and one or more suggested actions. The suggested actions may include obtaining a product list of similar products shown in the displayed video 652, summarizing the video, obtaining entity tags for the displayed video, finding similar videos, and / or obtaining additional information associated with the displayed video 652. For example, a user may request further information about the dress shown in the displayed video 652. One or more frames may be segmented from the video and searched, which may include frame cropping. Optionally and / or additionally, entity tags associated with the displayed frames may be obtained and searched. Search results may be provided to be displayed in a search window 654 superimposed on the displayed video. Search results may include product lists, web links, and / or other data. Suggestions may be generated based on information from multiple applications and / or multiple computing devices. The search window 654 may be provided on a device where the video player application is open and / or may be provided on another device.

[0101] Figure 7 A flowchart of an example method performed according to an example embodiment of the present disclosure is depicted. Figure 7 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The steps of method 700 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0102] At 702, the computing system may obtain input data. The input data may include a query associated with a particular user. The query may include a question associated with one or more topics, and the query may request a response to the question. The input data may include a voice command obtained via a microphone, a text string entered via a graphical keyboard, and / or a gesture obtained via a camera, an inertial measurement unit, and / or a touch sensor. In some implementations, the input data may include data describing input obtained from multiple computing devices within the environment (e.g., a voice command obtained via a microphone on a smartphone, a gesture obtained via a touch sensor of a smart watch, image data from a smart refrigerator, and / or a viewing history obtained from a smart TV).

[0103] At 704, the computing system may obtain environmental data. The environmental data may describe multiple computing devices in the user's environment. Multiple computing devices may be associated with multiple different output components. Multiple different output components may be associated with multiple corresponding output capabilities associated with multiple computing devices. In some implementations, each of the multiple corresponding output capabilities may describe the output type and quality available via the corresponding computing device. Multiple output components may include a speaker associated with a first device and a visual display associated with a second device. The environmental data may include registration data associated with a computing device, a WiFi router, a virtual assistant device, a user computing device, and / or a user profile registered for the environment. In some implementations, the environmental data may include an output hierarchy for multiple candidate output types, which may include a hierarchical representation of the performance capabilities of multiple computing devices for multiple different output types (e.g., visual display, audio output, tactile feedback, etc.).

[0104] At 706, the computing system may generate a prompt based on the input data and the environmental data. The prompt may include data describing the query and device information associated with at least a subset of the plurality of computing devices. The prompt may be generated by processing the input data and the environmental data with a prompt generation model. The prompt generation model may include a language model (e.g., a generative language model (e.g., a large language model)). The prompt generation model may be trained and / or tuned to generate prompts based on understanding the intent of the query and determining an output type associated with the intent and / or an output type available based on the environmental data. The prompt generation model may generate a prompt embedding to regulate output generation of the generative model.

[0105] In some implementations, the computing system may determine that a particular output component is associated with the intent of the query. A prompt may be generated based on the particular output component being associated with the intent of the query (e.g., a request for a song may be associated with a speaker output component, while a request to play a video may be associated with a visual display of a television).

[0106] Additionally and / or alternatively, the computing system may determine, based on the environment data, an output hierarchy based on specification information for a plurality of different output components. A prompt may be generated based on the output hierarchy and the query. For example, the prompt may include text and / or embedding that adjusts output generation based on output capabilities of computing devices within the environment.

[0107] At 708, the computing system may process the prompt with a generative model to generate a model-generated output and an output device instruction. The model-generated output may include a response to the query. In some implementations, the model-generated output may be generated to be provided by a specific output component among a plurality of different output components. The output device instruction may describe a specific computing device among a plurality of computing devices to provide the model-generated output. The specific computing device may be associated with a specific output component. In some implementations, the output device instruction may include an application programming interface call to send the model-generated output to a specific computing device.

[0108] At 710, the computing system may send the model-generated output to a specific computing device based on the output device instructions. In some implementations, the transmission may be performed via signal transmission over a network. Alternatively and / or additionally, a notification may be provided to the input computing device, indicating that the output configured for another device is available, and then the user may interact with the notification before the model-generated output is sent to the specific computing device. In some implementations, the generative model may generate multiple model-generated outputs, and the computing system may send the multiple model-generated outputs to multiple different computing devices within the environment (e.g., a slideshow may be sent to a smart TV for playback, a text document may be sent to an e-reader or personal computing device (e.g., a smartphone or tablet), an audio file may be sent to a smart speaker, and / or a scheduled set of color and brightness instructions may be sent to an RGB smart light fixture).

[0109] Figure 8 A flowchart of an example method performed according to an example embodiment of the present disclosure is depicted. Figure 8 The steps performed in a specific order are depicted for the purpose of illustration and discussion, but the method of the present disclosure is not limited to the specific illustrated order or arrangement. The steps of method 800 may be omitted, rearranged, combined and / or adapted in various ways without departing from the scope of the present disclosure.

[0110] At 802, a computing system may obtain environment data. The environment data may describe multiple computing devices within an environment associated with a particular user. The environment data may describe specification information for the multiple computing devices. In some implementations, the multiple computing devices may be determined based on device registration for a particular network, device registration for a particular user device, device registration for a particular user profile, proximity to a user device, and / or based on signal exchanges between the device and the user device. The user device may be a smartphone, a tablet computer, a smart watch, smart glasses, and / or other computing devices.

[0111] At 804, the computing system may process the environmental data to determine a plurality of corresponding input capabilities and a plurality of corresponding output capabilities associated with a plurality of computing devices. A plurality of corresponding input capabilities may be associated with candidate input types associated with a plurality of computing devices. In some implementations, a plurality of corresponding output capabilities may be associated with candidate output types associated with a plurality of computing devices. A plurality of input capabilities and / or a plurality of output capabilities may be determined based on specification information (and / or component information) associated with a plurality of computing devices. In some implementations, a plurality of input capabilities and / or a plurality of output capabilities may be determined based on one or more searches, based on heuristics, and / or based on one or more other determination techniques using a machine learning model. A plurality of input capabilities may describe available input types and a quality range associated with obtaining and / or generating the input types (e.g., a decibel range of a microphone and / or a resolution of a camera). A plurality of output capabilities may describe available output types and a quality range associated with providing the output type (e.g., audio quality (e.g., volume range, frequency range, etc.)).

[0112] At 806, the computing system may generate a plurality of corresponding interfaces for a plurality of computing devices based on a plurality of corresponding input capabilities and a plurality of corresponding output capabilities. Based on the plurality of computing devices being based on a plurality of corresponding input capabilities and a plurality of corresponding output capabilities, the plurality of corresponding interfaces may be specific to the plurality of computing devices. In some implementations, the plurality of corresponding interfaces may include a plurality of device indicators indicating a plurality of computing devices within an environment associated with a particular user. The plurality of computing devices may be configured as a user-specific device ecosystem that is communicatively connected to receive input and provide output. Each of the plurality of corresponding interfaces may be configured to receive a particular input type and provide a particular output type based on the respective input capabilities and respective output capabilities of a particular computing device in the plurality of computing devices.

[0113] At 808, the computing system may provide a plurality of corresponding interfaces to a plurality of computing devices. In response to interface generation and / or in response to a user interacting with a particular computing device, a plurality of corresponding interfaces may be sent to the plurality of computing devices. The plurality of corresponding interfaces may be stored by the server computing system and may be provided to the computing devices when used. Alternatively and / or additionally, the corresponding interfaces for the corresponding computing devices may be downloaded locally and may be provided during offline and online states.

[0114] In some implementations, the computing system may obtain user input via a first interface of a first computing device among the plurality of computing devices. The computing system may process the user input with a search engine to determine a plurality of search results, process the plurality of search results with a generative model to generate a model output, and provide the model output for display via a second interface of a second computing device among the plurality of computing devices.

[0115] There will be more and more users with multiple devices, which may or may not share operating systems, apps, and / or platforms. Base models (e.g., large base models, which may include generative models) can be used as the primary technology by which users interact with their devices. The system can provide a wide range of new experiences enabled by these base models being able to regulate both their input and output across all devices in an integrated manner.

[0116] A multi-device framework utilizing the base model can be used for a number of different companion tasks, which may include output regulation, interface generation, input acquisition facilitation, and / or other tasks.

[0117] For example, a multi-device framework can be used for an ambient agent companion with a fluid interaction mechanism that depends on the device and the surface. A user may have several devices that may use a disjointed ecosystem in the presence of an ambient LLM companion. An agent (e.g., a computing system including a multi-device framework) may include a base model (e.g., a base model with a generative model and / or a prompt generation model) that adapts the input / output interaction model based on the details of the device. For example, a speaker may have a voice-only interface as its primary interaction model, a device with a non-touch screen may have a hybrid text / voice interface, and / or a mobile phone and / or a smart wearable device (e.g., a smart watch, a smart jacket, and / or smart glasses) may have a specially generated user interface based on its form factor.

[0118] The interface may be generated to blend in with the existing machine surface (e.g., the user interface may be minimized to one or more app elements (e.g., the agent interface may become text and / or animation within a specific app interface)). The generated interface may have multi-device / machine surface awareness elements that may highlight which of the devices the ambient companion is actively listening to and / or monitoring based on the user's proximity to the device. When interacting with the companion on one device requires context from other devices and / or requires actions to be performed on other devices, the user interface may include user interface elements to show that these other devices are being utilized and / or taken into account.

[0119] Additionally and / or optionally, the systems and methods may adjust the ambient companion model output based on device form factor and characteristics. For example, a user in an environment with a smart TV and smart home devices (e.g., a virtual assistant device) may provide input (e.g., via a voice command) that issues a query (e.g., a query similar to "what movies were nominated for the Oscars for their best songs?"). A search engine / assistant / companion that supports LLM may obtain additional input to the query, where the additional input may describe the details of the device (e.g., input specifying that the environment includes a screen and speakers and their specific parameters). Higher quality smart speakers may be provided as part of an LLM prompt. Based on the prompt, a generative model (e.g., an LLM) may generate a response to a smart speaker (e.g., "I'll list out the movies and play samples of their award-winning soundtracks on your speaker"). At the same time, another user who issued the same query may be in an environment with the same smart TV, but with lower-quality speakers, so the user may receive the answer in the form of a visual display via the smart TV (for example, the response may include "I'll play parts of the trailers for those movies on your TV. You'll hear fragments of their award winning sound-tracks in these. (I'll play parts of the trailers for those movies on your TV. You'll hear fragments of their award winning sound-tracks in these.)").

[0120] In some implementations, the system and method may obtain and / or determine the availability and context of nearby devices, which may then be used to modify actions performed on behalf of the user. For example, a user may be in an environment with multiple Internet of Things (IoT) devices, mobile computing devices (e.g., smartphones, tablets, smart watches, etc.), cars, and / or laptops with an environmental companion. The user may then ask the companion (via the input sensors of the computing device) about traveling to a nearby recreational area. The environmental companion may then obtain and / or determine input in a manner that takes into account all devices in the environment. Input determination and / or providing a response may include responding to the query across more than one device. For example, the companion may identify typical routes and / or destinations, and may generate certain visualizations that may be provided within a map application of one or more specific computing devices. A car companion variant of the agent may perform local reasoning and may determine that the car will need to be recharged to travel to one or more destinations. On a map app used by the main agent companion, the query may be processed to output relevant charging stations highlighted on the illustrated route.

[0121] In some implementations, the systems and methods disclosed herein can be used to generate and / or provide interfaces across different computing devices that can be interconnected and / or have similar styles, layouts, and / or semantics, regardless of the manufacturer and / or operating system of the computing device. For example, the systems and methods can generate interfaces that obfuscate differences between native interfaces of different computing devices from different manufacturers and / or operating systems.

[0122] The systems and methods may include a fluid cross-device representation. A cross-device representation may be provided based on the output of a base model that can regulate input / output behavior across multiple devices and machine surfaces (e.g., apps). The cross-device representation may be achieved via: (1) a centralized base model running in the cloud that the device nodes report directly to; (2) a hybrid model where the centralized base model works with a decentralized local large model that can perform reasoning using device-specific context and coordinate with the main model to solve higher-level tasks; and / or (3) a decentralized architecture where each device has its companion version and the user who owns them all places them together in the same physical space or in the same logical unit. A certain level of reasoning (e.g., >10B parameters) may be required so that the models can interact with each other.

[0123] The hybrid approach may be implemented via a variety of different configurations, which may include adaptations to centralized and / or decentralized architectures.

[0124] For example, a user may have several connected devices registered. The characteristics of these devices may be known and / or determined. The degree of interoperability may vary and may include an API into which these devices may be fully controlled if plugged in. The device speakers may utilize the API to output audio. The API may be used to directly render elements on the operating system (and / or complex functionality within an app, etc.).

[0125] The API may appear in various configurations based on whether there is a local base model. If no base model exists, the primitive functions may be described in some accessible document library, and / or, alternatively, a natural language interface may be available if a local model exists and is provided. The device may also provide functionality to interface with other external systems (e.g., sensors that can be used to read ambient temperature, operate blinds, and / or autonomously navigate through the home to perform certain tasks). API examples and information may be fed into the base model, which drives the environmental companion agent in a manner that can be used to regulate response generation and task solving on behalf of the user.

[0126] Hint generation may include obtaining and / or generating zero-shot or few-shot hints for the base model to process to understand how to use the device's API. If not sufficient, the device may have a small dataset associated with it (e.g., about 1000 examples that may be used for hint tuning and / or weighting of the base model to understand how to operate the device). Examples may include Task->Use API for decomposition.

[0127] Once surfaced, the devices can be grouped together through a network, in which each device is a node. The edges of the network can be persistent (e.g., always online) and / or can have some associated weight that quantifies how related two devices are at a particular time. For example, two devices may be very close to each other, and the weight between the two devices can quantify this information by having a smaller / larger value. Two devices may also share some context (e.g., if they interoperate and / or display the same app (e.g., state can also be dynamically encoded in the edges of the graph through messaging)).

[0128] Available devices can be queried at inference time. The query can be fed as a list with metadata and / or can be represented by a graph network, which can also be fed into the environment companion base model at inference time. Prior to model inference, the base model may have been fine-tuned to work with the device network topology and / or features. The graph network can be a modality that the base model can operate with. In some implementations, the network can be serialized directly and can be passed to a generative model (e.g., LLM) as part of the prompt. If it is not enough, a large base model that drives the environment companion can be fine-tuned using multi-device graph network examples (e.g., there are also 1000 or more examples, each of which will have a certain task -> a step-by-step decomposition of how to use multiple devices to better solve the task).

[0129] In some implementations, the systems and methods may include prompting the LLM (or other generative model) with examples (e.g., “[device context][response][metadata: This response is appropriate for a smart speaker at breakfast time]” and / or “[device context][response][metadata: The response can be visualized by rendering a UI on the phone with three checkboxes for each answer]”). For the device decomposition example, the prompt may include “[device context][response][metadata: This response should be delivered to the smart speaker and a small notification should be displayed on the phone with a summary, as the user may not be next to the speaker]”. The user may then face one of their devices and may decide to interact with the ambient companion.

[0130] Input can be highly dependent on the details of the form factor. For example, when the user pulls out their phone, the companion can be activated in a voice-only manner. The companion can allow a keyboard input UI to render itself when the phone is locked or when the user unlocks their phone. Modality dependent on device state can be adjusted according to the context of the user's relationship to nearby devices. For example, if the user's watch is available, voice input can be activated there instead.

[0131] The query issued can be solved with the help of all devices. For example, if the user issues a statement of the type "I should go to bed soon, when do I need to wake up tomorrow?", the system can trigger the base model to use all the contexts available in the device and take actions based on the query and / or context. Based on the determined action, the response can first process the machine interface that can access the work and / or schedule context and respond with some actual suggested time on the speaker.

[0132] Processing may continue, for example in a smart home, the lighting system may determine the user's intent to fall asleep, and may activate its local reasoning companion to set the lighting to slowly adjust according to the user's determined bedtime routine. The information may be conveyed visually and may be rendered as a UI element on the user's phone and / or watch so that the user is aware of the decision.

[0133] There may be an active component to the ambient LLM companion. For example, it may interact with some devices to determine whether user input is needed. For example, a companion-enabled device with sensors and autonomous device operation may determine to receive input and / or generate prompts based on the user's context.

[0134] In some implementations, the car companion may be built in. The car companion may be prompted, configured, and / or trained to schedule fifteen minutes before the inferred user departure time for the user to check temperature, rain conditions, etc., and in response to the prompt, configuration, and / or training, the car companion may notify the user to perform an action (e.g., pick up X) and / or not to forget a raincoat.

[0135] The determined context may be received by the ambient companion, and a decision may be made thereon as to the mechanism by which the context is communicated to the user. For example, the companion may decide to use a smart speaker (e.g., an audio notification: "it's raining and your car can pick you up instead of having to walk to the parking lot"). Alternatively and / or additionally, the system may render the same information and / or action on the user's watch based on the device usage context.

[0136] Across all devices, there may be a uniform behavior in terms of input and / or output that implements brand elements. The uniform behavior may be achievable through device-specific prompts and available few-sample examples. The device may be equipped with five voice samples containing ten seconds of audio. The device may also be equipped with five UI examples of how the assistant may be rendered on that specific device. The context-based model may be adjusted based on these examples to generate responses and generate a uniform output of an interface from that manufacturer that the user may be familiar with.

[0137] A hybrid framework approach may include devices that have some degree of autonomy but can generally interoperate through a centralized partner. A variation may include devices that do not have the ability to run their local base model and then all decisions may be made by a centralized partner. A variation approach may include continuous streaming of information. Another variation may include devices that have complete autonomy and no central partner. In an approach without a central partner, the network backbone may be more important and may be based on simpler signals to guide which devices are interconnected at a particular time to solve a given query.

[0138] The systems and methods disclosed herein can use generative artificial intelligence models to connect fragmented systems and unlock entirely new value for users by providing a single service level companion.

[0139] The capabilities of a conversational interface (chatbot) can go beyond the questions and answers (Q&A) associated with a single device and / or application. For example, a user may be watching a shopping blog and say "show me the reviews for the products recommended by this article / video". The system may need to understand what the product is, get reviews from the Shopping Graph, and summarize the reviews in an easy-to-compare way. In another example, a user may be watching a recipe and say "add the ingredients to my Shopping App basket". The system may then need to extract the ingredients and call the Shopping App API. In another example, a user may be watching a travel video blog and say "show the places mentioned in this article / video on a Map". The system may need to understand the places mentioned, extract the addresses, and call an API to create a pin on a custom map app. In another example, while viewing a documents app, a user may be writing an essay about Abraham Lincoln by asking, “write a 1000 word essay about Lincoln, focusing on the civilwar, to explain his challenges in a way that a 5th grader would understand.” Therefore, the system may need to understand Lincoln’s key facts, a list of his books, the contents of those books, historian articles, and summarize all of this information. In another example, while viewing a maps app, a user may be looking at a map at a zoom level and say, “show me a video of things to do around here.” The system may need to understand the key things to do in the location, the places mentioned in the video, and retrieve the correct video.In another example, while viewing content on a phone, a user may be on a screen of a trail hiking app and say "show me restaurants near the trailhead." Thus, the system may need to understand the location of the trailhead on the screen being viewed and may fetch local results for restaurants near that location.

[0140] The underlying dependencies may be that chatbots on these products, no matter how they are integrated, may rely on common elements such as reasoning, grounding, retrieval, function calls, user state, personal preferences, etc., many of which come from search and / or knowledge graph services.

[0141] The systems and methods disclosed herein may utilize chatbot large language models (LLMs), search understanding large language models and search engines, cloud service large language models, and / or one or more other systems that support models. The architectures of chatbot large language models (LLMs) and search understanding large language models and search engines may look the same, but they can be implemented independently, with differences in each block, such as different RLHF (reinforcement learning from human feedback) training, different planning algorithms, different third-party plug-in sets, different search backend interfaces, and / or other differences. Different pipelines and / or systems can exchange information for reasoning, understanding, and / or context determination.

[0142] Separate architectures allow products to be developed independently and iteratively. However, while following best practices ensures the success of each product, failure to adopt a holistic, service-oriented approach can result in products that are ultimately disconnected.

[0143] Figure 5 Depicted is an example architecture for building a layer cake of achievable chatbot capabilities, along with an instantiated set of chatbot services that give the appropriate context and personality that may be relevant to the product in which the service is instantiated.

[0144] For example, there may be a common chatbot service that is built on a powerful LLM, fine-tuned for common tasks (such as instruction following), has access to a search backend, has access to 1p / 3p APIs and has the ability to call functions, etc. Examples of some services that can be leveraged to obtain input and / or determine context may include a search platform (e.g., by accessing user search history to obtain concise and real data), a document application (e.g., by accessing document files, emails, the current mail / file being viewed, favorites, etc. to obtain detailed and creative data), and / or a browser application (e.g., by accessing the current tab, other tabs, bookmarks, history, etc. to obtain concise and real data).

[0145] In some implementations, the systems and methods may include fine-tuning the underlying model differently for different products using use case-specific examples and content.

[0146] The systems and methods disclosed herein may include a companion model (eg, a base model that may include a prompt generation model and / or a generative model).

[0147] By integrating AI and LLM capabilities into the user interface (UI) as independent entities (separate from any other user activities), the system and method can ensure persistent availability for immediate access by the user whenever they are needed. This integration can allow AI and LLM to remain open and seamlessly connect with user operations as the user transitions between various applications and tasks.

[0148] This configuration can be helpful for users who want to quickly access information or perform tasks without leaving the app they are currently using. Additionally, a persistent system can be helpful for users who want to use Companion in conjunction with other products. For example, a user can use Companion to search for information about a topic while they are also using a map app for navigation.

[0149] The system and method may provide two additional benefits. First, the system may circumvent the need to build on existing solutions with potentially incompatible architectures, thereby avoiding potential integration challenges. Second, not changing existing solutions that users are already accustomed to may minimize disruption caused by introducing major changes that may not be optimal.

[0150] If a user is looking at a product in a store, the user can use LLM to get information about the product (e.g., price, reviews, and specifications). The user can also use the camera to take a picture of the product and then use LLM to search for similar products online. The multi-faceted system can allow the user to get all the information the user needs to know about the product without leaving the store.

[0151] Another example could be if a user is looking at a painting in a museum, the user could use the LLM to get information about the artist, the painting, and the history of the painting. The user could also use the camera to take a picture of the painting, and then use the LLM to search for other paintings by the same artist or paintings with similar themes. This system could allow the user to learn more about the artwork you are looking at, without having to rely on the information provided by the museum.

[0152] In some implementations, a user may be looking for a product but may not be able to find what they are looking for. The user can then open their phone and describe the product they are looking for, and the phone will search for and find the product.

[0153] The system and method can take advantage of the ambient ecosystem by allowing users to connect their other devices as companions. The connection can allow users to have a more seamless and integrated experience across their devices. For example, a user can start a task on their phone and then continue it on their laptop without having to re-enter any information. The ambient ecosystem can create a more personalized and more convenient experience for the user.

[0154] Fig.9A A block diagram of an example computing system 100 that performs multi-device output management according to an example embodiment of the present disclosure is depicted. System 100 includes a user computing system 102, a server computing system 130, and / or a third computing system 150 that are communicatively coupled via a network 180.

[0155] The user computing system 102 may include any type of computing device, such as, for example, a personal computing device (e.g., a laptop or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0156] The user computing system 102 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 114 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 may store data 116 and instructions 118 executed by the processor 112 to cause the user computing system 102 to perform operations.

[0157] In some implementations, the user computing system 102 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or may otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and / or linear models. The neural network may include a feedforward neural network, a recurrent neural network (e.g., a long short-term memory recurrent neural network), a convolutional neural network, or other forms of neural networks.

[0158] In some implementations, one or more machine learning models 120 may be received from the server computing system 130 via the network 180, stored in the user computing device memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing system 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., to perform parallel machine learning model processing across multiple instances of input data and / or detected features).

[0159] More specifically, the one or more machine learning models 120 may include one or more detection models, one or more classification models, one or more segmentation models, one or more enhancement models, one or more generative models, one or more natural language processing models, one or more optical character recognition models, and / or one or more other machine learning models. The one or more machine learning models 120 may include one or more Transformer models. The one or more machine learning models 120 may include one or more neural radiation field models, one or more diffusion models, and / or one or more autoregressive language models.

[0160] One or more machine learning models 120 may be used to detect one or more object features. The detected object features may be classified and / or embedded. The classification and / or embedding may then be used to perform a search to determine one or more search results. Alternatively and / or additionally, one or more detected features may be used to determine that an indicator (e.g., a user interface element indicating the detected feature) is to be provided to indicate that the feature has been detected. The user may then select the indicator to perform feature classification, embedding, and / or searching. In some implementations, the classification, embedding, and / or searching may be performed before selecting the indicator.

[0161] In some implementations, the one or more machine learning models 120 may process image data, text data, audio data, and / or potentially coded data to generate output data, which may include image data, text data, audio data, and / or potentially coded data. The one or more machine learning models 120 may perform optical character recognition, natural language processing, image classification, object classification, text classification, audio classification, context determination, action prediction, image correction, image enhancement, text enhancement, sentiment analysis, object detection, error detection, restoration, video stabilization, audio correction, audio enhancement, and / or data segmentation (e.g., mask-based segmentation).

[0162] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by a server computing system 130 that communicates with the user computing system 102 according to a client-server relationship. For example, the machine learning model 140 may be implemented by the server computing system 130 as part of a web service (e.g., a viewfinder service, a visual search service, an image processing service, an environmental computing service, and / or an overlay application service). Thus, one or more models 120 may be stored and implemented at the user computing system 102, and / or one or more models 140 may be stored and implemented at the server computing system 130.

[0163] The user computing system 102 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or a touchpad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, or other devices by which a user can provide user input.

[0164] In some implementations, the user computing system may store and / or provide one or more user interfaces 124, which may be associated with one or more applications. One or more user interfaces 124 may be configured to receive input and / or provide data for display (e.g., image data, text data, audio data, one or more user interface elements, augmented reality experiences, virtual reality experiences, and / or other data for display). The user interface 124 may be associated with one or more other computing systems (e.g., server computing system 130 and / or third-party computing system 150). The user interface 124 may include a viewfinder interface, a search interface, a generative model interface, a social media interface, and / or a media content gallery interface.

[0165] The user computing system 102 may include one or more sensors 126 and / or receive data from the one or more sensors. The one or more sensors 126 may be housed in a housing assembly that houses one or more processors 112, memory 114, and / or one or more hardware components that may store one or more software packages and / or cause the execution of one or more software packages. The one or more sensors 126 may include one or more image sensors (e.g., cameras), one or more lidar sensors, one or more audio sensors (e.g., microphones), one or more inertial sensors (e.g., inertial measurement units), one or more biosensors (e.g., heart rate sensors, pulse sensors, retinal sensors, and / or fingerprint sensors), one or more infrared sensors, one or more position sensors (e.g., GPS), one or more touch sensors (e.g., conductive touch sensors and / or mechanical touch sensors), and / or one or more other sensors. The one or more sensors may be utilized to obtain data associated with the user's environment (e.g., images of the user's environment, records of the environment, and / or the user's location).

[0166] The user computing system 102 may include and / or be part of a user computing device 104. The user computing device 104 may include a mobile computing device (e.g., a smartphone or tablet), a desktop computer, a laptop computer, a smart wearable device, and / or a smart appliance. Additionally and / or optionally, the user computing system may obtain data from one or more user computing devices 104 and / or generate data using the one or more user computing devices. For example, a smartphone camera may be used to capture image data describing an environment, and / or an overlay application of the user computing device 104 may be used to track and / or process data provided to the user. Similarly, one or more sensors associated with a smart wearable device may be used to obtain data about the user and / or about the user's environment (e.g., a camera housed in the user's smart glasses may be used to obtain image data). Additionally and / or optionally, data may be obtained and uploaded from other user devices that may be specifically used for data acquisition or generation.

[0167] The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 134 may store data 136 and instructions 138 that are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0168] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances where the server computing system 130 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0169] As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Fig. 9B Discuss example model 140 .

[0170] Additionally and / or alternatively, the server computing system 130 may include and / or be communicatively connected to a search engine 142 that can be used to crawl one or more databases (and / or resources). The search engine 142 can process data from the user computing system 102, the server computing system 130, and / or the third-party computing system 150 to determine one or more search results associated with the input data. The search engine 142 can perform a term-based search, a tag-based search, a Boolean-based search, an image search, an embedding-based search (e.g., a nearest neighbor search), a multimodal search, and / or one or more other search techniques.

[0171] The server computing system 130 may store and / or provide one or more user interfaces 144 for obtaining input data and / or providing output data to one or more users. The one or more user interfaces 144 may include one or more user interface elements, which may include input fields, navigation tools, content widgets, selectable tiles, widgets, data display carousels, dynamic animations, information pop-ups, image enhancements, text-to-speech, speech-to-text, augmented reality, virtual reality, feedback loops, and / or other interface elements.

[0172] User computing system 102 and / or server computing system 130 may train models 120 and / or 140 via interaction with a third-party computing system 150 communicatively coupled via network 180. Third-party computing system 150 may be separate from server computing system 130 or may be part of server computing system 130. Alternatively and / or additionally, third-party computing system 150 may be associated with one or more web resources, one or more web platforms, one or more other users, and / or one or more contexts.

[0173] The third-party computing system 150 may include one or more processors 152 and a memory 154. The one or more processors 152 may be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and may be one processor or a plurality of processors operatively connected. The memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 may store data 156 and instructions 158 executed by the processor 152 to cause the third-party computing system 150 to perform operations. In some implementations, the third-party computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0174] The network 180 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or some combination thereof, and may include any number of wired or wireless links. In general, communications over the network 180 may be conducted via any type of wired and / or wireless connection using a variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0175] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0176] In some implementations, the input to the machine learning model of the present disclosure may be image data. The machine learning model may process the image data to generate an output. As an example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of the image data, potential embedding of the image data, encoded representation of the image data, hash of the image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process the image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change to the image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded and / or compressed representation of the image data, etc.). As another example, the machine learning model may process the image data to generate an amplified image data output. As another example, the machine learning model may process the image data to generate a prediction output.

[0177] In some implementations, the input to the machine learning model of the present disclosure may be text or natural language data. The machine learning model may process the text or natural language data to generate an output. As an example, the machine learning model may process the natural language data to generate a language encoding output. As another example, the machine learning model may process the text or natural language data to generate a potential text embedding output. As another example, the machine learning model may process the text or natural language data to generate a translation output. As another example, the machine learning model may process the text or natural language data to generate a classification output. As another example, the machine learning model may process the text or natural language data to generate a text segmentation output. As another example, the machine learning model may process the text or natural language data to generate a semantic intent output. As another example, the machine learning model may process the text or natural language data to generate an amplified text or natural language output (e.g., text or natural language data with higher quality than the input text or natural language, etc.). As another example, the machine learning model may process the text or natural language data to generate a prediction output.

[0178] In some implementations, the input to the machine learning model of the present disclosure may be speech data. The machine learning model may process the speech data to generate an output. As an example, the machine learning model may process the speech data to generate a speech recognition output. As another example, the machine learning model may process the speech data to generate a speech translation output. As another example, the machine learning model may process the speech data to generate a potential embedding output. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded and / or compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an amplified speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a text representation output (e.g., a text representation of the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a predicted output.

[0179] In some implementations, the input to the machine learning model of the present disclosure may be sensor data. The machine learning model may process the sensor data to generate an output. As an example, the machine learning model may process the sensor data to generate a recognition output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a classification output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a segmentation output. As another example, the machine learning model may process the sensor data to generate a visualization output. As another example, the machine learning model may process the sensor data to generate a diagnostic output. As another example, the machine learning model may process the sensor data to generate a detection output.

[0180] In some cases, the input includes visual data, and the task is a computer vision task. In some cases, the input includes pixel data for one or more images, and the task is an image processing task. For example, the image processing task may be image classification, where the output is a set of scores, each score corresponding to a different object class and representing the possibility that one or more images depict an object belonging to the object class. The image processing task may be object detection, where the image processing output identifies one or more regions in one or more images, and for each region, identifies the possibility that the region depicts an object of interest. As another example, the image processing task may be image segmentation, where the image processing output defines, for each pixel in one or more images, a corresponding possibility for each category in a predetermined set of categories. For example, the set of categories may be foreground and background. As another example, the set of categories may be object classes. As another example, the image processing task may be depth estimation, where the image processing output defines a corresponding depth value for each pixel in one or more images. As another example, the image processing task may be motion estimation, where the network input includes multiple images, and the image processing output defines, for each pixel of one of the input images, the motion of the scene depicted at the pixel between images in the network input.

[0181] The user computing system may include multiple applications (e.g., applications 1 to N). Each application may include its own corresponding machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc.

[0182] Each application can communicate with multiple other components of the computing device (such as, for example, one or more sensors, a context manager, a device state component, and / or additional components). In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is specific to the application.

[0183] The user computing system 102 may include multiple applications (e.g., applications 1 to N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, etc. In some implementations, each application may communicate with the central intelligence layer (and the models stored therein) using an API (e.g., a public API across all applications).

[0184] The central intelligence layer may include multiple machine learning models. For example, a corresponding machine learning model (e.g., model) may be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer may provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of computing system 100.

[0185] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized data repository for the computing system 100. The central device data layer can communicate with many other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0186] Fig. 9B A block diagram of an example computing system 50 for performing multi-device output management according to an example embodiment of the present disclosure is depicted. Specifically, the example computing system 50 may include one or more computing devices 52, which may be used to obtain and / or generate one or more data sets, which may be processed by the sensor processing system 60 and / or the output determination system 80 for feedback to a user, who may provide information about features in one or more obtained data sets. The one or more data sets may include image data, text data, audio data, multimodal data, potentially encoded data, etc. The one or more data sets may be obtained via one or more sensors associated with the one or more computing devices 52 (e.g., one or more sensors in the computing devices 52). Additionally and / or optionally, the one or more data sets may be stored data and / or retrieved data (e.g., data retrieved from a web resource). For example, a user may interact with images, text, and / or other content items. Interaction with the content items may then be used to generate one or more determinations.

[0187] One or more computing devices 52 may obtain and / or generate one or more data sets based on image capture, sensor tracking, data storage retrieval, content download (e.g., downloading images or other content items from web resources via the Internet), and / or via one or more other technologies. One or more data sets may be processed using a sensor processing system 60. The sensor processing system 60 may perform one or more processing techniques using one or more machine learning models, one or more search engines, and / or one or more other processing techniques. One or more processing techniques may be performed in any combination and / or individually. One or more processing techniques may be performed serially and / or in parallel. Specifically, one or more data sets may be processed using a context determination block 62, which may determine a context associated with one or more content items. The context determination block 62 may identify and / or process metadata, user profile data (e.g., preferences, user search history, user browsing history, user purchase history, and / or user input data), previous interaction data, global trend data, location data, time data, and / or other data to determine a specific context associated with a user. A context may be associated with an event, a determined trend, a particular action, a particular type of data, a particular environment, and / or another context associated with a user and / or retrieved or obtained data.

[0188] The sensor processing system 60 may include an image pre-processing block 64. The image pre-processing block 64 may be used to adjust one or more values ​​of an acquired and / or received image to prepare the image to be processed by one or more machine learning models and / or one or more search engines 74. The image pre-processing block 64 may resize the image, adjust saturation values, adjust resolution, strip and / or add metadata, and / or perform one or more other operations.

[0189] In some implementations, the sensor processing system 60 may include one or more machine learning models, which may include a detection model 66, a segmentation model 68, a classification model 70, an embedding model 72, and / or one or more other machine learning models. For example, the sensor processing system 60 may include one or more detection models 66 that may be used to detect specific features in a processed dataset. Specifically, one or more detection models 66 may be used to process one or more images to generate one or more bounding boxes associated with detected features in the one or more images.

[0190] Additionally and / or alternatively, one or more segmentation models 68 may be utilized to segment one or more portions of a data set from one or more data sets. For example, one or more segmentation models 68 may utilize one or more segmentation masks (e.g., one or more segmentation masks generated manually and / or based on one or more bounding boxes) to segment a portion of an image, a portion of an audio file, and / or a portion of text. Segmentation may include isolating one or more detected objects and / or removing one or more detected objects from an image.

[0191] One or more classification models 70 may be used to process image data, text data, audio data, potentially encoded data, multimodal data, and / or other data to generate one or more classifications. One or more classification models 70 may include one or more image classification models, one or more object classification models, one or more text classification models, one or more audio classification models, and / or one or more other classification models. One or more classification models 70 may process data to determine one or more classifications.

[0192] In some implementations, one or more embedding models 72 may be used to process data to generate one or more embeddings. For example, one or more embedding models 72 may be used to process one or more images to generate one or more image embeddings in an embedding space. The one or more image embeddings may be associated with one or more image features of the one or more images. In some implementations, the one or more embedding models 72 may be configured to process multimodal data to generate multimodal embeddings. The one or more embeddings may be used to classify, search, and / or learn an embedding space distribution.

[0193] The sensor processing system 60 may include one or more search engines 74 that can be used to perform one or more searches. The one or more search engines 74 can crawl one or more databases (e.g., one or more local databases, one or more global databases, one or more private databases, one or more public databases, one or more dedicated databases, and / or one or more general databases) to determine one or more search results. The one or more search engines 74 can perform feature matching, text-based searches, embedding-based searches (e.g., k-nearest neighbor searches), metadata-based searches, multimodal searches, web resource searches, image searches, text searches, and / or application searches.

[0194] Additionally and / or alternatively, the sensor processing system 60 may include one or more multimodal processing blocks 76 that may be used to assist in processing multimodal data. The one or more multimodal processing blocks 76 may include generating multimodal queries and / or multimodal embeddings for processing by one or more machine learning models and / or one or more search engines 74.

[0195] The output of the sensor processing system 60 may then be processed using an output determination system 80 to determine one or more outputs to be provided to the user. The output determination system 80 may include a heuristic-based determination, a machine learning model-based determination, a user selection-based determination, and / or a context-based determination.

[0196] The output determination system 80 may determine how and / or where to provide one or more search results in the search results interface 82. Additionally and / or optionally, the output determination system 80 may determine how and / or where to provide one or more machine learning model outputs in the machine learning model output interface 84. In some implementations, one or more search results and / or one or more machine learning model outputs may be provided for display via one or more user interface elements. One or more user interface elements may be superimposed on the displayed data. For example, one or more detection indicators may be superimposed on a detected object in a viewfinder. One or more user interface elements may be selected to perform one or more additional searches and / or one or more additional machine learning model processes. In some implementations, user interface elements may be provided as dedicated user interface elements for a specific application and / or may be provided uniformly across different applications. One or more user interface elements may include pop-up displays, interface overlays, interface tiles and / or small pieces, carousel interfaces, audio feedback, animations, interactive widgets, and / or other user interface elements.

[0197] Additionally and / or alternatively, data associated with the output of the sensor processing system 60 may be utilized to generate and / or provide an augmented reality experience and / or a virtual reality experience 86. For example, one or more acquired data sets may be processed to generate one or more augmented reality rendering assets and / or one or more virtual reality rendering assets, which may then be used to provide an augmented reality experience and / or a virtual reality experience 86 to a user. The augmented reality experience may render information associated with an environment into the corresponding environment. Alternatively and / or additionally, objects associated with the processed data sets may be rendered into the user environment and / or the virtual environment. Rendering data set generation may include training one or more neural radiance field models to learn a three-dimensional representation of one or more objects.

[0198] In some implementations, one or more action prompts 88 may be determined based on the output of the sensor processing system 60. For example, a search prompt, a purchase prompt, a generate prompt, a reservation prompt, a call prompt, a redirect prompt, and / or one or more other prompts may be determined to be associated with the output of the sensor processing system 60. The one or more action prompts 88 may then be provided to the user via one or more selectable user interface elements. In response to selection of the one or more selectable user interface elements, a corresponding action of the corresponding action prompt may be performed (e.g., a search may be performed, a purchase application programming interface may be utilized, and / or another application may be opened).

[0199] In some implementations, one or more generative models 90 may be used to process one or more data sets and / or output of the sensor processing system 60 to generate model-generated content items that may then be provided to a user. The generation may be prompted based on a user selection and / or may be performed automatically (e.g., automatically based on one or more conditions, which may be associated with a threshold amount of search results that are not represented).

[0200] One or more generative models 90 may include a language model (e.g., a large language model and / or a visual language model), an image generation model (e.g., a text generation image generation model and / or an image enhancement model), an audio generation model, a video generation model, a graphics generation model, and / or other data generation models (e.g., other content generation models). One or more generative models 90 may include one or more Transformer models, one or more convolutional neural networks, one or more recurrent neural networks, one or more feedforward neural networks, one or more generative adversarial networks, one or more self-attention models, one or more embedding models, one or more encoders, one or more decoders, and / or one or more other models. In some implementations, one or more generative models 90 may include one or more autoregressive models (e.g., a machine learning model trained to generate predicted values ​​based on prior behavior data) and / or one or more diffusion models (e.g., a machine learning model trained to generate predicted data based on generating and processing distribution data associated with input data).

[0201] One or more generative models 90 may be trained to process input data and generate model-generated content items, which may include a plurality of predicted words, pixels, signals, and / or other data. Model-generated content items may include novel content items that are different from any existing work. One or more generative models 90 may utilize learned representations, sequences, and / or probability distributions to generate content items, which may include phrases, storylines, settings, objects, characters, beats, lyrics, and / or other aspects not included in existing content items.

[0202] The one or more generative models 90 may include a visual language model. The visual language model may be trained, tuned, and / or configured to process image data and / or text data to generate natural language output. The visual language model may utilize a pre-trained large language model (e.g., a large autoregressive language model) and one or more encoders (e.g., one or more image encoders and / or one or more text encoders) to provide detailed natural language output that simulates natural language written by humans.

[0203] The visual language model can be used for zero-shot image classification, few-shot image classification, image captioning, multimodal query refinement, multimodal question answering, and / or can be tuned and / or trained for multiple different tasks. The visual language model can perform visual question answering, image caption generation, feature detection (e.g., content monitoring (e.g., for inappropriate content)), object detection, scene recognition, and / or other tasks.

[0204] The visual language model can utilize a pre-trained language model and then tune it to achieve multimodality. The training and / or tuning of the visual language model may include image-text matching, masked language modeling, multimodal fusion of cross attention, contrastive learning, prefix language model training, and / or other training techniques. For example, the visual language model can be trained to process images to generate predicted text similar to real-value text data (e.g., real-value captions of images). In some implementations, the visual language model can be trained to replace masked tokens of a natural language template with text tokens describing features depicted in an input image. Optionally and / or additionally, training, tuning, and / or model reasoning may include multi-layer connections of visual and text embedding features. In some implementations, the visual language model may be trained and / or tuned via joint learning of image embedding and text embedding generation, which may include training and / or tuning the system to map embedding to a joint feature embedding space that maps text features and image features to a shared embedding space. Joint training may include parallel embedding of image-text pairs and / or may include triplet training. In some implementations, images can be utilized and / or processed as prefixes to a language model.

[0205] The output determination system 80 may use the data enhancement block 92 to process one or more data sets and / or the output of the sensor processing system 60 to generate enhanced data. For example, the data enhancement block 92 may be used to process one or more images to generate one or more enhanced images. Data enhancement may include data correction, data cropping, removal of one or more features, addition of one or more features, resolution adjustment, lighting adjustment, saturation adjustment, and / or other enhancements.

[0206] In some implementations, one or more data sets and / or outputs of the sensor processing system 60 may be stored based on the determination of the data storage block 94 .

[0207] The output of the output determination system 80 may then be provided to the user via one or more output components of the user computing device 52. For example, one or more user interface elements associated with the one or more outputs may be provided for display via a visual display of the user computing device 52.

[0208] The process may be performed iteratively and / or continuously.One or more user inputs to the provided user interface elements may adjust and / or affect the continuous processing loop.

[0209] The technology discussed herein relates to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a variety of possible configurations, combinations, and partitions of tasks and functions between and among components. For example, the processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system, or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0210] Although the subject matter has been described in detail with respect to various specific example embodiments of the subject matter, each example is provided by way of explanation, not limitation of the present disclosure. Those skilled in the art, after understanding the foregoing, may easily produce changes, modifications, and equivalents of such embodiments. Therefore, the present disclosure does not exclude such modifications, changes, and / or additions to the subject matter that would be readily understood by those of ordinary skill in the art. For example, a feature shown or described as part of one embodiment may be used together with another embodiment to produce a further yet another embodiment. Therefore, the present disclosure is intended to cover such changes, modifications, and equivalents.

Claims

1. A computing system for determining an output device for providing a query response, the system comprising: one or more processors; as well as one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations comprising: obtaining input data, wherein the input data includes a query associated with a particular user; obtaining environmental data, wherein the environmental data describes a plurality of computing devices in an environment of the user, wherein the plurality of computing devices are associated with a plurality of different output components; generating a prompt based on the input data and the environmental data, wherein the prompt includes data describing the query and device information associated with at least a subset of the plurality of computing devices; processing the prompt with a generative model to generate a model-generated output, wherein the model-generated output comprises a response to the query, wherein the model-generated output is generated to be provided by a particular computing device among the plurality of computing devices; and An output generated by the model is sent to the particular computing device.

2. The system of claim 1, wherein the output generated by the model is generated to be provided by a specific output component of the plurality of different output components; wherein the generative model generates output device instructions, wherein the output device instructions describe a particular computing device among the plurality of computing devices for providing an output generated by the model, wherein the particular computing device is associated with the particular output component; and Wherein the output generated by the model is sent to the specific computing device based on the output device instructions.

3. The system of claim 1 , wherein processing the prompt with the generative model to generate the model-generated output comprises: generating a plurality of model outputs, wherein the plurality of model outputs comprises a plurality of candidate responses; and Wherein sending the output generated by the model to the specific computing device comprises: sending a first model output of the plurality of model outputs to a first computing device of the plurality of computing devices; as well as A second model output of the plurality of model outputs is sent to a second computing device of the plurality of computing devices.

4. The system of claim 3, wherein the first model output comprises visual data for display via a visual display, and wherein the second model output comprises audio data for playback via a speaker assembly.

5. The system of claim 4, wherein the first computing device comprises a smart television, and wherein the second computing device comprises a smart speaker.

6. The system of claim 1 , wherein generating the prompt based on the input data and the environmental data comprises: determining an environment-specific device configuration based on the environment data; obtaining a prompt template from a prompt library based on the environment-specific device configuration; as well as The prompt template is enhanced based on the input data to generate the prompt.

7. The system of claim 6, wherein the environment-specific device configurations describe respective output types and respective output qualities of the plurality of computing devices, and wherein the prompt library comprises a plurality of different prompt templates associated with a plurality of different device configurations.

8. The system of claim 1, wherein the plurality of computing devices are connected via a cloud computing system, wherein each of the plurality of computing devices is registered with a platform of the cloud computing system, wherein the environmental data is obtained using the cloud computing system, and wherein the output generated by the model is sent via the cloud computing system.

9. The system of claim 1, wherein the plurality of computing devices are located proximate to each other computing device within the plurality of computing devices, wherein the plurality of computing devices are communicatively connected via a local network, and wherein particular computing devices within the plurality of computing devices facilitate transmission of input data acquisition and model-generated output.

10. The system of claim 1, wherein the generative model is communicatively coupled to a search engine via an application programming interface, and wherein processing the prompt with the generative model to generate the model-generated output comprises: generating an application programming interface call based on the prompt; determining a plurality of search results with the search engine based on the application programming interface call; as well as The plurality of search results are processed with the generative model to generate an output generated by the model.

11. A computer-implemented method, the method comprising: obtaining, by a computing system including one or more processors, input data, wherein the input data includes a query associated with a particular user; obtaining, by the computing system, environmental data, wherein the environmental data describes a plurality of computing devices in an environment of the user, wherein the plurality of computing devices are associated with a plurality of different output components; generating, by the computing system, a prompt based on the input data and the environmental data, wherein the prompt includes data describing the query and device information associated with at least a subset of the plurality of computing devices; processing, by the computing system, the prompt with a generative model to generate a model-generated output and output device instructions, wherein the model-generated output comprises a response to the query, wherein the model-generated output is generated to be provided by a particular output component of the plurality of different output components, and wherein the output device instructions describe a particular computing device of the plurality of computing devices to provide the model-generated output, wherein the particular computing device is associated with the particular output component; as well as Output generated by the model is sent, by the computing system, to the particular computing device based on the output device instructions.

12. A method as claimed in claim 11, wherein the plurality of different output components are associated with a plurality of corresponding output capabilities associated with the plurality of computing devices, and wherein each of the plurality of corresponding output capabilities describes a type and quality of output available via a corresponding computing device.

13. The method of claim 11, wherein the output device instructions include application programming interface calls to send output generated by the model to the particular computing device.

14. The method of claim 11, wherein the plurality of different output components comprises a speaker associated with the first device and a visual display associated with the second device.

15. The method of claim 11, further comprising: determining, by the computing system, that the particular output component is associated with the intent of the query; and Wherein the prompt is generated based on associating the particular output component with the intent of the query.

16. The method of claim 11, further comprising: determining, by the computing system and based on the environmental data, an output hierarchy based on specification information of the plurality of different output components; and Wherein the prompt is generated based on the output hierarchy and the query.

17. One or more non-transitory computer-readable media collectively storing instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations comprising: obtaining environmental data, wherein the environmental data describes a plurality of computing devices within an environment associated with a particular user; processing the environmental data to determine a plurality of corresponding input capabilities and a plurality of corresponding output capabilities associated with the plurality of computing devices, wherein the plurality of corresponding input capabilities are associated with candidate input types associated with the plurality of computing devices, and wherein the plurality of corresponding output capabilities are associated with candidate output types associated with the plurality of computing devices; generating a plurality of respective interfaces for the plurality of computing devices based on the plurality of respective input capabilities and the plurality of respective output capabilities, wherein the plurality of respective interfaces are specific to the plurality of computing devices based on the plurality of respective input capabilities and the plurality of respective output capabilities; as well as The plurality of respective interfaces are provided to the plurality of computing devices.

18. One or more non-transitory computer-readable media as described in claim 17, wherein the plurality of corresponding interfaces include a plurality of device indicators indicating the plurality of computing devices within the environment associated with the particular user, and wherein the plurality of computing devices are configured as a user-specific device ecosystem that is communicatively connected to receive input and provide output.

19. One or more non-transitory computer-readable media as described in claim 17, wherein each of the plurality of corresponding interfaces is configured to receive a specific input type and provide a specific output type based on the corresponding input capabilities and the corresponding output capabilities of the particular computing device among the plurality of computing devices.

20. The one or more non-transitory computer-readable media of claim 17, wherein the operations further comprise: obtaining user input via a first interface of a first computing device among the plurality of computing devices; processing the user input with a search engine to determine a plurality of search results; processing the plurality of search results with a generative model to generate a model output; as well as The model output is provided for display via a second interface of a second computing device of the plurality of computing devices.

Images