Providing a composite graphical assistant interface for controlling various connected devices

Abstract

To provide composite graphical assistant interfaces for controlling various connected devices.SOLUTION: Composite graphical assistant interfaces are created in response to a user providing a request for an automated assistant to cause a connected device to execute a specific function. In response to the automated assistant receiving the request, the automated assistant can specify other connected devices, and other functions can be executed by the other connected devices. The other functions can then be mapped to various graphical control elements in order to provide a composite graphical assistant interface from which the user can interact with different connected devices. Each graphical control element can be arranged to reflect how each connected device operates simultaneous with the presentation of the composite graphical assistant interface.SELECTED DRAWING: Figure 3A

Description

[Technical field]

[0001] Providing a composite graphical assistant interface for controlling various connected devices. [Background technology]

[0002] Humans may engage in human-to-computer dialogues using conversational software applications referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "conversational personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, humans (who may be referred to as "users" when interacting with an automated assistant) may provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances) and / or by providing textual (e.g., typed) natural language input, which may in some cases be converted to text and then processed.

[0003] Some automated assistants may be used to control Internet of Things ("IoT") devices. However, many IoT devices may come with their own corresponding applications. As a result, a user who has collected several different IoT devices may be tasked with installing various third-party applications on their personal computing devices. Having various similar applications on a single device may be inefficient, since applications may generally require memory, network bandwidth, updates, and / or other resources. A user may rely on an automated assistant to mediate interactions between the user and a particular IoT device, but such interactions may often require audio-to-text conversion of spoken utterances from the user. To accomplish the audio-to-text conversion, the audio data often must be sent over a network, thereby consuming significant network bandwidth. Furthermore, when a user is surrounded by people who are conversing or is otherwise in an environment with background noise, mediating interactions in such a manner may prove unreliable. As a result, a dialog session between a user and an automated assistant can waste computational and network resources when requests to various IoT devices to perform specific functions are not conclusively identified or otherwise fulfilled. Summary of the Invention [Means for solving the problem]

[0004] The present disclosure generally relates to methods, apparatus, and (transient and non-transient) computer-readable media for providing a composite graphical assistant interface through which multiple different devices are controlled. The composite graphical assistant interface may be used to control a variety of different devices, even though the different devices are manufactured by different third parties. In this manner, a user does not necessarily have to install multiple different third party applications to control their respective IoT devices. Rather, the user may rely on the automated assistant to recognize the particular context in which the user wishes to control a particular IoT device and provide a composite graphical assistant interface that includes suggestions for the particular IoT device to be controlled.

[0005] For example, a user may enter a kitchen that may include a variety of different devices, such as a client device that provides access to an automated assistant, and various connected devices in communication with the client device. The connected devices may include a stereo system, an oven, smart lighting, and a coffee maker. Each of the connected devices may be manufactured by a different third-party manufacturer, at least to the manufacturer of the automated assistant. Generally, in the morning when the user enters the kitchen, the user may provide a command such as "Assistant, turn on the kitchen lights." The user may then access various applications or manually interact with the connected devices to have another function performed. The user may initialize the automated assistant with language to perform each function (e.g., "Assistant, turn on the coffee maker... Assistant, play music"), but this would require additional speech processing, thereby consuming computational and network resources. Furthermore, if the user was required to access different applications to control each device, each application may require a certain amount of network connectivity, thereby further consuming computational and network resources inefficiently. To achieve more efficient use of such resources, the automated assistant may provide a composite graphical assistant interface in response to receiving an initial request to perform a function and / or in response to a user selection in a particular assistant interface.

[0006] For example, in response to receiving a voice utterance such as "Assistant, turn on the kitchen lights," the automated assistant may identify the current status of another connected device in the kitchen to identify a particular function to recommend to the user. For example, if it is determined that the coffee maker in the kitchen is turned on at the time the voice utterance is received, the automated assistant may exclude any reference to the coffee maker in the compiled composite graphical assistant interface. In this way, the automated assistant will not make redundant suggestions and can free up space on the composite graphical assistant interface for more suitable suggestions. Additionally, if the automated assistant determines that the stereo system in the kitchen is not currently playing music, the automated assistant may determine a function to suggest to the user to control the stereo system. For example, the automated assistant may provide a composite graphical assistant interface that includes graphical control elements for turning on the stereo system and / or changing the volume of the stereo system. The composite graphical assistant interface may be provided on a display panel integral to the client device or on a display panel associated with the client device (e.g., a tablet device also located in the kitchen). In this way, the user has a less computationally intensive modality through which to turn on the stereo system, which can be provided to the user in a timely manner in response to their initial voice utterance.

[0007] In some embodiments, suggestions for a particular function to be performed by a connected device may be based on data indicative of a user's historical tendency to interact with a particular connected device in a particular context. For example, a user may have a historical tendency to turn on their stereo system when their kitchen lights are on or immediately after a request to turn on the kitchen lights. When such a context later occurs, the automated assistant may provide a composite graphical assistant interface including a graphical control element for controlling the stereo system on a display panel in the kitchen or on another suitable display panel. Additionally, the automated assistant may identify another connected device associated with the user, a request from the user, the context of the request, and / or any other information related to the user to provide further suggestions in the composite graphical assistant interface. For example, a user may also have a historical tendency to preheat the oven in the kitchen after turning on the kitchen lights and music from the stereo system at night. When such a context later occurs, the automated assistant may provide a composite graphical assistant interface including graphical control elements for controlling one or more functions of the oven, such as baking temperature and timer settings, and graphical control elements controlling features of the kitchen lights and / or stereo system. In this manner, the composite graphical assistant interface generated by the automated assistant may provide control over a variety of different connected devices made by a variety of different manufacturers.

[0008] The above description is provided as a summary of some embodiments of the present disclosure, further descriptions of which, as well as other embodiments, are described in more detail below.

[0009] In some embodiments, a method implemented by one or more processors is described that includes operations such as determining that a voice utterance detected through an automated assistant interface of a client device includes a request for a connected device to perform a function. The connected device may be linked to the client device, and the voice utterance is provided by a user. The method may further include, in response to receiving the request, identifying a function corresponding to the received request, and causing the connected device to perform the function corresponding to the received request based on the identifying function, and in response to receiving the request, determining, by the automated assistant, another function to recommend to the user based on data accessible to the automated assistant. The another function is different from the identified function and can be performed by another connected device. The method may also include, based on determining the another function, identifying a graphical control element for controlling another function of the other connected device, and in response to receiving the request, causing a composite graphical assistant interface to be provided on a display panel associated with the client device, the composite graphical assistant interface comprising the identified graphical control element. The method may further include determining that a selection of a graphical control element has been received in the composite graphical assistant interface, and in response to determining that a selection of the graphical control element has been received, causing another connected device to perform another function.

[0010] In some embodiments, the data identifies a current status of the other connected device, and the current status is changed to a different status in response to the selection of the graphical control element. In some embodiments, determining the other function includes determining, based on the data, that the particular function at least partially realizes the current status of the other connected device, and selecting the other function for recommendation to the user based at least in part on the particular function being different from the other function. In some embodiments, determining the other function further includes determining one or more slot values ​​of the function that at least in part realizes the current status of the other connected device, and identifying one or more parameters of one or more candidate functions that can affect the current status of the connected device. Selecting the other function for recommendation to the user based at least in part on the particular function being different from the other function may include selecting the other function from the one or more candidate functions according to whether the other function includes a parameter that is different from the other parameter of the particular function.

[0011] In some embodiments, the method may include determining speakable commands that, when provided to the automated assistant interface, initialize the automated assistant to perform another function of another connected device. Providing the composite graphical assistant interface on the display panel may include causing the display panel to present the speakable commands.

[0012] In some embodiments, determining another function to recommend to the user based on data accessible to the automated assistant includes identifying one or more additional connected devices from the plurality of connected devices based on a stored device topology indicating a relationship of the one or more additional connected devices to the client device or the connected device, and identifying one or more additional functions that can be performed by the one or more additional connected devices. The another function can be determined from the identified one or more additional functions. In some embodiments, the data can indicate a status of the one or more additional connected devices based on having previously performed or not performed another function, and the configuration of the graphical control elements in the composite graphical assistant interface can be based on the indicated status of the one or more additional connected devices.

[0013] In another embodiment, a method implemented by one or more processors is described that includes determining that a voice utterance detected through an automated assistant interface of a client device includes a request for the automated assistant to perform a function, the voice utterance being provided by a user. The method may also include, in response to determining that the request is provided by a user, identifying a function corresponding to the request provided by the user, and causing the automated assistant to perform the function corresponding to the request provided by the user based on the identifying function. The method may further include determining a first current status of a first connected device linked to the client device. The first current status of the connected device may be affected by performing or not performing the first function. The method may also include determining a second current status of a second connected device linked to the client device. The second current status of the second connected device may be affected by performing or not performing the second function. The method may further include determining a first graphical control element for controlling a first function of the first connected device and a second graphical control element for controlling a second function of the second connected device, and determining a configuration for each of the first and second graphical control elements based on the first current status of the first connected device and the second current status of the second connected device. The method may also include, in response to determining that the request is received, causing a display panel associated with the client device to provide a composite graphical assistant interface for controlling the first and second connected devices. The composite graphical assistant interface may include the first and second graphical control elements arranged according to the determined configuration for each graphical control element.

[0014] In some embodiments, the method may include accessing a profile for the user based on voice characteristics associated with the voice utterance. Identifying the at least one of the first and second functions may be based on information available from the profile. In some embodiments, the profile may indicate a historical tendency of the user to interact with at least one of the first and second connected devices in a particular context. In some embodiments, the particular context may correspond to one or more of a status of the first or second connected device, a geographic location of the user, a time, an event, or a presence of one or more other users.

[0015] In some embodiments, determining a first current status of the first connected device may include requesting the first current status and command data associated with a first function from a remote server device. The first function may be different from the function performed by the automated assistant. In some embodiments, the method may include receiving a selection of a first graphical control element for controlling the first function and causing the client device to interact with the first connected device according to the command data associated with the first function.

[0016] In yet another embodiment, a method implemented by one or more processors is described that includes operations such as determining that a user has provided a request for the automated assistant to perform a function. The request may be received at an automated assistant interface of a client device. The method may further include, in response to determining that a user has provided a request, causing the automated assistant to perform a function corresponding to the request, and in response to determining that a user has provided a request, determining a user preference that characterizes a historical tendency of the user to control another function of the connected device in a context related to the function performed by the automated assistant. The connected device may be linked to the client device. In some embodiments, the method may include, based on determining the user preference, identifying the connected device and another function of the connected device, and determining a current status of the connected device based on identifying the connected device. The current status may be affected by performing or not performing the other function. The method may also include, based on identifying the other function and the current status of the connected device, determining a graphical control element for controlling the other function of the connected device. The graphical control element may indicate the ability of the graphical control element to cause the connected device to perform the other function, which will result in the current status of the connected device being changed to a different status. The method may further include, in response to determining that a user has provided the request, causing a display panel associated with the client device to provide a composite graphical assistant interface including the graphical control element.

[0017] In some embodiments, the method may include determining a selectable suggestion element associated with the separate connected device of the connected device based on a user preference. The step of causing the display panel to provide a composite graphical assistant interface may include causing the composite graphical assistant interface to include the selectable suggestion element. In some embodiments, the method may include determining that a selection of the selectable suggestion element is received at the composite graphical assistant interface, and causing one or more graphical control elements to be presented at the display panel to control one or more functions of the separate connected device. In some embodiments, the method may include determining a voice expressible command that, when received at the automated assistant interface of the client device, activates the automated assistant to cause the separate connected device to perform one or more functions of the separate connected device. The step of causing the display panel associated with the client device to provide a composite graphical assistant interface may include causing the composite graphical assistant interface to include a natural language representation of the voice expressible command.

[0018] In some embodiments, the method may include determining a final result status of the connected device in response to causing the connected device to execute the function. The final result status may be affected by the execution of the function. Further, causing a display panel associated with the client device to provide a composite graphical assistant interface may include causing the composite graphical assistant interface to include a graphical interface element that indicates the final result status of the connected device.

[0019] In some embodiments, the user preferences characterize a user's historical tendency to control another function based at least on a previous interaction between the user and the automated assistant. In some embodiments, the previous interaction corresponds to a dialog session between the user and the automated assistant before the user selected the graphical control element. During the dialog session, the user would have provided a request to have the connected device perform another function.

[0020] In another embodiment, a method implemented by one or more processors is described that includes operations such as determining that a user has provided a request for the automated assistant to perform a function. The request may be received at an automated assistant interface of a client device. The method may also include, in response to determining that the user has provided a request, causing the automated assistant to perform a function corresponding to the request, and in response to determining that the user has provided a request, determining a user preference that characterizes a historical tendency of the user to control another function of the connected device in a context related to the function performed by the automated assistant. The connected device may be linked to the client device. In some embodiments, the method may include, based on determining the user preferences, identifying the connected device and a desired status for the connected device, and based on identifying the connected device, determining a current status of the connected device. The current status may be affected by performing or not performing the other function. The method may further include, when the desired status is different from the current status of the connected device, determining a graphical control element for controlling another function of the connected device based on identifying the other function and the current status of the connected device. The graphical control element may indicate a capability of the graphical control element to cause another function to be performed by the connected device that will result in a current status of the connected device being changed to a desired status. Further, the method may include, in response to determining that a user has provided a request, causing a display panel associated with the client device to provide a composite graphical assistant interface that includes the graphical control element.

[0021] In some embodiments, the method may include, in response to determining that a user has provided a request when the desired status is equal to a current status of the connected device, causing a display panel associated with the client device to provide a specific composite graphical assistant interface that omits specific graphical control elements for controlling other functions of the connected device.

[0022] Other embodiments may include a non-transitory computer-readable storage medium having stored thereon instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphical processing unit (GPU), and / or a tensor processing unit (TPU)) for performing methods such as one or more of the methods described above and / or elsewhere herein. Still other embodiments may include one or more computers and / or one or more robotic systems including one or more processors operable to execute stored instructions to perform methods such as one or more of the methods described above and / or elsewhere herein.

[0023] It should be appreciated that all combinations of the foregoing concepts, and additional concepts described in more detail herein, are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief description of the drawings]

[0024] [Figure 1] FIG. 1 illustrates a system for providing a composite graphical assistant interface for interacting with connected devices. [Figure 2A] 1 is a view of a composite graphical assistant interface in which multiple different graphical control elements are provided for controlling multiple different connected devices. [Figure 2B]1 is a view of a composite graphical assistant interface in which multiple different graphical control elements are provided for controlling multiple different connected devices. [Figure 3A] A view in which suggestions are provided to a user for controlling one or more connected devices based on an interaction between the user and the automated assistant and / or a current status of the one or more connected devices. [Figure 3B] A view in which suggestions are provided to a user for controlling one or more connected devices based on an interaction between the user and the automated assistant and / or a current status of the one or more connected devices. [Figure 4A] FIG. 1 illustrates a method for providing a composite graphical assistant interface that includes suggestions for functions that can be performed by multiple different connected devices. [Figure 4B] FIG. 1 illustrates a method for providing a composite graphical assistant interface that includes suggestions for functions that can be performed by multiple different connected devices. [Diagram 5] FIG. 1 illustrates a method for replacing a composite graphical assistant interface with a graphical control element for controlling a connected device. [Figure 6] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] FIG. 1 illustrates a system 100 for assembling a composite graphical assistant interface for interacting with connected devices. The composite graphical assistant interface may be generated in part by an automated assistant 118 provided in one or more computing devices, such as the client device 102 (e.g., assistant device 128), additional client devices (e.g., tablet device 150), and / or server device 112. A user 120 may interact with the client automated assistant 104 via an assistant interface 110, which may be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, the user 120 may initialize the client automated assistant 104 by providing verbal, textual, and / or graphical input to the assistant interface 110 and / or the tablet device 150 to cause the client automated assistant 104 to perform an action (e.g., provide data, control a peripheral device, access an agent, etc.). Client device 102 may include a display device, which may be a display panel including a touch interface for receiving touch input and / or gestures to enable a user to control applications of client device 102 via a touch interface. Alternatively, client device 102 may be assistant device 128 that does not include a display panel but still provides one or more interfaces through which user 120 may interact with client automated assistant 104 (e.g., through voice utterances provided by user 120 into a microphone at assistant device 128).

[0026] The client device 102 may be in communication with the server device 112 via a network 122, such as the Internet. The client device 102 may offload computational tasks to the server device 112 to conserve computational resources at the client device 102 and / or to leverage more robust resources at the server device 112. For example, the server device 112 may host an automated assistant 118, and the client device 102 may send inputs received at one or more assistant interfaces to the server device 112. However, in some embodiments, the automated assistant 118 may be fully hosted on the client device 102. In various embodiments, all or less than all aspects of the client automated assistant 104 may be implemented on the client device 102. In some of those embodiments, aspects of the automated assistant 118 are implemented via the client automated assistant 104 of the client device 102 and interface with the server device 112, which implements other aspects of the client automated assistant 104. The server device 112 may optionally serve multiple users and their associated assistant applications via multiple threads. In some embodiments in which all or less than all aspects of the automated assistant 118 are implemented via the client automated assistant 104 of the client device 102, the client automated assistant 104 may be an application that is separate from the operating system of the client device 102 (e.g., installed "on top of" the operating system), or alternatively, may be implemented directly by the operating system of the client device 102 (e.g., an operating system application, but integral to the operating system).

[0027] In some embodiments, the server device 112 may include a speech-to-text engine 116 that may process audio data received at the assistant interface 110 to identify text embodied within the audio data. The process for converting the audio data to text may include speech recognition algorithms that may employ neural networks, word2vec algorithms, and / or statistical models to identify groups of audio data that correspond to words or phrases. The text converted from the audio data may be parsed by the text parser engine 108 and made available to the automated assistant 104 as text data that may be used to generate and / or identify command phrases from a user and / or third party application.

[0028] The user 120 may use their client device 102 and / or client automated assistant 104 to interact with the first connected device 132 and / or the second connected device 146. Each connected device may be one or more IoT devices that may be connected to or in communication with the client device 102 and / or tablet device 150 over a local area network and / or a wide area network. For example, the client device 102, the tablet device 150, the first connected device 132, and / or the second connected device 146 may be connected to a Wi-Fi network in a home or other environment. In order for the user 120 to interact with the first connected device 132, the user 120 may initialize an application associated with the first connected device 132. However, if the user 120 frequently interacts with multiple different connected devices, the user 120 may be required to operate a variety of different applications to control each connected device. To use computing resources more efficiently and prevent wasting network bandwidth, the client automated assistant 104 and / or the automated assistant 118 may provide a composite graphical assistant interface from which a user can interact with one or more different connected devices (e.g., the first connected device 132 and the second connected device 146).

[0029] In order for a composite graphical assistant interface to be generated and / or provided, the user 120 may provide a voice utterance 130 to the assistant interface 110. The voice utterance may embody a request indicating that the user 120 desires the first connected device 132 to perform a particular function. Audio data corresponding to the voice utterance 130 may be transmitted from the client device 102 to the server device 112. The audio data may then be processed by the speech-to-text engine 116 and by the text parser engine 114. The text resulting from the processing of the audio data may be used by the function identification engine 142 to identify one or more functions that the user 120 refers to in the voice utterance 130. For example, the function identification engine 142 may compare the text from the voice utterance 130 to the function data 136 made available in the connected device server 134. The function data 136 may provide a comprehensive list of functions that one or more connected devices (e.g., the first connected device 132 and the second connected device 146) are capable of performing. The function identification engine 142 can compare the text to the function data 136 to identify one or more functions referred to by the user that can be performed by the first connected device 132. The function data 136 can additionally or alternatively be stored locally to the server device 112.

[0030] In response to determining that the text corresponds to one or more functions, the automated assistant 118 or the client automated assistant 104 may cause the first connected device 132 to execute one or more identified functions. For example, when the first connected device 132 is an IoT device such as, but not limited to, an oven, the function may include preheating the oven to a particular temperature. A command to cause the first connected device 132 to execute a function may be provided from the server device 112 to the first connected device 132. Alternatively, a command to cause the first connected device 132 to execute a function may be provided from the server device 112 to the connected device server 134, which communicates with the first connected device 132 to cause the first connected device 132 to execute the function. Alternatively, the client device 102 may communicate a command to the first connected device 132, or the server device 112 may communicate a command to the client device 102, and the client device 102 may communicate with the first connected device 132 to cause the first connected device 132 to perform a function.

[0031] In response to receiving the voice utterance 130 to cause the first connected device 132 to perform a function, the automated assistant can cause a composite graphical assistant interface to be provided on a display device associated with the client device 102 (e.g., tablet device 150, television, laptop, cell phone). To generate the composite graphical assistant interface, the automated assistant 118 can identify from the capability data 136 one or more other functions that can be performed by the first connected device 132. In some embodiments, in response to determining that the voice utterance 130 was received at the client device 102, the automated assistant can determine one or more other functions that can be performed by the second connected device 146 (e.g., a different IoT device), the tablet device 150, and / or any other device associated with the client device 102. For example, the capability data 136 can include one or more lists of capabilities that correspond to devices associated with the user 120, the client device 102, and / or the first connected device 132. The automated assistant may determine one or more functions to provide suggestions or recommendations for other functions that the user 120 may want to be performed, in light of at least that request to have the first connected device 132 perform a function.

[0032] For example, in some embodiments, a profile for a user is identified based on audio characteristics of voice utterances received at assistant interface 110. The profile may indicate preferences associated with each of a number of particular connected devices associated with the user, such as preferences learned based on the user's historical interactions with the connected devices. Additionally, the profile may identify a group of connected devices with which the user 120 prefers to interact simultaneously, sequentially, and / or in a particular context or contexts. For example, the user 120 may request that the automated assistant have a first connected device 132 perform a first function and a second connected device 146 perform a second function, and the profile may identify the first function and the second function as related, for example, based on having occurred one or more times in close temporal proximity to each other. Thereafter, when the user 120 interacts with the automated assistant to cause the first connected device 132 to perform a first function, the automated assistant may access the profile to identify a second function based on its stored relationship with the first function and provide the user 120 with selectable suggestions for causing the second connected device 146 to perform the second function.

[0033] In some embodiments, historical interactions between the user 120 and the automated assistant may be processed with permission from the user 120 to determine the user 120's historical tendency to interact with a particular connected device in a particular context. For example, the user 120 may have a history of interacting with two IoT devices in the morning, and thus the automated assistant may inform the user 120 that when the user 120 is interacting with at least one of the two IoT devices in the morning, the user 120 is likely to interact with the other IoT device, at least until the morning has passed. Additionally or alternatively, the automated assistant may provide suggestions based on the location of the user 120. For example, when the user 120 is in his kitchen 148, the user 120 may have a history of having his oven (e.g., the first connected device 132) perform a first function and his tablet device 150 perform a second function. Thus, when the user 120 causes the first connected device 132 to execute a first function and it is determined that the user 120 is in the kitchen 148, the automated assistant may have a selectable suggestion element provided in the composite graphical assistant interface. The selectable suggestion element may operate as a suggestion for the user 120 to execute a second function on the second connected device 146. The user may be determined to be in the kitchen 148, for example, based on the client device 102 or the tablet device 150 being utilized by the user to execute the first function. The "kitchen" may be optionally identified as an explicit location, for example, based on the device 102 and / or the device 150 being assigned a kitchen label 148. Alternatively, the user's location may be a location that corresponds more generally to the client device 102 or the tablet device 150. In some embodiments, the context may be whether the user 120 is attending a particular event, such as an event identified in calendar data accessible to the automated assistant.In some embodiments, the context may be whether one or more other users, different from user 120, are present at a particular location.

[0034] When the automated assistant identifies an alternative function to suggest to the user 120, the identified alternative function may be used as a basis for identifying a graphical control element 140. For example, a function such as turning a fan on and off may be controlled by a graphical control element 140 corresponding to a two-way switch. Additionally, a function such as adjusting an oven temperature or music volume may be controlled using a graphical control element 140 corresponding to a dial that may be rotated to select a value in a range of values. The automated assistant 118 or the client automated assistant 104 may map the identified alternative function to a graphical control element 140 that may be stored at or otherwise accessible to the server device 112.

[0035] The composite interface engine 144 may map the graphical control elements 140 to the identified alternative functions to generate a composite graphical assistant interface. In some embodiments, the automated assistant 118 may access status data 138 corresponding to the status of the first connected device 132. The status data 138 may indicate a device status that may be affected by the execution or non-execution of one or more functions of the first connected device 132 and / or the second connected device 146. For example, when no functions of the first connected device 132 are being executed, the status of the first connected device 132 may be "off". Alternatively, the status may correspond to a table or string of data indicating the state or parameters of a particular function of the first connected device 132 and / or the second connected device 146. The status for each of the first connected device 132 and the second connected device 146 may be used as a basis to determine a configuration for each graphical control element 140 that is mapped therefrom to the functions of the first connected device 132 and the second connected device 146. For example, if the first connected device 132 includes a heating element that is in an off state and a function of the first connected device 132 is to turn on the heating element, a graphical control element corresponding to a bidirectional switch may be provided in the composite graphical assistant interface and configured in such a manner as to indicate that the heating element is turned off. In other words, each graphical control element may reflect the respective current state of the first connected device 132 and the second connected device 146 when the composite graphical assistant interface is presented on the tablet device 150.

[0036] In some embodiments, the placement and / or selection of graphical control elements for the composite graphical assistant interface may be based on data related to interactions between the user 120 and the automated assistant and / or the first connected device 132. For example, a particular group of graphical control elements corresponding to a particular group of functions may be selected for the composite graphical assistant interface based on the user having previously performed such functions within a threshold time period. Alternatively, or in addition, a particular group of graphical control elements corresponding to a particular group of functions may be selected for the composite graphical assistant interface based on how the functions affect a particular user, a particular location, a particular device, and / or any other relationships that may be indicated by the device's functions. For example, a composite graphical assistant interface generated for a user may include graphical control elements corresponding to functions of multiple different connected devices that affect the residence where the user 120 is currently located. Thus, the selection of graphical control elements may be based on the user's location, identifiers for the connected devices, the user's calendar schedule, and / or any other data that may be used to estimate geolocation data for the user.

[0037] In some embodiments, the user 120 may provide a voice utterance 130 to an assistant device 128 that lacks an integrated display panel or is otherwise unable to easily present a composite graphical assistant interface at the assistant device 128. In such embodiments, the automated assistant may receive the voice utterance 130 at the assistant device 128 and have the function requested by the user 120 performed by the first connected device 132. However, in response to receiving a request to perform a function and determining that a display panel is not available at the assistant device 128, the automated assistant may identify one or more candidate devices. The candidate devices may be identified based on their ability to display the assistant interface, their proximity to the user 120, their proximity to the assistant device 128, their proximity to the first connected device 132, and / or the user 120's preferences for a particular candidate device. For example, when the display panel determines that an automated assistant is not available on the assistant device 128 and the tablet device 150 or the television is the next closest display device to the assistant device 128, the automated assistant can cause the tablet device 150 or the television to present the composite graphical assistant interface. In this manner, the automated assistant can still generate the composite graphical assistant interface on a nearby display device because the user 120 is not always near the display panel when the user requests that a particular function be performed by the connected device. In some embodiments, the automated assistant can cause the assistant device 128 to audibly notify the user 120 of the location of the composite graphical assistant interface in response to the user 120 providing a request to the connected device 132 to perform a function.For example, in response to user 120 providing a voice utterance of "Assistant, preheat the oven," the automated assistant can cause assistant device 128 to audibly output the response "Ok, I've provided an interface for additional controls on your tablet device." When user 120 goes to retrieve the tablet device, the automated assistant can thus inform user 120 that user 120 can find additional controls for first connected device 132 and second connected device 146.

[0038] 2A and 2B show views 200 and 210 of a composite graphical assistant interface 222 in which multiple different graphical control elements are provided for controlling multiple different connected devices. In response to a user 202 providing a voice utterance 208 to the automated assistant via the automated assistant interface 214, the composite graphical assistant interface 222 can be provided to the user 202. The automated assistant interface 214 can be, for example, at least a microphone capable of providing a signal in response to a user 202 providing an audible voice utterance 208 to the automated assistant interface 214. For example, the voice utterance 208 can be "Assistant, turn up the temperature in my kitchen." In response to receiving the voice utterance 208, the automated assistant can determine the function and / or connected device with which the user 202 seeks to interact and can cause the connected device to perform the function that the user 202 is requesting to be performed. For example, in response to receiving the voice utterance 208, the automated assistant can cause a thermostat in the user's 202 residence to adjust the temperature of at least the kitchen of the residence.

[0039] To alleviate unnecessary processing of subsequent voice utterances provided by the user 202, thereby saving computational and network resources, the automated assistant can have function suggestions provided on the client device 204, or on the display panel 206 of a separate device. For example, in response to receiving the voice utterance 208, the automated assistant, the client device 204, and / or the remote server device can access data from which the automated assistant provides recommendations for controlling another function of another connected device. The connected device can be a device that is linked to or otherwise capable of communicating with the client device 204 and / or the automated assistant. The data can provide information regarding the status of each connected device, and the suggestions provided in the composite graphical assistant interface 222 can be based on the particular status of the particular connected device. For example, the status of the kitchen light at the time the user 202 provided the voice utterance 208 can be "off," and therefore, since the user 202 expressed an interest in changing the kitchen environment, the user 202 can also be interested in changing the status of the light in the kitchen. Specifically, the lighting in the kitchen may be a non-binary parameter related to its brightness, and thus, in response to determining that the user 202 may also be interested in adjusting the kitchen lighting, the automated assistant may select a dial graphical control element that may function as the kitchen brightness control 218.

[0040] In some embodiments, the kitchen brightness control 218 may be selected for the composite graphical assistant interface 222 based on the presence of a kitchen light in which the user 202 has specifically expressed an interest in changing its temperature, within a general location as the kitchen. Alternatively, or in addition, the kitchen brightness control 218 may be selected for recommendation to the user 202 in the composite graphical assistant interface 222 based on the user's 202 historical tendency to modify the kitchen temperature and kitchen lighting in a particular context. The user's 202 historical tendency may be shown as part of a user profile or preference data that characterizes the tendency based on previous interactions between the user 202 and the automated assistant and / or between the user 202 and one or more connected devices. In some embodiments, such interactions may occur in a particular context, such as a particular time 212 (e.g., after 6:00 a.m.), and thus the automated assistant may generate the composite graphical assistant interface 222 based on the particular context. Alternatively, or in addition, the context may be characterized by one or more statuses of one or more respective connected devices.

[0041] Alternatively or additionally, the automated assistant may identify a request from a user and compare the request with available data (e.g., preference data and / or historical interaction data) to determine one or more functions to suggest to the user. When the automated assistant has identified one or more functions to suggest, the automated assistant may identify each connected device corresponding to each of the one or more identified functions. The status of each connected device may then be determined to ensure that suggestions for functions to perform are not redundant or would not result in a change to the status of the connected device. For example, a user typically turns on their kitchen lights in the morning and then their coffee maker. In such a scenario, the automated assistant may provide a voice utterance (e.g., "Assistant, turn on my kitchen lights") to initialize the automated assistant to turn on the kitchen lights. Based on data indicating a user's historical interactions to turn on both the kitchen lights and the coffee maker, the automated assistant may identify the coffee maker and / or a function to turn on the coffee maker in response to receiving a request to turn on the kitchen lights. To ensure that the functionality suggestion is meaningful, the automated assistant may compare the current status of the coffee maker to the status that the user typically desires for the connected device when the user turns on the kitchen light. If the current status of the coffee maker reflects the desired status, the automated assistant may exclude providing a graphical control element in the composite graphical assistant interface for controlling the coffee maker. However, if the current status of the coffee maker does not reflect the desired status (e.g., the coffee maker is off and therefore does not express the desired status), the automated assistant may select a graphical control element for controlling the functionality to turn on the coffee maker.In this way, the composite graphical assistant interface can be optimized to provide graphical control elements that will effectuate changes that a user would typically desire, rather than providing suggestions to connected devices that may have already occurred.

[0042] For example, the composite graphical assistant interface 222 may include a variety of different graphical control elements that are selected for the composite graphical assistant interface 222 based on the status of the particular connected devices to which they correspond. For example, because the user 202 has asked to turn up the temperature in the kitchen, the automated assistant may identify other connected devices in the kitchen that the user may be interested in controlling, such as the oven and / or the coffee maker. Because each of these connected devices is off (as indicated by the configuration of the oven temperature control 216 and the coffee maker control 220), these connected devices may be candidate connected devices for the control suggestion. Additionally, the user 202 may have a history of turning up the temperature in the kitchen, adjusting the lights in the kitchen, preheating its oven, and turning on its coffee maker, all within a threshold time range after 6:00 a.m. Even though each of these devices and / or their respective control applications are provided by different third parties, the automated assistant may compile the composite graphical assistant interface 222 to allow the user 202 to control each device from a single interface.

[0043] In some embodiments, as a result of the user 202 selecting one of the graphical control elements in the composite graphical assistant interface 222, the status of the corresponding connected device may change. For example, initially, the brightness of the kitchen lights is shown as "off" based at least on the kitchen brightness control 218 being at a lowest level. When the user 202 adjusts the kitchen brightness control 218 to a higher level, the status of the kitchen lights may be updated to have an "on" status. Furthermore, the status of the kitchen lights may be updated to reflect a particular slot value generated in response to the adjustment of the graphical control element. For example, the slot value may be "60%" indicating that the kitchen brightness control 218 has been adjusted up to 60%, and thus the status of the kitchen lights may be updated to be "kitchen_lights:status(lights[on], brightness

[60] , hue[null])". Any other subsequent suggestions may then also be based on the updated status of the kitchen lights. For example, because the kitchen light currently has a status of "on," the automated assistant can have the composite graphical assistant interface 222 provide a graphical control element for modifying the hue of the kitchen light based at least on the fact that the kitchen light is on and that the "hue" parameter of the kitchen light is "null," i.e., the default setting.

[0044] In some embodiments, the composite graphical assistant interface 222 may present a status indication 224 of the function to be executed in response to the provision of the voice utterance 208 by the user 202. Additionally, in response to receiving the voice utterance 208, the automated assistant may have the composite graphical assistant interface 222 provide suggestions implemented as natural language that characterize the voice utterance. The voice utterance may be a voice utterance that, when provided to the automated assistant interface, initializes the automated assistant to cause one or more connected devices to perform one or more functions. For example, in response to receiving the voice utterance 208, the automated assistant may identify the status of connected devices located within an area (e.g., the kitchen) affected by the voice utterance 208. The automated assistant may then identify a natural language input that, when received at the automated assistant interface, will initialize the automated assistant to cause the connected devices to perform a function. For example, when the connected device is a coffee maker, the automated assistant may identify a voice utterance previously employed to control the coffee maker, such as "Assistant, turn on the coffee maker." In response to identifying such a voice utterance, the automated assistant can cause the graphical assistant interface 222 to present the voice utterance as a selectable suggestion element that includes at least the text "Turn on the coffee maker." In this manner, the user 202 can either select the selectable suggestion element to turn on the coffee maker or speak the text as a voice utterance to initialize the automated assistant to turn on the coffee maker. The automated assistant can also retrieve command data that can be used by the client device 204 to interact with a particular connected device. In this manner, upon determining a particular suggestion for the automated assistant to make to the user 202, the automated assistant can also locally provide data that will enable the suggested function to be performed by the connected device.As a result, the automated assistant and / or client device do not necessarily have to communicate with a remote server in response to a user's selection of a suggestion to obtain additional information regarding how to cause the connected device to perform a function.

[0045] 3A and 3B show views 300 and 344 in which suggestions for controlling one or more connected devices are provided to the user 302 based on the interaction between the user 302 and the automated assistant and / or the current status of the one or more connected devices. For example, the user 302 may enter his kitchen 320 and initialize the client automated assistant 330 via the assistant interface 332 of the client device 328 (e.g., the assistant device 326). The user 302 may initialize the client automated assistant by providing a voice utterance 322, such as "Assistant, play music in the kitchen." The client automated assistant 330 may receive the voice utterance 322 and have it processed at the client device 328 or a separate remote server to identify an action or function the user 302 desires to be performed. Because the client device 328 does not include a display panel, the client device 328 may access the device topology to determine from there a suitable device that provides a composite graphical assistant interface 306 that further controls some functionality related to the requested function (i.e., playing music) and / or some other functionality related to another connected device identified within the device topology.

[0046] For example, a stored device topology accessible to the client automated assistant 330 may identify the tablet device 316, the first connected device 318, and the second connected device 324 in a device topology that identifies the devices in the kitchen 320. The device topology may further identify the tablet device 316 as being the only device with a display panel, and thus any composite graphical assistant interface to be presented to the user 302 while the user 302 is in the kitchen 320 may be provided on the tablet device 316. Furthermore, to generate a suitable composite graphical assistant interface, the automated assistant may identify the status of each connected device and / or determine a historical trend of the user 302 interacting with one of the connected devices after a requested function (i.e., play music) is performed. For example, because the current status of the first connected device 318 (e.g., oven) is off, the automated assistant may suggest that the user control the oven by presenting a selectable suggestion element 310 in the composite graphical assistant interface 306. Further, because the current status of the second connected device 324 (e.g., the dishwasher) is off, the automated assistant may suggest that the user control the dishwasher by presenting a selectable suggestion element 312 in the composite graphical assistant interface 306. Moreover, the automated assistant may determine whether there is any other function related to the originally requested function of playing music, and provide a graphical control element in the composite graphical assistant interface 306 for controlling the other function. For example, a function of turning on music, which may be executed by the client automated assistant 330 at the client automated assistant 330, may cause the status of the client device to be at least "client_device:status(music[on], volume[default])".In response to a parameter of the client device 328 being in a default mode, the automated assistant may provide a graphical control element to control the corresponding parameter. For example, the automated assistant may provide a kitchen volume control 314 to control the volume of music in the kitchen 320. However, if the user 302 did not provide a previous request corresponding to turning on the music, the automated assistant may exclude the kitchen volume control 314 from the composite graphical assistant interface 306.

[0047] FIG. 3B may show a view 344 of a different composite graphical assistant interface 340 provided by the automated assistant in response to a selection by the user 302 of a selectable suggestion element in the composite graphical assistant interface 306 of FIG. 3A. For example, in response to the user 302's selection of the "Control the oven" selectable suggestion element 310, the automated assistant may provide a different composite graphical assistant interface 340 including a dial 334 for setting the oven temperature. An initial configuration of the dial 334 may be "0" to reflect the current status of the first connected device 318 and / or one or more current parameters of the first connected device 318. For example, the current status of the first connected device 318 may be "oven:status(power[off], temperature[0])" and thus the configuration of the dial 334 may reflect multiple current parameters of the first connected device 318. In other words, the dial 334 in its lowest state represents that the oven itself is off and thus the temperature is set to 0. Such parameters are consolidated and reflected in a single graphical control element to conserve space in the composite graphical assistant interface 340 and provide a more intuitive modality through which the user 302 is provided with more information. If the user 302 selects the “control the dishwasher” selectable suggestion element 312, a switch 336 is presented in a different composite graphical assistant interface 340 based on the particular second connected device 324 with a current status of “off” and a single parameter for control (on or off). In some embodiments, metadata corresponding to the requested function or the status of the requested function (e.g., play music) may be presented as a status indication 342. In this way, the user 302 does not necessarily have to re-launch the automated assistant to determine the current status associated with the request, or the user 302 does not have to swap between different interfaces to determine such information.Rather, the automated assistant metadata can be presented with graphical control elements associated with other third-party devices to provide a unified interface that would reduce waste of computational and network resources.

[0048] In some embodiments, the automated assistant may exclude providing a graphical control element for another connected device because the current status of that connected device indicates that a parameter related to the other connected device is set. For example, if the current status of the kitchen light is “on” when the user 302 provides the voice utterance 322, the automated assistant may acknowledge the current status of the kitchen light and exclude suggesting a function to turn the light off. Such exclusion may be based on one or more statuses of one or more devices in the kitchen or other areas of the residence and / or the user 302's historical tendency to not turn the light off when the user 302 requests to play music in the kitchen. In some embodiments, identifying the historical tendency may be based on voice characteristics of the user 302, which allows the automated assistant to identify data related to a particular user that will be processed to make suitable suggestions to one or more users. In some embodiments, the inclusion or exclusion of a particular graphical control element may be based on a particular context in which the user provided the voice utterance 322, such as before or after a particular time 304 or within a threshold time relative to a particular time, a geographic location, an event, and / or the presence of one or more users.

[0049] 4A and 4B show methods 400 and 410 for providing a composite graphical assistant interface that includes suggestions for functions that can be performed by multiple different connected devices. Method 410 may be a continuation of method 400 as indicated by the circled continuation element "A" in FIGS. 4A and 4B. Methods 400 and 410 may be performed by one or more computing devices, applications, and / or any other device or module capable of interacting with a connected device or an automated assistant. Method 400 may include an operation 402 of determining that a voice utterance detected via an automated assistant interface of a client device includes a request for a connected device to perform a function. The automated assistant interface may be one or more of a microphone, a touch display panel, a camera, a sensor, and / or any other device through which a user may provide input to the device. Additionally, the client device may be a computing device that may include a touch display panel that is integral with or otherwise in communication with the computing device. For example, the client device may be an assistant device connected to a WiFi network, and the assistant device may be communicating with a separate display panel (e.g., a television or other display device) over the WiFi network. Any device with which the client device is communicating may be a connected device. For example, any IoT device in a home that includes a WiFi network to which the client device is connected may be considered a connected device. Alternatively or additionally, any device with which the client device is communicating over a local area network, a wide area network, and / or the Internet may be a connected device. The automated assistant may receive commands from a user through an automated assistant interface and communicate with the connected device by having the client device interact with the connected device.Alternatively or additionally, in response to receiving a command by the automated assistant, the automated assistant can cause the remote server to issue a command to the connected device. Alternatively or additionally, the automated assistant can cause the remote server to issue a command to a connected device server associated with the connected device, such that the connected device server will cause the connected device to perform a particular function.

[0050] The method 400 may further include an operation 404 of identifying a function corresponding to the received request in response to receiving the request. The function may be identified by accessing a store or list of available functions that can be executed by one or more connected devices in communication with the client device. In some embodiments, a remote server device that at least partially hosts the automated assistant may query a separate connected device server that may provide data identifying one or more functions that can be executed by the connected device. When the connected device is an IoT device, the function may be any executable operation that can be executed by the connected device, such as, but not limited to, providing information from a sensor, modifying input parameters for a hardware component of the connected device, causing the connected device to perform a network operation, causing the connected device to communicate with another connected device, and / or any other function that can be executed by the device. For example, a function, when executed by the connected device, may cause the connected device to turn off any lights that can be controlled by the connected device (e.g., "function(connected_device:lights:{off})", where "off" is a slot value of the parameter "lights").

[0051] The method 400 may also include an operation 406 of causing the connected device to execute the function corresponding to the received request based on identifying the function. The client device or the automated assistant may cause the connected device to execute the function by sending or causing the connected device to send command data to the connected device. As a result of the connected device executing the function, the status of the connected device may be modified to reflect any changes effected by the execution of the connected device. The status of the connected device may be characterized by data such as "connected_device:status(lights[off], brightness[null], hue[null])" where certain parameters or slot values ​​may be empty or null as a result of certain slot values ​​(e.g., "off") provided to the connected device. In some embodiments, depending on the context in which the function was executed by the connected device, a profile associated with the user, or context, may be updated or otherwise modified to reflect the request to execute the function. In this manner, data corresponding to historical interactions between the user and the connected device and / or the automated assistant may be later processed to enhance future interactions between the user and the connected device and / or the automated assistant. For example, when a feature corresponds to a request to turn off the device's lights at night, the profile may be updated to rank certain features higher at night and / or otherwise indicate the user's historical tendency to request features to be performed at night. The profile may then be used as a basis to provide recommendations for features to be performed by various different connected devices, at least according to the user's historical tendency to request those or similar features in particular contexts. In some embodiments, the profile may indicate features that should be suggested according to the status of one or more connected devices.In this manner, the automated assistant may examine one or more statuses of one or more connected devices and determine a user's historical tendency to request particular functions based on the one or more statuses of the one or more connected devices.

[0052] Method 400 may further include operation 408, in response to receiving the request, determining, by the automated assistant, another function to recommend to the user based on data accessible to the automated assistant. The another function may be different from the function performed in operation 406 and may be performed by a different connected device than the connected device that performed the function in operation 406. The another function may be determined based on data accessible to the automated assistant that causes the another function to be performed in a current context or that indicates a historical trend of a user interacting with another connected device. For example, the another function may be determined based on a current status of the connected device that performed the function in operation 406. Thus, when the current status of the connected device is "connected_device:status(lights[off], brightness[null], hue[null])", the suggested another function may be to turn on a security alarm of a security system (e.g., another connected device) based at least on previous interactions when the user turned off the lights, changed the status of the lights, and turned on the security system. By providing such suggestions, computational and network resources may be conserved compared to having the user issue a voice command to perform such a function, thereby effecting voice processing.

[0053] Method 400 may proceed to method 410, as indicated by the circled continuation element "A" in FIGS. 4A and 4B. Method 410 may include operation 412 of identifying a graphical control element for controlling another function of another connected device based on determining the another function. The graphical control element may be selected from a plurality of graphical control elements stored in a memory accessible to the client device and / or the automated assistant. In some embodiments, each graphical control element may be stored in association with metadata indicating an input that may be characterized by the graphical control element and / or an output that may be provided in response to a selection or gesture received by the graphical control element. For example, a graphical control element representing a bidirectional switch may be stored in association with metadata indicating that the graphical control element may receive touch input and / or a swipe gesture, and / or the graphical control element may provide a binary output, such as a 1 or 0, and / or on or off. The memory may store a variety of different graphical control elements, each capable of providing two or more different outputs and / or receiving a variety of different inputs.

[0054] The method 410 may also include an operation 414 of causing a composite graphical assistant interface to be provided at a display device associated with the client device in response to receiving the request. The composite graphical assistant interface may be a graphical user interface with one or more selectable elements that can cause a particular function to be executed at the connected device when a selection and / or gesture is received at the composite graphical assistant interface. The composite graphical assistant interface may function as a modality through which a user may interact with the automated assistant and / or multiple different connected devices, even though the connected devices are associated with other different third party applications for controlling those connected devices. In this manner, a user does not necessarily have to download and / or access multiple different applications to control their respective IoT devices, but rather may simply have the composite graphical assistant interface displayed at a display panel. Additionally, the composite graphical assistant interface may be adapted by the automated assistant according to one or more statuses of one or more respective connected devices and / or a context in which the user is interacting with the automated assistant and / or the client device. In some embodiments, the composite graphical assistant interface may be associated with data including slot values ​​for particular graphical control elements presented in the composite graphical assistant interface. The slot value may be provided to a corresponding function, which may then be executed by a connected device corresponding to any respective graphical control element.

[0055] The method 410 may further include an operation 416 of determining that a selection of a graphical control element has been received in the composite graphical assistant interface. The selection of the graphical control element may be determined based on data transmitted by a display panel providing the composite graphical assistant interface in response to a touch or otherwise providing a gesture on the display panel by a user. Any slot values ​​corresponding to the determined selection of the graphical control element may then be provided to a respective connected device, such as the connected device itself, or a device responsible for further executing a function in the connected device server.

[0056] The method 410 may also include an operation 418 of causing another connected device to execute another function in response to determining that a selection of the graphical control element has been received. The another function may be, for example, turning on a security system at the user's residence. The another function may be suggested to the user based at least on the user's historical tendency to manually turn off its lights and turn on its security system. The another function may also be suggested based on the current status of the light (e.g., that the light is turned off). Alternatively, or in addition, the another function may be suggested based on the automated assistant determining that a current status of the light is being requested to be changed to "off" and / or the current context in which the request was made corresponds to the time of day, a geographic location relative to the user, the status of one or more other devices, and / or any other information that may be used to cause a computer function to be executed. In response to the execution of the another function, the current status may be updated in a table or other data store that includes identifiers related to the status of the various connected devices. In this manner, the automated assistant may generate additional ones from the updated combination of statuses and as the statuses change over time. For example, at this time, the security alarm is on and the lights are off, so the automated assistant may determine a graphical control element to control a function of the communication device to allow the user to set the communication device to silent mode by interacting with the graphical control element in the new composite graphical assistant interface.

[0057] FIG. 5 illustrates a method 500 for replacing a composite graphical assistant interface with a graphical control element for controlling a connected device. The method 500 may be performed by one or more computing devices, applications, and / or any other apparatus or module capable of interacting with a connected device and / or an automated assistant. The method 500 may include an operation 502 of determining that a user has selected a graphical control element in a first composite graphical assistant interface provided in a display panel associated with a client device. The display panel may be connected to the client device as an integral part of the client device or as a device in communication with the client device over a network. The first composite graphical assistant interface may be presented in the display panel to provide a user with a modality through which the user may control various different connected devices, even though the connected devices are manufactured by the same or different entities. Thus, the first composite graphical assistant interface may include a plurality of different graphical control elements through which the user may control various different connected devices.

[0058] Method 500 may also include an operation 504 of, in response to determining that the user has selected a graphical control element, causing the connected device to perform a function corresponding to the selected graphical control element. The connected device may be caused to perform the function via the client device and / or the automated assistant. For example, in response to receiving a selection of the graphical control element through the first composite graphical assistant interface, the client device and / or the automated assistant may send data to the connected device to cause the connected device to perform the function. Alternatively or additionally, the client device may communicate data to a remote server, which may communicate with the connected device to cause the connected device to perform the function.

[0059] The method 500 may further include an operation 506 of determining a user preference characterizing a user's historical tendency to control another function of another connected device in a context related to the connected device. The user preference may be data indicative of a user's tendency or preference to control a particular function of one or more connected devices in a particular context. For example, the context may correspond to one or more statuses of one or more respective devices, the time of day, a user's gesture or gaze, a scheduled event, a match with a previous interaction between the user and the automated assistant, and / or any other data that may be used as a basis for suggesting a function for the user to control. In some embodiments, the user preference or other data may rank the functions according to the demand or frequency of those functions that a particular user controls in a particular context. To give the functions a ranking, a learning model may be employed to receive input regarding the functions to be performed and output data corresponding to a rank for the functions to be suggested to the user. The learning model may be trained based on data characterizing interactions between the automated assistant and the user and / or between the user and one or more connected devices.

[0060] Method 500 may also include an operation 508 of identifying another function for another connected device based on determining the user preferences. The user preference data may identify the other function, a number of different connected devices, one or more parameters and / or slot values ​​corresponding to the other function, slot values ​​previously provided for the other function in a particular context, and / or any other data that may be useful in performing the particular function.

[0061] The method 500 may include an operation 510 of determining, based on identifying the other function, another graphical control element for controlling another function of another connected device. The other graphical control element may be selected from memory accessible to the client device and / or the automated assistant and may be associated with metadata that provides at least a certain correspondence to the other identified function. For example, the function may be associated with a parameter that may receive two or more different slot values, as opposed to a binary parameter that may utilize only two different slot values. As a result, the graphical control element to be selected to control the graphical control element may be a graphical control element capable of providing two or more different valued outputs, such as a dial or scroll bar.

[0062] The method 500 may further include an operation 512 of causing a display panel associated with the client device to replace the first composite graphical assistant interface with a second composite graphical assistant interface including other graphical control elements. The second composite graphical assistant interface may replace the first composite graphical assistant interface to provide further suggestions for the user to control a particular connected device in a particular context. In some embodiments, the first composite graphical assistant interface may correspond to a single connected device provided by a third party, and the second composite graphical assistant interface may correspond to multiple different connected devices provided by multiple different third parties. In this manner, a user may not necessarily need to install and / or open various different applications to control different connected devices, but rather may interact with a single composite graphical assistant interface to control various different connected devices.

[0063] 6 is a block diagram of an exemplary computer system 610. The computer system 610 generally includes at least one processor 614 that communicates with several peripheral devices via a bus subsystem 612. These peripheral devices may include, for example, a storage subsystem 624 including a memory 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow a user to interact with the computer system 610. The network interface subsystem 616 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.

[0064] The user interface input devices 622 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphical tablet, a scanner, a touch screen integrated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all conceivable types of devices and methods for inputting information into the computer system 610 or onto a communications network.

[0065] The user interface output devices 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as an audio output device. In general, use of the term "output device" is intended to include all conceivable types of devices and methods for outputting information from the computer system 610 to a user or to another machine or computer system.

[0066] The storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include logic for performing selected aspects of the method 400, the method 410, the method 500, and / or for implementing one or more of the connected device server 134, the server device 112, the automated assistant 118, the client device 102, the first connected device 132, the second connected device 146, the capability identification engine 142, the composite interface engine 144, the client device 204, the client device 328, the assistant device 326, and / or any other operations or apparatus discussed herein.

[0067] These software modules are generally executed by the processor 614 alone or in combination with other processors. The memory 625 used within the storage subsystem 624 may include several memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution, and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide permanent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of some embodiments may be stored by the file storage subsystem 626 within the storage subsystem 624 or within other machines accessible by the processor 614.

[0068] Bus subsystem 612 provides a mechanism for allowing the various components and subsystems of computer system 610 to communicate with each other as intended. Although bus subsystem 612 is illustrated generally as a single bus, alternative embodiments of the bus system may use multiple buses.

[0069] Computer system 610 may be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or some other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of computer system 610 shown in Figure 6 is merely intended to be a specific example to illustrate some embodiments. Many other configurations of computer system 610 are possible, having more or fewer components than the computer system shown in Figure 6.

[0070] In situations where the systems described herein may collect or utilize personal information about users (or as is often referred to herein as "participants"), the user may be provided with an opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or to control whether and / or how content that may be more relevant to the user is received from a content server. Also, certain data may be handled in one or more ways before it is stored or used such that personally identifiable information is removed. For example, the user's identifying information may be handled such that no personally identifiable information can be determined about the user, or if geographic location information is obtained (such as to the city, zip code, or state level), the user's geographic location may be generalized such that the user's specific geographic location cannot be determined. Thus, the user may have control over how information about the user is collected and / or used.

[0071] Although several embodiments have been described and illustrated herein, various other means and / or structures for performing the functions and / or obtaining the results and / or one or more of the advantages described herein may be utilized, and each of such variations and / or modifications are deemed to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or applications in which the / these teachings are used. Those skilled in the art will recognize, or be able to ascertain, using no more than routine experimentation, many equivalents to the specific embodiments described herein. Thus, the foregoing embodiments are presented by way of example only, and it should be understood that, within the scope of the appended claims and equivalents thereto, embodiments other than those specifically described and claimed may be practiced. The embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent. [Explanation of symbols]

[0072] 100 Systems 102 client devices 104 Client Automated Assistant, Automated Assistant 108 Text Parser Engine 110 Assistant Interface 112 Server Device 114 Text Parser Engine 116 Speech to Text Engine 118 Automated Assistants 120 users 122 Network 128 Assistant Devices 130 Voice utterances 132 First Connecting Device 134 Connected Device Server 136 Functional Data 138 Status Data 140 Graphical Control Elements 142 Feature Identification Engine 144 Composite Interface Engine 146 Secondary Connecting Device 148 Kitchen Labels 150 tablet devices, devices 200 Views 202 users 204 Client Device 206 Display Panel 208 Speech, Audible Speech 210 Views 214 Automated Assistant Interface 216 Oven Temperature Control 218 Kitchen Brightness Control 220 Coffee maker control 222 Composite Graphical Assistant Interface, Graphical Assistant Interface 300 Views 302 users 304 Specific Time 306 Composite Graphical Assistant Interface 310 Suggestion Elements 312 Suggestion Elements 314 Kitchen Volume Control 316 Tablet Devices 318 First Connecting Device 320 Kitchen 322 Voice utterances 324 Secondary Connecting Device 326 Assistant Device 328 client devices 330 Client Automated Assistant 332 Assistant Interface 334 Dial 336 Switch 340 Composite Graphical Assistant Interface 342 Status Indicator 344 Views 610 Computer Systems 612 Bus Subsystem 614 processor 616 Network Interface Subsystem 620 User Interface Output Device 622 User Interface Input Devices 624 Memory Subsystem 625 Memory 626 File Storage Subsystem

Claims

1. 1. A method executed by one or more processors, comprising: determining that a voice utterance detected via an automated assistant interface on a client device includes a request for the automated assistant to perform a function, the voice utterance being provided by a user; in response to determining that the request was provided by the user, identifying the function corresponding to the request provided to the user; causing the automated assistant to execute the function corresponding to the request provided by the user based on the identification of the function; determining a first current status of a first connected device linked to the client device, the first current status of the first connected device being affected by performing or not performing a first function; determining a second current status of a second connected device linked to the client device, the second current status of the second connected device being affected by performing or not performing a second function; determining a first graphical control element for controlling the first function of the first connected device and a second graphical control element for controlling the second function of the second connected device; determining a configuration for each of the first and second graphical control elements based on the first current status of the first connected device and the second current status of the second connected device; in response to determining that the request has been received, causing a display panel associated with the client device to provide a composite graphical assistant interface for controlling the first connected device and the second connected device, the composite graphical assistant interface including the first graphical control element and the second graphical control element arranged according to the determined configuration for each graphical control element; Including, The method comprises: accessing a profile for the user based on voice characteristics associated with the voice utterance, and determining at least one of the first function and the second function based on information available from the profile; 11. The method of claim 10, wherein determining the first graphical control element for controlling the first function is based on determining that the first function is not redundant in light of the first current status of the first connected device and the information available from the profile.

2. The method of claim 1 , wherein the information available from the profile indicates a historical tendency of the user to interact with at least one of the first connected device and the second connected device in a particular context.

3. 3. The method of claim 2, wherein the particular context corresponds to one or more of a status of the first connected device or the second connected device, a geographic location of the user, a time, an event, or a presence of one or more other users.

4. The method of claim 3 , wherein the particular context corresponds to the status of the first connected device and the status of the second connected device.

5. 2. The method of claim 1, wherein determining the first current status of the first connected device comprises requesting the first current status from a remote server device.

6. receiving a selection of the first graphical control element for controlling the first function; causing the client device to interact with the first connected device according to command data associated with the first function; 6. The method of claim 5, further comprising: wherein the first function is different from the function performed by the automated assistant.

7. The method of claim 6 , further comprising the step of requesting the command data from the remote server device.

8. 1. A system comprising: a memory operatively coupled to the one or more processors, the memory storing instructions for causing the one or more processors to perform a plurality of operations in response to execution of the instructions by the one or more processors, the plurality of operations comprising: An act of determining that a voice utterance detected via an automated assistant interface of a client device includes a request for the automated assistant to perform a function, the voice utterance being provided by a user; and in response to determining that the request was provided by the user, identifying the function corresponding to the request provided to the user; causing the automated assistant to execute the function corresponding to the request provided by the user based on the identification of the function; determining a first current status of a first connected device linked to the client device, the first current status of the first connected device being affected by performing or not performing a first function; determining a second current status of a second connected device linked to the client device, the second current status of the second connected device being affected by performing or not performing a second function; determining a first graphical control element for controlling the first function of the first connected device and a second graphical control element for controlling the second function of the second connected device; determining a configuration for each of the first and second graphical control elements based on the first current status of the first connected device and the second current status of the second connected device; and in response to determining that the request has been received, an operation of causing a display panel associated with the client device to provide a composite graphical assistant interface for controlling the first connected device and the second connected device, the composite graphical assistant interface including the first graphical control element and the second graphical control element arranged according to the determined configuration for each graphical control element. Including, The system comprises: the plurality of operations further comprising an operation of accessing a profile for the user based on voice characteristics associated with the voice utterance; determining at least one of the first function and the second function based on information available from the profile; The system, wherein determining the first graphical control element for controlling the first function is based on determining that the first function is not redundant in light of the first current status of the first connected device and the information available from the profile.

9. 10. The system of claim 8, wherein the information available from the profile indicates a historical tendency of the user to interact with at least one of the first connected device and the second connected device in a particular context.

10. 10. The system of claim 9, wherein the particular context corresponds to one or more of a status of the first connected device or the second connected device, a geographic location of the user, a time, an event, or a presence of one or more other users.

11. The system of claim 10 , wherein the particular context corresponds to the status of the first connected device and the status of the second connected device.

12. 10. The system of claim 8, wherein the act of determining the first current status of the first connected device comprises an act of requesting the first current status from a remote server device.

13. 1. A computer-readable storage medium configured to store instructions, comprising: The instructions, when executed by one or more processors, cause the one or more processors to perform a plurality of operations, the plurality of operations including: An act of determining that a voice utterance detected via an automated assistant interface of a client device includes a request for the automated assistant to perform a function, the voice utterance being provided by a user; and in response to determining that the request was provided by the user, identifying the function corresponding to the request provided to the user; causing the automated assistant to execute the function corresponding to the request provided by the user based on the identification of the function; determining a first current status of a first connected device linked to the client device, the first current status of the first connected device being affected by performing or not performing a first function; determining a second current status of a second connected device linked to the client device, the second current status of the second connected device being affected by performing or not performing a second function; determining a first graphical control element for controlling the first function of the first connected device and a second graphical control element for controlling the second function of the second connected device; determining a configuration for each of the first and second graphical control elements based on the first current status of the first connected device and the second current status of the second connected device; and in response to determining that the request has been received, an operation of causing a display panel associated with the client device to provide a composite graphical assistant interface for controlling the first connected device and the second connected device, the composite graphical assistant interface including the first graphical control element and the second graphical control element arranged according to the determined configuration for each graphical control element. Including, The computer-readable recording medium includes: the plurality of operations further comprising an operation of accessing a profile for the user based on voice characteristics associated with the voice utterance; determining at least one of the first function and the second function based on information available from the profile; A computer-readable storage medium, wherein determining the first graphical control element for controlling the first function is based on determining that the first function is not redundant in light of the first current status of the first connected device and the information available from the profile.

14. 14. The computer-readable medium of claim 13, wherein the information available from the profile indicates a historical tendency of the user to interact with at least one of the first connected device and the second connected device in a particular context.

15. 15. The computer-readable storage medium of claim 14, wherein the particular context corresponds to one or more of a status of the first connected device or the second connected device, a geographic location of the user, a time, an event, or a presence of one or more other users.

16. 16. The computer-readable medium of claim 15, wherein the particular context corresponds to the status of the first connected device and the status of the second connected device.

Images