Providing composite graphical assistant interface for controlling various connected devices

Abstract

The invention relates to providing a composite graphical assistant interface for controlling various connected devices. Methods, apparatus, systems, and computer-readable media are provided for customizing a composite graphical assistant interface for interacting with a plurality of different connected devices. A composite graphical assistant interface can be generated in response to a user providing a request for an automated assistant to cause a connected device to perform a particular function. In response to the automatic assistant receiving the request, the automatic assistant can identify other connected devices, as well as other functions that can be performed by the other connected devices. Other functionality can then be mapped to various graphical control elements to provide a composite graphical assistant interface from which a user can interact with different connected devices. Each graphical control element can be arranged to reflect how each connected device is operating while presenting the composite graphical assistant interface.

Description

Description of the case This application is a divisional application of the patent application with application number 201980039852.2 (PCT / US2019 / 016371), which was submitted to the China Patent Office on December 14, 2020, with an international application date of February 1, 2019, and the invention name is "Providing a composite graphical assistant interface for controlling various connected devices." Background Art

[0001] Humans may participate in human-machine conversations with interactive software applications referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, humans (who may be referred to as "users" when they interact with the automated assistant) may provide commands and / or requests to the automated assistant using spoken natural language input (i.e., utterances) and / or by providing textual (e.g., typed) natural language input, which in some cases may be converted to text and then processed.

[0002] Some automated assistants can be used to control Internet of Things (“IoT”) devices. However, many IoT devices can come with their own corresponding applications. As a result, users who have acquired multiple different IoT devices may be assigned the task of installing various third-party applications on their personal computing devices. Because applications may typically require memory, network bandwidth, updates, and / or other resources, it may be inefficient to have a variety of similar applications on a single device. Although users can rely on automated assistants to mediate interactions between users and specific IoT devices, such interactions may frequently require converting spoken words from users from audio to text. In order to cause the conversion of audio to text, audio data must often be sent over the network, thereby consuming significant network bandwidth. In addition, when the user is surrounded by other people who are talking or otherwise in an environment with background noise, mediating interactions in this manner may prove to be unreliable. As a result, when requests to perform specific functions on various IoT devices are ultimately not recognized or otherwise not fulfilled, the conversation session between the user and the automated assistant may waste computing resources and network resources. Summary of the invention

[0003] The present disclosure generally relates to methods, apparatus, and computer-readable media (transitory and non-transitory) for providing a composite graphical assistant interface through which multiple different devices are controlled. The composite graphical assistant interface can be used to control a variety of different devices even though the different devices are manufactured by different third parties. In this way, users do not have to install multiple different third-party applications to control their corresponding IoT devices. Instead, users can rely on their automated assistants to identify specific scenarios in which the user wants to control a specific IoT device, and provide a composite graphical assistant interface that includes suggestions for controlling the specific IoT device.

[0004] For example, a user may enter their kitchen, which may include a variety of different devices, such as a client device that provides access to an automated assistant and various connected devices that communicate with the client device. The connected devices may include a stereo system, an oven, a smart light, and a coffee maker. Each connected device may be manufactured by a different third-party manufacturer, at least relative to the manufacturer of the automated assistant. Typically in the morning when the user enters the kitchen, the user may provide a command such as "Assistant, turn on the lights in the kitchen". The user may then access various applications or manually interact with the connected devices to enable other functions to be performed. Although the user may verbally initialize the automated assistant to enable each function to be performed (e.g., "Assistant, turn on the coffeemaker...Assistant, play some music"), this will require additional speech processing, thereby consuming computing and network resources. In addition, if the user is required to access a different application to control each device, each application may require a certain amount of network connectivity, further consuming computing and network resources inefficiently. To provide more efficient use of these resources, the automated assistant may provide a composite graphical assistant interface in response to receiving an initial request to perform a function and / or in response to a user selection at a particular assistant interface.

[0005] For example, in response to receiving a spoken utterance such as "Assistant, turn on the kitchen lights", the automated assistant can identify the current state of other connected devices in the kitchen to facilitate identification of specific functions recommended to the user. For example, if it is determined that the coffee machine in the kitchen is on when the spoken utterance is received, the automated assistant can omit any reference to the coffee machine in the composite graphical assistant interface being compiled. In this way, the automated assistant will not make redundant suggestions, but can make room for more appropriate suggestions on the composite graphical assistant interface. In addition, if the automated assistant determines that the stereo system in the kitchen is not currently playing music, the automated assistant can determine the functions for controlling the stereo system that are recommended to the user. For example, the automated assistant can provide a composite graphical assistant interface that includes graphical control elements for turning on the stereo system and / or changing the volume of the stereo system. The composite graphical assistant interface can be provided at a display panel that is integral to the client device, or at a display panel associated with the client device (e.g., a tablet device also located in the kitchen). In this way, the user has a less computationally intensive modality through which to turn on the stereo system, and the modality can be provided to the user in a timely manner in response to the user's initial spoken utterance.

[0006] In some embodiments, suggestions for specific functions performed by a connected device can be based on data indicating a user's historical tendency to interact with a specific connected device in a specific context. For example, a user may have a historical tendency to turn on their stereo system when their kitchen lights are turned on or shortly after their kitchen lights are requested to be turned on. When such a context subsequently occurs, the automated assistant may provide a composite graphical assistant interface including graphical control elements at a display panel in the kitchen or another suitable display panel for controlling the stereo system. In addition, the automated assistant may identify other connected devices associated with the user, requests from the user, the context of the request, and / or any other information associated with the user, so as to provide further suggestions at the composite graphical assistant interface. For example, a user may also have a historical tendency to preheat the kitchen oven at night after turning on the kitchen lights and music from the stereo system. When such a context subsequently occurs, the automated assistant may provide a composite graphical assistant interface including graphical control elements for controlling one or more functions of their oven such as baking temperature and timer settings, as well as graphical control elements for controlling the features of their kitchen lights and / or stereo systems. In this way, the composite graphical assistant interface generated by the automated assistant may provide controls for a variety of different connected devices manufactured by a variety of different manufacturers.

[0007] The above description is provided as an overview of some embodiments of the present disclosure. Further descriptions of these and other embodiments are described in more detail below.

[0008] In some embodiments, a method implemented by one or more processors is described as including operations such as determining that a spoken utterance detected via an automated assistant interface of a client device includes a request for a connected device to perform a function. The connected device may be linked to the client device, and the spoken utterance is provided by a user. The method may further include: identifying a function corresponding to the received request in response to receiving the request; causing the connected device to perform the function corresponding to the received request based on identifying the function; and determining, by the automated assistant, another function for recommending to the user based on data accessible to the automated assistant in response to receiving the request. The other function is different from the identified function and can be performed by another connected device. The method may also include: identifying a graphical control element for controlling the other function of the other connected device based on determining the other function; and causing a composite graphical assistant interface to be provided at a display panel associated with the client device in response to receiving the request, the composite graphical assistant interface including the identified graphical control element. The method may further include: determining that a selection of the graphical control element is received at the composite graphical assistant interface; and causing the other connected device to perform the other function in response to determining that the selection of the graphical control element is received.

[0009] In some embodiments, the data identifies a current state of the other connected device, and in response to the selection of the graphical control element, causes the current state to change to a different state. In some embodiments, determining the other function includes: determining, based on the data, that a specific function is at least partially responsible for the current state of the other connected device; and selecting the other function for recommendation to the user based at least in part on the specific function being different from the other function. In some embodiments, determining the other function further includes: determining one or more slot values ​​of the function that is at least partially responsible for the current state of the other connected device; and identifying one or more parameters of one or more candidate functions that can affect the current state of the connected device. Selecting the other function for recommendation to the user based at least on the specific function being different from the other function may include: selecting the other function from the one or more candidate functions based on whether the other function includes a parameter that is different from another parameter of the specific function.

[0010] In some implementations, the method may include determining a speakable command that, when provided to the automated assistant interface, initializes the automated assistant to cause the another function of the another connected device to be performed. Causing the composite graphical assistant interface to be provided at the display panel may include causing the display panel to show the speakable command.

[0011] In some embodiments, determining another function for recommending to the user based on data accessible to the automated assistant includes: identifying the one or more additional connected devices from a plurality of connected devices based on a stored device topology indicating a relationship of the one or more additional connected devices to the client device or the connected device; and identifying one or more additional functions that can be performed by the one or more additional connected devices. The another function can be determined from the identified one or more additional functions. In some embodiments, the data can indicate a state of the one or more additional connected devices based on a previous execution or non-execution of the another function, and the configuration of the graphical control element at the composite graphical assistant interface can be based on the indicated state of the one or more additional connected devices.

[0012] In other embodiments, a method implemented by one or more processors is described as including operations such as determining that a spoken utterance detected via an automated assistant interface of a client device includes a request for an automated assistant to perform a function, wherein the spoken utterance is provided by a user. The method may also include identifying a function corresponding to the request provided by the user in response to determining that the request is provided by the user; and causing the automated assistant to perform the function corresponding to the request provided by the user based on identifying the function. The method may further include determining a first current state of a first connected device linked to the client device. The first current state of the connected device may be affected by the execution or non-execution of a first function. The method may also include determining a second current state of a second connected device linked to the client device. The second current state of the second connected device may be affected by the execution or non-execution of a second function. The method may further include determining a first graphical control element for controlling a first function of the first connected device and a second graphical control element for controlling a second function of the second connected device; and determining a configuration of each of the first graphical control element and the second graphical control element based on the first current state of the first connected device and the second current state of the second connected device. The method may also include, in response to determining that the request is received, causing a display panel associated with the client device to provide a composite graphical assistant interface for controlling the first connected device and the second connected device. The composite graphical assistant interface may include a first graphical control element and a second graphical control element, the first graphical control element and the second graphical control element being arranged according to the determined configuration of each graphical control element.

[0013] In some embodiments, the method may include accessing a profile of the user based on voice characteristics associated with the spoken utterance. Identifying at least one of the first function and the second function may be based on information available from the profile. In some embodiments, the profile may indicate a historical tendency of the user to interact with at least one of the first connected device and the second connected device within a specific context. In some embodiments, the specific context may correspond to one or more of: a state of the first connected device or the second connected device, a geographic location of the user, a time, an event, or the presence of one or more other users.

[0014] In some embodiments, determining a first current state of the first connected device may include requesting the first current state and command data associated with a first function from a remote server device. The first function may be different from the function performed by the automated assistant. In some embodiments, the method may include: receiving a selection of the first graphical control element for controlling the first function; and causing the client device to interact with the first connected device according to the command data associated with the first function.

[0015] In yet other embodiments, a method implemented by one or more processors is described as including operations such as determining that a user has provided a request for an automated assistant to perform a function. The request may be received at an automated assistant interface of a client device. The method may further include causing the automated assistant to perform the function corresponding to the request in response to determining that the user has provided the request; and determining a user preference in response to determining that the user has provided the request, the user preference characterizing the user's historical tendency to control another function of a connected device in a context associated with the function performed by the automated assistant. The connected device may be linked to the client device. In some embodiments, the method may include: identifying the connected device and the other function of the connected device based on determining the user preference; and determining a current state of the connected device based on identifying the connected device. The current state may be affected by the execution or non-execution of the other function. The method may also include determining a graphical control element for controlling the other function of the connected device based on identifying the other function and the current state of the connected device. The graphical control element may indicate the ability of the graphical control element to cause the other function to be performed by the connected device, thereby causing the current state of the connected device to change to a different state. The method may further include, in response to determining that the user has provided the request, causing a display panel associated with the client device to provide a composite graphical assistant interface including the graphical control element.

[0016] In some embodiments, the method may include determining an optional suggestion element associated with a separate connected device in the connected devices based on the user preference. Causing the display panel to provide the composite graphical assistant interface may include causing the composite graphical assistant interface to include the optional suggestion element. In some embodiments, the method may include determining that a selection of the optional suggestion element is received at the composite graphical assistant interface; and causing one or more graphical control elements to be presented on the display panel to control one or more functions of the separate connected device. In some embodiments, the method may include determining a speakable command that, when received at the automatic assistant interface of the client device, invokes the automatic assistant to cause the separate connected device to perform the one or more functions of the separate connected device. Causing the display panel associated with the client device to provide the composite graphical assistant interface may include causing the composite graphical assistant interface to include a natural language representation of the speakable command.

[0017] In some embodiments, the method may include determining a result state of the connected device in response to causing the connected device to perform the function. The result state is affected by the performance of the function. In addition, causing the display panel associated with the client device to provide the composite graphical assistant interface includes causing the composite graphical assistant interface to include a graphical interface element indicating the result state of the connected device.

[0018] In some implementations, the user preference characterizes a historical tendency of the user to control the other function based at least on a previous interaction between the user and the automated assistant. In some implementations, the previous interaction corresponds to a conversation session between the user and the automated assistant prior to the user selecting the graphical control element. During the conversation session, the user provides a request to cause the connected device to perform the other function.

[0019] In other embodiments, a method implemented by one or more processors is described as including operations such as determining that a user has provided a request for an automated assistant to perform a function. The request may be received at an automated assistant interface of a client device. The method may also include: causing the automated assistant to perform the function corresponding to the request in response to determining that the user has provided the request; and determining a user preference in response to determining that the user has provided the request, the user preference characterizing the user's historical tendency to control another function of a connected device in a context associated with the function performed by the automated assistant. The connected device may be linked to the client device. In some embodiments, the method may include identifying the connected device and a desired state of the connected device based on determining the user preference; and determining a current state of the connected device based on identifying the connected device. The current state may be affected by the execution or non-execution of the other function. The method may further include: when the desired state is different from the current state of the connected device: determining a graphical control element for controlling the other function of the connected device based on identifying the other function and the current state of the connected device, wherein the graphical control element indicates the ability of the graphical control element to cause the other function to be performed by the connected device, thereby causing the current state of the connected device to change to the desired state. Additionally, the method may include causing a display panel associated with the client device to provide a composite graphical assistant interface including the graphical control element in response to determining that the user has provided the request.

[0020] In some embodiments, the method may include: when the expected state is the same as the current state of the connected device: in response to determining that the user has provided the request, causing a display panel associated with the client device to provide a specific composite graphical assistant interface, wherein the specific composite graphical assistant interface omits a specific graphical control element for controlling the other function of the connected device.

[0021] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform one or more of the methods such as described above and / or described elsewhere herein. Still other embodiments may include a system of one or more computers and / or one or more robots including one or more processors operable to execute stored instructions to perform one or more of the methods such as described above and / or described elsewhere herein.

[0022] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Illustrated is a system for providing a composite graphical assistant interface for interacting with connected devices.

[0024] Figure 2A and Figure 2B Illustrated is a view of a composite graphical assistant interface provided with a plurality of different graphical control elements for controlling a plurality of different connected devices.

[0025] Figure 3A and 3B The illustration shows a view in which a user is provided with suggestions for controlling one or more connected devices based on interactions between the user and the automated assistant and / or the current state of the one or more connected devices.

[0026] Figure 4A and Figure 4B Illustrated is a method for providing a composite graphical assistant interface that includes suggestions for functions that can be performed by multiple different connected devices.

[0027] Figure 5 Illustration of a method for replacing a composite graphical assistant interface with graphical control elements for controlling connected devices.

[0028] Figure 6 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0029] Figure 1A system 100 for assembling a composite graphical assistant interface for interacting with connected devices is illustrated. The composite graphical assistant interface can be generated in part by an automated assistant 118 provided at one or more computing devices such as a client device 102 (e.g., assistant device 128), an additional client device (e.g., tablet device 150), and / or a server device 112. A user 120 can interact with a client automated assistant 104 via an assistant interface 110, which can be a microphone, a camera, a touch screen display, a user interface, and / or any other means capable of providing an interface between a user and an application. For example, the user 120 can initialize the client automated assistant 104 by providing speech, text, and / or graphical input to the assistant interface 110, and / or the tablet device 150, to cause the client automated assistant 104 to perform an operation (e.g., provide data, control peripherals, access an agent, etc.). The client device 102 can include a display device, which can be a display panel including a touch interface for receiving touch input and / or gestures for allowing the user to control an application of the client device 102 via the touch interface. Alternatively, client device 102 may be an assistant device 128 that does not include a display panel, but still provides one or more interfaces through which user 120 may interact with client automated assistant 104 (eg, through spoken utterances provided by user 120 into a microphone at assistant device 128 ).

[0030] The client device 102 can communicate with the server device 112 over a network 122, such as the Internet. The client device 102 can offload computing tasks to the server device 112 in order to conserve computing resources at the client device 102 and / or facilitate utilizing more robust resources on the server device 112. For example, the server device 112 can host the automated assistant 118, and the client device 102 can transmit input received at one or more assistant interfaces to the server device 112. However, in some embodiments, the automated assistant 118 can be fully hosted at the client device 102. In various embodiments, all or less than all aspects of the client automated assistant 104 can be implemented on the client device 102. In some of those embodiments, various aspects of the automated assistant 118 are implemented via the client automated assistant 104 of the client device 102 and interface with the server device 112, which implements other aspects of the client automated assistant 104. The server device 112 can optionally serve multiple users and their associated assistant applications via multiple threads. In some embodiments where all or less than all aspects of automated assistant 118 are implemented via client-side automated assistant 104 of client device 102, client-side automated assistant 104 can be an application separate from the operating system of client device 102 (e.g., installed "on top of" the operating system) - or can alternatively be implemented directly by the operating system of client device 102 (e.g., considered an application of the operating system, but integral to the operating system).

[0031] In some embodiments, the server device 112 can include a speech-to-text engine 116 that can process audio data received at the assistant interface 110 to identify text embodied in the audio data. The process for converting the audio data to text can include a speech recognition algorithm that can employ a neural network, a word2vec algorithm, and / or a statistical model to identify an audio data set corresponding to a word or phrase. The text converted from the audio data can be parsed by the text parser engine 108 and made available to the automated assistant 104 as text data that can be used to generate and / or recognize command phrases from users and / or third-party applications.

[0032] The user 120 can use their client device 102 and / or the client automated assistant 104 to interact with the first connected device 132 and / or the second connected device 146. Each connected device can be one or more IoT devices that can connect to or communicate with the client device 102 and / or the tablet device 150 via a local area network and / or a wide area network. For example, the client device 102, the tablet device 150, the first connected device 132, and / or the second connected device 146 can all connect to a Wi-Fi network in a home or other environment. In order for the user 120 to interact with the first connected device 132, the user 120 can initialize an application associated with the first connected device 132. However, if the user 120 frequently interacts with multiple different connected devices, the user 120 may need to operate a variety of different applications in order to control each connected device. To more efficiently use computing resources and prevent wasting network bandwidth, client automated assistant 104 and / or automated assistant 118 can provide a composite graphical assistant interface from which a user can interact with one or more different connected devices (e.g., first connected device 132 and second connected device 146).

[0033] In order to generate and / or provide a composite graphical assistant interface, the user 120 can provide a spoken utterance 130 to the assistant interface 110. The spoken utterance can embody a request indicating that the user 120 wants the first connected device 132 to perform a specific function. Audio data corresponding to the spoken utterance 130 can be transmitted from the client device 102 to the server device 112. The audio data can then be processed by the voice-to-text engine 116 and by the text parser engine 114. The text generated by the processing of the audio data can be used by the function recognition engine 142 to identify one or more functions that the user 120 is referring to in his spoken utterance 130. For example, the function recognition engine 142 can compare the text from the spoken utterance 130 with the function data 136 that is made available at the connection device server 134. The function data 136 can provide a comprehensive list of functions that one or more connection devices (e.g., the first connection device 132 and the second connection device 146) can perform. The function recognition engine 142 can compare the text with the function data 136 to identify one or more functions that can be performed by the first connection device 132 and that the user is referring to. The functional data 136 may additionally or alternatively be stored locally with respect to the server device 112 .

[0034] In response to determining that the text corresponds to one or more functions, the automated assistant 118 or the client automated assistant 104 can cause the first connected device 132 to perform the one or more identified functions. For example, when the first connected device 132 is an IoT device such as, but not limited to, an oven, the functions can include preheating the oven to a specific temperature. Commands for causing the functions to be performed at the first connected device 132 can be provided from the server device 112 to the first connected device 132. Alternatively, the commands for causing the first connected device 132 to perform the functions can be provided from the server device 112 to a connected device server 134, which can communicate with the first connected device 132 to cause the connected device 132 to perform the functions. Alternatively, the client device 102 can communicate the commands to the first connected device 132, or the server device 112 can communicate the commands to the client device 102, which can communicate with the first connected device 132 to cause the first connected device 132 to perform the functions.

[0035] In response to receiving the spoken utterance 130 to cause the first connected device 132 to perform a function, the automated assistant can cause a composite graphical assistant interface to be provided at a display device (e.g., tablet device 150, television, laptop, mobile phone) associated with the client device 102. To generate the composite graphical assistant interface, the automated assistant 118 can identify one or more other functions that can be performed by the first connected device 132 from the function data 136. In some implementations, in response to determining that the spoken utterance 130 is received at the client device 102, the automated assistant can determine one or more other functions that can be performed by the second connected device 146 (e.g., a different IoT device), the tablet device 150, and / or any other device associated with the client device 102. For example, the function data 136 can include one or more function lists corresponding to devices associated with the user 120, the client device 102, and / or the first connected device 132. The automated assistant can determine the one or more functions to provide suggestions or recommendations for other functions that the user 120 may want to perform based at least on the user 120's request to cause the first connected device 132 to perform a function.

[0036] For example, in some embodiments, a user's profile may be identified based on voice characteristics of a spoken utterance received at the assistant interface 110. The profile may indicate preferences associated with each of a plurality of specific connected devices associated with the user, such as preferences learned based on historical interactions of the user with the connected devices. In addition, the profile may identify a group of connected devices that the user 120 prefers to interact with simultaneously, sequentially, and / or within a particular context or multiple contexts. For example, the user 120 may request the automated assistant to cause the first connected device 132 to perform a first function and the second connected device 146 to perform a second function, and the profile may identify the first function and the second function as being related based on, for example, having occurred one or more times in close proximity to each other in time. Thereafter, when the user 120 interacts with the automated assistant to cause the first connected device 132 to perform the first function, the automated assistant may access the profile to identify the second function based on its stored relationship with the first function, and provide the user 120 with optional suggestions for causing the second connected device 146 to perform the second function.

[0037] In some embodiments, historical interactions between user 120 and the automated assistant may be processed with permission from user 120 to determine a historical tendency of user 120 to interact with a particular connected device in a particular context. For example, user 120 may have a history of interacting with two IoT devices in the morning, so at least until the end of the morning, the automated assistant may be reminded that when user 120 interacts with at least one of the two IoT devices in the morning, user 120 will tend to interact with another IoT device. Additionally or alternatively, the automated assistant may provide suggestions based on the location of user 120. For example, user 120 may have a history of causing their oven (e.g., first connected device 132) to perform a first function and causing their tablet device 150 to perform a second function when user 120 is in their kitchen 148. Therefore, when user 120 causes first connected device 132 to perform the first function and determines that user 120 is in kitchen 148, the automated assistant may cause a selectable suggestion element to be provided on the composite graphical assistant interface. The selectable suggestion element may serve as a suggestion to user 120 to cause the second function to be performed at second connected device 146. It can be determined that the user is in kitchen 148 based on, for example, client device 102 or tablet device 150 being used by the user to perform the first function. For example, "kitchen" can be optionally identified as an explicit location based on device 102 and / or device 150 being assigned kitchen tag 148. Alternatively, the user's location can more generally be a location corresponding to client device 102 or tablet device 150. In some embodiments, the context can be whether user 120 is participating in a particular event, such as an event identified in calendar data accessible to the automated assistant. In some embodiments, the context can be whether there are one or more other users, different from user 120, at a particular location.

[0038] When the automated assistant has identified other functions with which to suggest the user 120, the identified other functions can be used as a basis for identifying graphical control elements 140 therefrom. For example, functions such as turning a fan on and off can be controlled by a graphical control element 140 corresponding to a two-way switch. In addition, functions such as adjusting the temperature of an oven or the volume of music can be controlled using a graphical control element 140 corresponding to a dial that can be turned to select a value for a range of values. The automated assistant 118 or client automated assistant 104 can map the identified other functions to graphical control elements 140, which can be stored at the server device 112 or otherwise accessible by the server device 112.

[0039] The composite interface engine 144 can map the graphical control element 140 to the identified other functions in order to generate a composite graphical assistant interface. In some embodiments, the automated assistant 118 can access state data 138 corresponding to the state of the first connected device 132. The state data 138 can indicate a device state that may be affected by the execution or non-execution of one or more functions of the first connected device 132 and / or the second connected device 146. For example, when no function of the first connected device 132 is executed, the state of the first connected device 132 can be "off". Alternatively, the state can correspond to a data table or string indicating the status or parameters of a specific function of the first connected device 132 and / or the second connected device 146. The state of each of the first connected device 132 and / or the second connected device 146 can be used as a basis for determining a configuration for each graphical control element 140 that has been mapped to the function of the first connected device 132 and / or the second connected device 146. For example, if the first connection device 132 includes a heating element in an off state, and the function of the first connection device 132 is to turn on the heating element, a graphic control element corresponding to the bidirectional switch can be provided at the composite graphical assistant interface and configured in a manner to show that the heating element is off. In other words, when the composite graphical assistant interface is presented at the tablet device 150, each graphic control element can reflect each current state of the first connection device 132 and the second connection device 146.

[0040] In some embodiments, the arrangement and / or selection of graphical control elements for the composite graphical assistant interface can be based on data associated with the interaction between the user 120 and the automated assistant and / or the first connected device 132. For example, a specific group of graphical control elements corresponding to a specific group of functions can be selected for the composite graphical assistant interface based on the fact that the user has previously caused such functions to be executed within a threshold time period. Alternatively or additionally, a specific group of graphical control elements corresponding to a specific group of functions can be selected for the composite graphical assistant interface based on how the functions affect a specific user, a specific location, a specific device, and / or any other relationship that can be exhibited by the functions of the device. For example, a composite graphical assistant interface generated for a user can include graphical control elements corresponding to functions of multiple different connected devices that affect the home in which the user 120 is currently located. Thus, the selection of graphical control elements can be based on the user's location, an identifier of a connected device, the user's calendar schedule, and / or any other data that can be used to estimate the user's geolocation data.

[0041] In some embodiments, user 120 is able to provide spoken utterance 130 to an assistant device that lacks an integrated display panel or is otherwise unable to easily present a composite graphical assistant interface at assistant device 128. In such embodiments, the automated assistant is able to receive spoken utterance 130 at the assistant device and cause the function requested by user 120 to be performed by connected device 132. However, in response to receiving a request to perform a function and determining that a display panel is not available at assistant device 128, the automated assistant is able to identify one or more candidate devices. Candidate devices can be identified based on the ability of the candidate device to display an assistant interface, the proximity of the candidate device to user 120, the proximity of the candidate device to the assistant device, the proximity of the candidate device to the first connected device 132, and / or the user 120's preference for a particular candidate device. For example, when the automated assistant determines that a display panel is not available at assistant device 128 and determines that tablet device 150 or a television is the next closest display device to assistant device 128, the automated assistant is able to cause tablet device 150 or the television to present a composite graphical assistant interface. In this way, because the user 120 may not always be near the display panel when they are requesting that a particular function be performed by the connected device 132, the automated assistant is able to still cause the composite graphical assistant interface to be generated at a nearby display device. In some embodiments, the automated assistant is able to cause the assistant device 128 to audibly notify the user 120 of the location of the composite graphical assistant interface in response to the user 120 providing a request for the connected device 132 to perform a function. For example, in response to the user 120 providing the verbal utterance "Assistant, please pre-heat my oven", the automated assistant is able to cause the assistant device 128 to audibly output the response "Ok, I've provided an interface with additional controls at your tablet device". The automated assistant is thus able to cause the user 120 to be aware that if they go to retrieve their tablet device, they will be able to find additional controls for the first connected device 132 and the second connected device 146.

[0042] Figure 2A and Figure 2BThe views 200 and 210 of the illustrated composite graphical assistant interface 222 are provided with a plurality of different graphical control elements for controlling a plurality of different connected devices. The composite graphical assistant interface 222 may be provided to the user 202 in response to the user 202 providing a spoken utterance 208 to the automated assistant via the automated assistant interface 214. The automated assistant interface 214 may be, for example, at least one microphone capable of providing a signal in response to the user 202 providing an audible spoken utterance 208 to the automated assistant interface 214. For example, the spoken utterance 208 may be "Assistant, increase the temperature in my kitchen". In response to receiving the spoken utterance 208, the automated assistant may determine the function and / or connected device with which the user 202 is attempting to interact, and may cause the connected device to perform the function that the user 202 is requesting to perform. For example, in response to receiving the spoken utterance 208, the automated assistant may cause the thermostat in the home of the user 202 to adjust at least the temperature of the kitchen in the home.

[0043] In order to alleviate unnecessary processing of subsequent spoken utterances provided from user 202 and thereby preserve computing and network resources, the automated assistant may provide functional suggestions on a display panel 206 of client device 204 or a separate device. For example, in response to receiving the spoken utterance 208, the automated assistant, client device 204, and / or remote server device may access data from which recommendations for other functions for controlling other connected devices may be provided. A connected device may be a device that is linked to or otherwise capable of communicating with client device 204 and / or the automated assistant. The data may provide information about the state of each connected device, and the suggestions provided at the composite graphical assistant interface 222 may be based on the specific state of a particular connected device. For example, the state of the kitchen lights may be "off" when user 202 provides the spoken utterance 208, and therefore, because user 202 has expressed an interest in changing the kitchen environment, user 202 may also be interested in changing the state of the kitchen lights. Specifically, the lights in the kitchen may have non-binary parameters associated with their brightness, and therefore, in response to determining that user 202 may also be interested in adjusting the kitchen lights, the automated assistant may select a dial graphical control element that may serve as a kitchen brightness control 218.

[0044] In some embodiments, kitchen brightness control 218 may be selected for composite graphical assistant interface 222 based on the kitchen light being in a common location with the kitchen for which user 202 has explicitly expressed interest in changing its temperature. Alternatively or additionally, kitchen brightness control 218 may be selected for recommendation to user 202 at composite graphical assistant interface 222 based on user 202's historical tendency to modify kitchen temperature and kitchen lights in a particular context. User 202's historical tendency may be indicated as part of a user profile or preference data characterizing the tendency based on previous interactions between user 202 and an automated assistant and / or user 202 and one or more connected devices. In some embodiments, such interactions may occur within a particular context, such as a particular time 212 (e.g., after 6:00 AM), and thus, the automated assistant may generate composite graphical assistant interface 222 based on the particular context. Alternatively or additionally, the context may be characterized by one or more states of one or more corresponding connected devices.

[0045] Alternatively or additionally, the automated assistant may identify a request from a user and compare the request with available data (e.g., preference data and / or historical interaction data) to determine one or more functions to suggest to the user. When the automated assistant has identified one or more functions to suggest, the automated assistant may identify each connected device corresponding to each of the one or more identified functions. The state of each connected device may then be determined in order to ensure that the suggestion for the function to be performed is not redundant or will cause the state of the connected device to not change. For example, a user may typically turn on the kitchen lights in the morning and then turn on their coffee machine. In this scenario, the automated assistant may provide a verbal utterance to initialize the automated assistant to turn on the kitchen lights (e.g., "Assistant, turn on my kitchen lights"). Based on data indicating historical interactions in which the user turns on both the kitchen lights and the coffee machine, the automated assistant may identify the coffee machine and / or a function for turning on the coffee machine in response to receiving a request to turn on the kitchen lights. In order to ensure that the function suggestion is not unreasonable, when the user turns on the kitchen lights, the automated assistant may compare the current state of the coffee machine with the state of the connected device that the user typically expects. If the current state of the coffee machine reflects the desired state, the automated assistant may omit providing a graphical control element on the composite graphical assistant interface for controlling the coffee machine. However, if the current state of the coffee machine does not reflect the desired state (e.g., the coffee machine is off and therefore does not exhibit the desired state), the automated assistant may select a graphical control element that controls the function of turning on the coffee machine. In this way, the composite graphical assistant interface may be optimized to provide graphical control elements that will cause changes that the user would typically expect, rather than providing suggestions for connected devices that may have already occurred.

[0046] For example, the composite graphical assistant interface 222 may include a variety of different graphical control elements, which are selected for the composite graphical assistant interface 222 based on the status of the specific connected devices to which they correspond. For example, because the user 202 has asked the automated assistant to turn up the kitchen temperature, the automated assistant may recognize that the user may be interested in controlling other connected devices in the kitchen such as an oven and / or a coffee machine. Because each of these connected devices is turned off (as shown in the configuration of the oven temperature control 216 and the coffee machine control 220), these connected devices may be candidate connected devices for control suggestions. In addition, the user 202 may have a history of increasing the kitchen temperature, adjusting the lights in the kitchen, preheating its oven, and turning on its coffee machine within a threshold time range after 6:00 AM. Although each of these devices and / or their corresponding control applications are provided by different third parties, the automated assistant can compile a composite graphical assistant interface 222 so as to allow the user 202 to control each device from a single interface.

[0047] In some embodiments, as a result of user 202 selecting one of the graphical control elements at composite graphical assistant interface 222, the state of the corresponding connected device can be changed. For example, based at least on kitchen brightness control 218 being at a minimum level, the brightness of the kitchen lights is initially indicated as "off." When user 202 adjusts kitchen brightness control 218 to a higher level, the state of the kitchen lights can be updated to have an "on" state. In addition, the state of the kitchen lights can be updated to reflect a particular slot value generated in response to the adjustment of the graphical control element. For example, the slot value can be "60%," indicating that kitchen brightness control 218 is adjusted up to 60%, so the state of the kitchen lights can be updated to "kitchen_lights: status(lights[on], brightness

[60] , hue[null])." Thereafter, any other subsequent suggestions can be based on the updated state of the kitchen lights. For example, because the kitchen light now has a state of "on," the automated assistant may cause composite graphical assistant interface 222 to provide a graphical control element for modifying the hue of the kitchen light based at least on the kitchen light being on and the kitchen light's "hue" parameter being "null" or in a default setting.

[0048] In some embodiments, the composite graphical assistant interface 222 may provide a status indication 224 of a function that is performed in response to the user 202 providing the spoken utterance 208. In addition, in response to receiving the spoken utterance 208, the automated assistant may cause the composite graphical assistant interface 222 to provide a suggestion that is embodied in natural language that characterizes the spoken utterance. The spoken utterance may be a spoken utterance that, when provided to the automated assistant interface, initializes the automated assistant to cause one or more connected devices to perform one or more functions. For example, in response to receiving the spoken utterance 208, the automated assistant may identify a status of a connected device that is located within an area (e.g., a kitchen) affected by the spoken utterance 208. The automated assistant may then identify a natural language input that, when received on the automated assistant interface, will initialize the automated assistant to cause the connected device to perform a function. For example, when the connected device is a coffee maker, the automated assistant may recognize a previously employed spoken utterance for controlling the coffee maker, such as "Assistant, turn on the coffeemaker". In response to recognizing the aforementioned spoken utterance, the automated assistant may cause the graphical assistant interface 222 to present the spoken utterance as a selectable suggestion element that includes at least the text "turn on the coffeemaker." In this manner, the user 202 may select the selectable suggestion element to turn on the coffee maker, or speak the text as a spoken utterance, in order to initialize the automated assistant to cause the coffee maker to turn on. The automated assistant may also retrieve command data that may be used by the client device 204 to interact with a particular connected device. In this manner, when the automated assistant determines a particular recommendation to make to the user 202, the automated assistant may also locally provide data that will allow the suggested function to be performed by the connected device. As a result, in response to the user's selection of a recommendation, the automated assistant and / or the client device will not have to communicate with a remote server to obtain additional information about how to cause the connected device to perform a function.

[0049] Figure 3A and Figure 3BViews 300 and 344 illustrate user 302 providing suggestions for controlling one or more connected devices based on an interaction between user 302 and an automated assistant and / or a current state of one or more connected devices. For example, user 302 may walk into their kitchen 320 and initialize client automated assistant 330 via assistant interface 332 of client device 328 (e.g., assistant device 326). User 302 may initialize client automated assistant 330 by providing spoken utterance 322 such as "Assistant, play music in the kitchen." Client automated assistant 330 may receive spoken utterance 322 and cause spoken utterance 322 to be processed on client device 328 or a separate remote server to facilitate identifying an action or function that user 302 wishes to perform. Because client device 328 does not include a display panel, client device 328 can access the device topology to determine an appropriate device through which to provide a composite graphical assistant interface 306, from which any functions associated with the requested function (i.e., playing music) and / or any other functions associated with other connected devices identified in the device topology are further controlled.

[0050] For example, the stored device topology accessible to the client automated assistant 330 may identify the tablet device 316, the first connected device 318, and the second connected device 324 in the device topology that identifies the devices in the kitchen 320. The device topology may further identify the tablet device 316 as the only device with a display panel, so any composite graphical assistant interface to be presented to the user 302 when the user 302 is in the kitchen 320 may be provided on the tablet device 316. In addition, in order to generate a suitable composite graphical assistant interface, the automated assistant may identify the state of each connected device and / or determine the historical tendency of the user 302 to interact with one of the connected devices after the requested function (i.e., playing music) is performed. For example, because the current state of the first connected device 318 (e.g., oven) is off, the automated assistant may suggest to the user to control the oven by presenting the selectable suggestion element 310 at the composite graphical assistant interface 306. In addition, because the current state of the second connected device 324 (e.g., dishwasher) is off, the automated assistant may suggest to the user to control the dishwasher by presenting the selectable suggestion element 312 on the composite graphical assistant interface 306. In addition, the automated assistant can determine whether there are any other functions associated with the initially requested function of playing music, and provide graphical control elements at the composite graphical assistant interface 306 to control the other functions. For example, a function of turning on music, which can be executed by the client automated assistant 330 at the client automated assistant 330, can cause the status of the client device to be at least "client_device: status(music[on], volume[default])". In response to a parameter that the client device 328 is in a default mode, the automated assistant can provide graphical control elements for controlling the corresponding parameters. For example, the automated assistant can provide a kitchen volume control 314 to facilitate controlling the volume of music in the kitchen 320. However, if the user 302 did not provide a previous request corresponding to turning on music, the automated assistant can omit the kitchen volume control 314 from the composite graphical assistant interface 306.

[0051] Figure 3B It can be illustrated that in response to user 302 Figure 3A310 . For example, in response to user 302 selecting the “control oven” selectable suggestion element 310 , the automated assistant may provide a different composite graphical assistant interface 340 that includes a dial 334 for setting the temperature of the oven. The initial configuration of dial 334 may be “0” so as to reflect the current state of first connected device 132 and / or one or more current parameters of first connected device 132 . For example, the current state of first connected device 132 may be “oven: status(power[off], temperature[0])” and thus the configuration of dial 334 may reflect multiple current parameters of first connected device 318 . In other words, dial 334 is in its lowest state indicating that the oven is off and therefore the temperature is set to 0. These parameters may be consolidated and reflected on a single graphical control element to conserve space in composite graphical assistant interface 340 and provide a more intuitive modality through which more information is provided to user 302 . If the user 302 has selected the "control dishwasher" selectable suggestion element 312, the switch 336 can be presented on a different composite graphical assistant interface 340 based on the particular second connected device 324 having a current state of "off" and a single parameter to be controlled (on or off). In some embodiments, metadata corresponding to the requested function or the state of the requested function (e.g., playing music) can be presented as a status indication 342. In this way, the user 302 does not have to invoke the automated assistant again to determine the current state associated with their request, nor does the user 302 have to switch between different interfaces to determine such information. Instead, the automated assistant metadata can be presented using graphical control elements associated with other third-party devices to provide an integrated interface that will reduce waste of computing and network resources.

[0052] In some embodiments, because the current state of another connected device in the kitchen indicates that the parameters associated with the other connected device have been set, the automated assistant can omit providing a graphical control element for the connected device. For example, if the current state of the kitchen lights is "on" when the user 302 provides the spoken utterance 322, the automated assistant can confirm the current state of the kitchen lights and omit the function of suggesting turning off the lights. Such an omission can be based on one or more states of one or more devices in the kitchen or other areas of the home, and / or a historical tendency of the user 302 not to turn off their lights when they request to play music in the kitchen. In some embodiments, identifying the historical tendency can be based on the voice characteristics of the user 302, which allows the automated assistant to identify data associated with a specific user to be processed in order to make appropriate suggestions for one or more users. In some embodiments, the inclusion or omission of a particular graphical control element can be based on the specific context in which the user provides the spoken utterance 322, such as after or before a specific time 304, or at a threshold time period relative to a specific time, a geographic location, an event, and / or the presence of one or more users.

[0053] Figure 4A and 4B Methods 400 and 410 are illustrated for providing a composite graphical assistant interface including suggestions for functions that can be performed by multiple different connected devices. Method 410 may be a continuation of method 400, such as Figure 4A and Figure 4B. Methods 400 and 410 may be performed by one or more computing devices, applications, and / or any other means or modules capable of interacting with a connected device or an automated assistant. Method 400 may include an operation 402 of determining that a spoken utterance detected via an automated assistant interface of a client device includes a request for a connected device to perform a function. The automated assistant interface may be one or more of a microphone, a touch display panel, a camera, a sensor, and / or any other means by which a user may provide input to a device. In addition, the client device may be a computing device that may include a touch display panel integral to the computing device, or otherwise communicate with the computing device. For example, the client device may be an assistant device connected to a WiFi network, and the assistant device may communicate with a separate display panel (e.g., a television or other display device) via the WiFi network. Any device with which the client device communicates may be a connected device. For example, any IoT device in a home that includes a WiFi network connected to the client device may be considered a connected device. Alternatively or additionally, any device with which the client device communicates via a local area network, a wide area network, and / or the Internet may be a connected device. The automated assistant may communicate with the connected device by receiving commands from the user via the automated assistant interface and causing the client device to interact with the connected device. Alternatively or additionally, in response to the automated assistant receiving the commands, the automated assistant may cause a remote server to issue commands to the connected device. Alternatively or additionally, the automated assistant may cause the remote server to issue commands to a connected device server associated with the connected device so that the connected device server will cause the connected device to perform a specific function.

[0054] The method 400 may further include an operation 404 of identifying a function corresponding to the received request in response to receiving the request. The function may be identified by accessing a stored or available list of functions that can be performed by one or more connected devices in communication with the client device. In some embodiments, a remote server device that at least partially hosts the automated assistant may query a separate connected device server that may provide data identifying one or more functions that can be performed by the connected device. When the connected device is an IoT device, the function may be any executable operation that can be performed by the connected device, such as, but not limited to, providing information from a sensor, changing an input parameter of a hardware component of the connected device, causing the connected device to perform a network operation, causing the connected device to communicate with another connected device, and / or any other function that can be performed by the device. For example, the function, when executed by the connected device, may cause the connected device to turn off any lights that are controllable by the connected device (e.g., "function (connected_device: lights: {off})", where "off" is the slot value of the parameter "lights").

[0055] Method 400 may also include an operation 406 of causing the connected device to perform a function corresponding to the received request based on identifying the function. The client device or the automated assistant may cause the connected device to perform the function by transmitting command data to the connected device, or causing command data to be transmitted to the connected device. As a result of the connected device performing the function, the state of the connected device may be modified to reflect any changes caused by the execution of the connected device. The state of the connected device may be characterized by data such as "connected_device: status(lights[off], brightness[null], hue[null])", where a particular parameter or slot value may be empty or null as a result of a particular slot value (e.g., "off") provided to the connected device. In some embodiments, depending on the context in which the function is performed by the connected device, a profile or context associated with the user may be updated or otherwise modified to reflect the request for the function to be performed. In this manner, data corresponding to historical interactions between the user and the connected device and / or automated assistant may be subsequently processed to enhance future interactions between the user and the connected device and / or automated assistant. For example, when the function corresponds to a request to turn off the lights of a device at night, the profile can be updated to rank the particular function higher at night and / or otherwise indicate the user's historical tendency to request the performance of functions at night. Thereafter, the profile can be used as a basis for providing suggestions for functions to be performed for various different connected devices based on at least the user's historical tendency to request those functions or similar functions within a particular context. In some embodiments, the profile can indicate functions that should be suggested based on the state of one or more connected devices. In this way, the automated assistant can examine one or more states of one or more connected devices and determine the user's historical tendency to request particular functions based on the one or more states of the one or more connected devices.

[0056] Method 400 may further include an operation 408 in which the automated assistant determines, in response to receiving the request, another function for recommendation to the user based on data accessible to the automated assistant. The other function may be different from the function performed at operation 406 and may be performed by a connected device different from the connected device that performs the function at operation 406. The other function may be determined based on data accessible to the automated assistant and indicating the user's historical tendency to cause other functions to be performed or to interact with other connected devices in the current context. For example, the other function may be determined based on the current state of the connected device that performs the function at operation 406. Thus, when the current state of the connected device is "connected_device: status(lights[off], brightness [null], hue[null])", the suggested other function may be to turn on a security alarm of a security system (e.g., the other connected device) based on at least a previous interaction when the user turned off the lights - causing the state of the lights to change - and turned on their security system. By providing such a suggestion, computing resources and network resources may be conserved compared to a user issuing a voice command to cause such a function to be performed, resulting in speech processing.

[0057] Method 400 may proceed to method 410, such as Figure 4A and Figure 4B . The method 410 may include an operation 412 of identifying a graphical control element for controlling the other function of the other connected device based on determining the other function. The graphical control element may be selected from a plurality of graphical control elements stored in a memory accessible to the client device and / or the automated assistant. In some embodiments, each graphical control element may be stored in association with metadata indicating an input that can be represented by the graphical control element and / or an output that can be provided in response to a selection or gesture received by the graphical control element. For example, a graphical control element representing a bidirectional switch may be stored in association with metadata indicating that the graphical control element may receive touch input and / or swipe gestures, and / or that the graphical control element may provide a binary output, such as 1 or 0, and / or open or close. The memory may store a variety of different graphical control elements, each of which may provide two or more different outputs, and / or receive a variety of different inputs.

[0058] Method 410 may also include an operation 414 of providing a composite graphical assistant interface at a display device associated with a client device in response to receiving the request. The composite graphical assistant interface may be a graphical user interface having one or more optional elements, which can perform specific functions at a connected device when a selection and / or gesture is received on the composite graphical assistant interface. The composite graphical assistant interface may act as a modality through which a user can interact with an automated assistant and / or a plurality of different connected devices, although the connected devices are associated with other different third-party applications to control them. In this way, the user does not necessarily have to download and / or access a plurality of different applications to control their corresponding IoT devices, but can simply cause the composite graphical assistant interface to be provided on the display panel. In addition, the composite graphical assistant interface may be adapted by the automated assistant according to one or more states of one or more corresponding connected devices and / or the context in which the user is interacting with the automated assistant and / or the client device. In some embodiments, the composite graphical assistant interface may be associated with data including slot values ​​of specific graphical control elements presented on the composite graphical assistant interface. The slot value may be provided to a corresponding function, which may then be performed by a connected device corresponding to any corresponding graphical control element.

[0059] The method 410 may further include an operation 416 of determining that a selection of a graphical control element is received at the composite graphical assistant interface. The selection of the graphical control element may be determined based on data transmitted by a display panel in response to a user touching or otherwise providing a gesture on the display panel providing the composite graphical assistant interface. Any slot value corresponding to the determined selection of the graphical control element may then be provided to a device responsible for further enabling the function to be performed at the corresponding connected device, such as the connected device itself or a connected device server.

[0060] Method 410 may also include an operation 418 of causing the other connected device to perform the other function in response to determining that a selection of the graphical control element is received. The other function may be, for example, to turn on the security system at the user's home. The other function may be suggested to the user based at least on the historical tendency of the user to manually turn off their lights and turn on their security system. The other function may also be suggested based on the current state of the light (e.g., the light is off). Alternatively or additionally, the other function may be suggested based on the automated assistant determining that a request has been made to change the current state of the light to "off" and / or the current context of the request corresponds to the time of day, the user's geographic location, the state of one or more other devices, and / or any other information that can be used to enable the execution of computer functions. In response to executing the other function, the current state may be updated at a table or other data store including identifiers for the states of various connected devices. In this way, the automated assistant may generate additions based on updated combinations of states and as the states change over time. For example, since the security alarm is on and the light is off, the automated assistant may determine a graphical control element for controlling the communication device function so as to allow the user to place the communication device in silent mode by interacting with the graphical control element on a new composite graphical assistant interface.

[0061] Figure 5 The diagram illustrates a method 500 for replacing a composite graphical assistant interface with a graphical control element for controlling a connected device. The method 500 may be performed by one or more computing devices, applications, and / or any other apparatus or module capable of interacting with a connected device and / or an automated assistant. The method 500 may include an operation 502 of determining that a user has selected a graphical control element at a first composite graphical assistant interface provided at a display panel associated with a client device. The display panel may be connected to the client device as an integral part of the client device or as a device that communicates with the client device over a network. The first composite graphical assistant interface may be presented on the display panel so as to provide the user with a modality through which the user can control a variety of different connected devices, even though the connected devices are manufactured by the same or different entities. Thus, the first composite graphical assistant interface may include a plurality of different graphical control elements through which the user can control a variety of different connected devices.

[0062] Method 500 may also include an operation 504 of causing the connected device to perform a function corresponding to the selected graphical control element in response to determining that the user has selected the graphical control element. The connected device may be caused to perform the function via the client device and / or the automated assistant. For example, in response to the first composite graphical assistant interface receiving a selection of the graphical control element, the client device and / or the automated assistant may transmit data to the connected device to cause the connected device to perform the function. Alternatively or additionally, the client device may communicate the data to a remote server, and the remote server may communicate with the connected device to cause the connected device to perform the function.

[0063] Method 500 may further include an operation 506 of determining a user preference that characterizes a user's historical tendency to control another function of another connected device in a context associated with the connected device. User preferences may be data indicating a user's tendency or preference to control a specific function of one or more connected devices within a specific context. For example, a context may correspond to one or more states of one or more corresponding devices, a time of day, a user's gesture or gaze, a scheduled event, a correspondence with a previous interaction between the user and the automated assistant, and / or any other data that may be used as a basis for suggesting functions to be controlled by the user. In some embodiments, user preferences or other data may rank functions according to their popularity or the frequency with which a particular user controls a function within a specific context. In order to provide a ranking for a function, a learning model may be employed to receive input related to the function being performed and output data corresponding to the ranking of functions that should be suggested to the user. The learning model may be trained based on data characterizing the interaction between the automated assistant and the user and / or the user and one or more connected devices.

[0064] The method 500 may also include an operation of identifying the other function of the other connected device based on determining the user preference 508. The user preference data may identify the other function, a plurality of different connected devices, one or more parameters and / or slot values ​​corresponding to the other function, slot values ​​that have been previously provided for the other function in a particular context, and / or any other data that may be useful for performing a particular function.

[0065] The method 500 may also include an operation 510 of determining another graphical control element for controlling the other function of the other connected device based on identifying the other function. The other graphical control element may be selected from a memory accessible to the client device and / or the automated assistant and may be associated with metadata that provides at least some correspondence with the other identified function. For example, the function may be associated with a parameter that may receive two or more different slot values, as opposed to a binary parameter that may only take two values. As a result, the graphical control element that will be selected for controlling the graphical control element may be a graphical control element that is capable of providing an output that takes two or more different values, such as a dial or a scroll bar.

[0066] Method 500 may further include operation 512 of causing a display panel associated with the client device to replace the first composite graphical assistant interface with a second composite graphical assistant interface that includes the other graphical control element. The second composite graphical assistant interface may replace the first composite graphical assistant interface to provide further suggestions to the user to control a specific connected device within a specific scene. In some embodiments, the first composite graphical assistant interface may correspond to a single connected device provided by a third party, and the second composite graphical assistant interface may correspond to a plurality of different connected devices provided by a plurality of different third parties. In this way, the user does not necessarily need to install and / or open a variety of different applications to control different connected devices, but can interact with a single composite graphical assistant interface to facilitate control of a variety of different connected devices.

[0067] Figure 6 6 is a block diagram of an example computer system 610. The computer system 610 typically includes at least one processor 614 that communicates with a number of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624 including, for example, a memory 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow a user to interact with the computer system 610. The network interface subsystem 616 provides an interface to an external network and couples to corresponding interface devices in other computer systems.

[0068] The user interface input devices 622 may include a keyboard, a pointing device such as a mouse, trackball, touchpad or graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a speech recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 610 or onto a communication network.

[0069] The user interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, for example, via an audio output device. Typically, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 610 to a user or another machine or computer system.

[0070] The storage subsystem 624 stores program and data structures that provide some or all of the functionality of the modules described herein. For example, the storage subsystem 624 may include logic for performing selected aspects of the methods 400, 410, and 500 and / or implementing one or more of the following: the connection device server 134, the server device 112, the automated assistant 118, the client device 102, the first connection device 132, the second connection device 146, the function identification engine 142, the composite interface engine 144, the client device 204, the client device 328, the assistant device 326, and / or any other operations or means discussed herein.

[0071] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory 625 used in the storage subsystem 624 can include many memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 can provide persistent storage for program and data files, and can include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations can be stored in the storage subsystem 624 by the file storage subsystem 626, or in other machines accessible by the processor 614.

[0072] The bus subsystem 612 provides a mechanism for the various components and subsystems of the computer system 610 to communicate with each other as intended. Although the bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0073] Computer system 610 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 6The description of the computer system 610 depicted in FIG. 6 is intended only as a specific example for the purpose of illustrating some implementations. Many other configurations of the computer system 610 may have more Figure 6 The computer system depicted in FIG.

[0074] In situations where the systems described herein collect personal information about users (or often referred to herein as "participants") or may utilize personal information, users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current geographic location) or to control whether and / or how content that may be more relevant to the user is received from a content server. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the identity of a user may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be generalized (such as to a city, zip code, or state level) in the case where geographic location information is obtained so that the user's specific geographic location cannot be determined. Thus, users may control how information is collected and / or used about them.

[0075] Although several implementations have been described and illustrated herein, various other means and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of such changes and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on one or more specific applications to which these teachings are used. Those skilled in the art will recognize or be able to use more than routine experiments to ascertain many equivalents of the specific implementations described herein. Therefore, it should be understood that the foregoing implementations are given only as examples, and within the scope of the appended claims and their equivalents, implementations may be practiced in a manner different from that specifically described and claimed. The implementations disclosed herein are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure when such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent.

Claims

1. A method implemented by one or more processors, the method comprising: determining that a spoken utterance detected via an automated assistant interface of a client device comprises a request for an automated assistant to perform a function, wherein the spoken utterance is provided by a user; identifying the function corresponding to the request provided by the user in response to determining that the request was provided by the user; causing the automated assistant to perform the function corresponding to the request provided by the user based on identifying the function; determining a current context of a user based at least in part on the spoken utterance; identifying a connected device linked to the client device, wherein identifying the connected device is based on a historical tendency of one or more users to change a state of the connected device while in the context; determining a graphical control element for changing the state of the connected device; as well as In response to determining that the request is received, causing a display panel associated with the client device to provide a graphical assistant interface for controlling the connected device, wherein the display panel includes the graphical control element.

2. The method according to claim 1, wherein: Identifying the connecting device is based on the historical tendencies of the user.

3. The method according to claim 1, wherein: Identifying the connecting device is based on the historical tendencies of one or more other users.

4. The method according to claim 1, wherein: The current context of the user is based on the location of the user.

5. The method according to claim 1, wherein: The current context of the user is based on a current time.

6. The method according to claim 1, further comprising: identifying a second connected device linked to the client device, wherein identifying the second connected device is based on a historical tendency of one or more users to change a state of the second connected device while the user is in the context; and A second graphical control element for changing the state of the second connected device is determined, wherein the display panel further comprises the second graphical control element. 7 . The method of claim 6 , wherein the graphical control elements and the second graphical element are arranged according to a determined configuration for each graphical control element.

8. A method implemented by one or more processors, the method comprising: determining that a spoken utterance detected via an automated assistant interface of a client device comprises a request for an automated assistant to perform a function, wherein the spoken utterance is provided by a user; identifying a function corresponding to the request provided by the user in response to determining that the request was provided by the user; causing the automated assistant to perform the function corresponding to the request provided by the user based on identifying the function; determining a first current state of a first connection device linked to the client device, wherein the first current state of the connection device is affected by execution or non-execution of a first function; determining a second current state of a second connection device linked to the client device, wherein the second current state of the second connection device is affected by execution or non-execution of a second function; determining a first graphical control element for controlling a first function of the first connected device and a second graphical control element for controlling a second function of the second connected device; determining a configuration of each of the first graphical control element and the second graphical control element based on a first current state of the first connected device and a second current state of the second connected device; as well as In response to determining that the request is received, causing a display panel associated with the client device to provide a composite graphical assistant interface for controlling the first connected device and the second connected device, The composite graphic assistant interface includes the first graphic control element and the second graphic control element, and the first graphic control element and the second graphic control element are arranged according to the determined configuration of each graphic control element.

9. The method according to claim 8, further comprising: A profile of the user is accessed based on voice characteristics associated with the spoken utterance, wherein identifying at least one of the first function and the second function is based on information available from the profile.

10. The method of claim 9, wherein the profile indicates a historical tendency of the user to interact with at least one of the first and second connected devices within a particular context.

11. The method of claim 10, wherein the specific context corresponds to one or more of: a state of the first connected device or the second connected device, a geographic location of the user, a time, an event, or the presence of one or more other users.

12. A method according to any of the preceding claims, wherein determining a first current state of the first connected device includes requesting the first current state and command data associated with the first function from a remote server device, wherein the first function is different from the function performed by the automated assistant.

13. The method according to claim 12, further comprising: receiving a selection of the first graphical control element for controlling the first function; and The client device is caused to interact with the first connection device according to the command data associated with the first function.

14. A method implemented by one or more processors, the method comprising: determining that a user has provided a request for an automated assistant to perform a function, wherein the request is received at an automated assistant interface of a client device; causing the automated assistant to perform the function corresponding to the request in response to determining that the user has provided the request; determining a user preference in response to determining that the user has provided the request, the user preference characterizing a historical tendency of the user to control another function of a connected device in a context associated with the function performed by the automated assistant, wherein the connected device is linked to the client device; identifying the connected device and the another function of the connected device based on determining the user preference; determining a current state of the connected device based on identifying the connected device, wherein the current state is affected by the execution or non-execution of the another function; determining a graphical control element for controlling the other function of the connected device based on identifying the other function and the current state of the connected device, wherein the graphical control element indicates a capability of the graphical control element to cause the other function to be performed by the connected device, thereby causing the current state of the connected device to change to a different state; as well as In response to determining that the user has provided the request, causing a display panel associated with the client device to provide a composite graphical assistant interface including the graphical control element.

15. The method according to claim 14, further comprising: Determining selectable suggestion elements associated with individual ones of the connected devices based on the user preferences, wherein causing the display panel to provide the composite graphical assistant interface includes causing the composite graphical assistant interface to include the selectable suggestion elements.

16. The method according to claim 15, further comprising: determining that a selection of the selectable suggestion element is received at the composite graphical assistant interface; and One or more graphical control elements are caused to be presented on the display panel to control one or more functions of the individual connected devices.

17. The method according to claim 15, further comprising: determining a speakable command that, when received at the automated assistant interface of the client device, invokes the automated assistant to cause the separate connected device to perform the one or more functions of the separate connected device; Wherein causing the display panel associated with the client device to provide the composite graphical assistant interface includes causing the composite graphical assistant interface to include a natural language representation of the speakable command.

18. The method according to claim 15, further comprising: determining a resulting state of the connected device in response to causing the connected device to perform the function, wherein the resulting state is affected by the performance of the function, Wherein causing the display panel associated with the client device to provide the composite graphical assistant interface includes causing the composite graphical assistant interface to include a graphical interface element indicating a resulting status of the connected device.

19. The method of any of the preceding claims, wherein the user preference characterizes the user's historical propensity to control the another function based at least on previous interactions between the user and the automated assistant.

20. The method according to claim 19, wherein: The previous interaction corresponds to a conversational session between the user and the automated assistant prior to the user selecting the graphical control element, wherein during the conversational session the user provided a request to cause the connected device to perform the other function.

21. A method implemented by one or more processors, the method comprising: determining that a user has provided a request for an automated assistant to perform a function, wherein the request is received at an automated assistant interface of a client device; causing the automated assistant to perform the function corresponding to the request in response to determining that the user has provided the request; determining a user preference in response to determining that the user has provided the request, the user preference characterizing a historical tendency of the user to control another function of a connected device in a context associated with the function performed by the automated assistant, wherein the connected device is linked to the client device; identifying the connected device and a desired state of the connected device based on determining the user preference; determining a current state of the connected device based on identifying the connected device, wherein the current state is affected by execution or non-execution of the another function; When the desired state is different from the current state of the connected device: determining a graphical control element for controlling the other function of the connected device based on identifying the other function and the current state of the connected device, wherein the graphical control element indicates a capability of the graphical control element to cause the other function to be performed by the connected device, thereby causing the current state of the connected device to change to the desired state; as well as In response to determining that the user has provided the request, a display panel associated with the client device is caused to provide a composite graphical assistant interface including the graphical control element.

22. The method according to claim 21, further comprising: When the desired state is the same as the current state of the connected device: In response to determining that the user has provided the request, a display panel associated with the client device is caused to provide a specific composite graphical assistant interface, the specific composite graphical assistant interface omitting specific graphical control elements for controlling the other function of the connected device.

23. A computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any preceding claim.

24. A computer-readable storage medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1 to 22.

25. A system comprising one or more processors configured to execute the method according to any one of claims 1 to 22.

Images