Method, system and apparatus for providing a composite graphical assistant interface for controlling connected devices

Abstract

The present invention provides a method, apparatus, system, and computer-readable medium for arranging a synthetic graphical assistant interface to interact with multiple different connected devices. [Solution] A synthetic graphical assistant interface can be generated in response to a user giving an automated assistant a request to have the connected device perform a specific function. In response to the automated assistant receiving the request, the automated assistant can identify other functions that the connected device can perform. These other functions can then be mapped to various graphical control elements to provide a synthetic graphical assistant interface from which the user can interact with the connected device. Each graphical control element can be arranged according to the status of the connected device to reflect how the connected device is operating simultaneously with the presentation of the synthetic graphical assistant interface.

Description

[Technical field]

[0001] The present invention relates to a method, system and apparatus for providing a composite graphical assistant interface for controlling connected devices. [Background technology]

[0002] Humans may conduct human-computer conversations using interactive software applications referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," etc.). For example, humans (which may be referred to as "users" when interacting with an automated assistant) may provide commands and / or requests to the automated assistant using voice natural language input (i.e., speech), which may in some cases be converted to text and processed, and / or by providing textual (e.g., typed) natural language input.

[0003] Some automated assistants may be used to control Internet of Things ("IoT") devices. However, many IoT devices may come equipped with their own corresponding applications. As a result, a user who acquires multiple different IoT devices may be tasked with installing a variety of third-party applications on the user's personal computing device. Having a variety of similar applications on a single device may be inefficient, as applications may generally require memory, network bandwidth, updates, and / or other resources. Although a user may rely on an automated assistant to mediate interactions between the user and a particular IoT device, such interactions may often require that spoken utterances from the user be converted from audio to text. To effect the audio-to-text conversion, audio data must often be sent over the network, thereby consuming significant network bandwidth. Moreover, mediating interactions in such a manner may prove unreliable when the user is surrounded by other people who are conversing, or in some cases, in an environment with background noise. As a result, an interaction session between a user and an automated assistant may waste computational and network resources when a request for an IoT device to perform a particular function is not ultimately identified or, in some cases, satisfied. Summary of the Invention [Means for solving the problem]

[0004] The present disclosure is generally directed to methods, apparatus, and (transient and non-transient) computer-readable media for providing a composite graphical assistant interface through which multiple different devices are controlled. As network connectivity of common devices such as light bulbs and other appliances becomes more prevalent, the ways to interface with such devices over a network may increase. This may be due in part to the increase in the number of manufacturers making similar devices. For example, there may be dozens of smart light bulb manufacturers, with most of the light bulb manufacturers providing applications or interfaces for controlling their respective light bulbs. As a result, users who install light bulbs from different manufacturers may download various applications to their personal computing devices to control the light bulbs. This may waste memory. Because having multiple applications on one device may cause the device to check for updates, shared data, and / or push notifications for each manufacturer, installing multiple applications may take up a significant amount of device memory while wasting network bandwidth. To overcome these and / or other limitations, some implementations described herein provide various techniques for an automated assistant to generate a composite graphical assistant interface for controlling various devices, even though the devices are provided by different third parties.

[0005] A composite graphical assistant interface may be an interface compiled from one or more graphical elements to effect control of one or more devices. The composite graphical assistant interface may be employed by an automated assistant that may act as an intermediary between a user and any connected device. For example, a user may provide a voice utterance such as "Assistant, turn on my kitchen lights." The voice utterance may be received at a client device, which may include one or more microphones and a display panel, and in response, the automated assistant may turn on any lights in the kitchen of the user's home. Additionally, the automated assistant may identify the type of "Internet of Things" (IoT) device that the user has requested the automated assistant to control to identify one or more control parameters associated with the type of IoT device. The control parameters may correspond to slot values ​​that may be provided to the IoT device to modify the state of the IoT device, and thus, the control parameters may be identified by analyzing the slot values ​​acceptable for the IoT device. For example, an IoT device such as a smart light bulb may receive slot values ​​such as "on" and "off," which may correspond to the light state of the light bulb, and / or color values ​​such as "red," "green," and "blue," which may correspond to the color state of the light bulb. Additionally, the IoT device may receive slot values, such as any number between 0 and 100, which may correspond to the brightness state of a light bulb. When the automated assistant has identified all possible slot values ​​and / or resulting states of the IoT device, the automated assistant may identify suitable graphical control elements to compile a composite graphical assistant interface from them.

[0006] For example, in some implementations, the automated assistant can identify the total number of slot values ​​for a given function of a particular IoT device to correlate functions with graphical control elements. When the automated assistant selects a graphical control element for a function with two slot values ​​(e.g., "on" and "off"), it can select a graphical control element with two states, similar to a two-position electric switch. Furthermore, when the automated assistant selects a graphical control element for a function with color names as slot values ​​(e.g., "red", "green" and "blue"), it can select a graphical control element with color-coded inputs, such as an array of colored buttons. Moreover, when the automated assistant selects a graphical control element for a function with at least one threshold number of slot values ​​(e.g., 3 or more, 6 or more, 11 or more, etc.), it can select a graphical control element such as a dial control, such as a potentiometer.

[0007] When the automated assistant has selected one or more suitable graphical control elements from which to generate a composite graphical assistant interface, the automated assistant can then cause the client device to display the composite graphical assistant interface. For example, in response to receiving a voice utterance of "Assistant, turn on my kitchen lights," the automated assistant can turn on the kitchen lights and cause the client device to display the composite graphical assistant interface with the selected graphical control elements. Thus, the composite graphical assistant interface is displayed on the display panel of the client device with graphical control elements such as a two-position switch for turning lights on and off, a color-coded input for controlling the color of the lights, and / or a dial control for adjusting the brightness of the lights. In these and other ways, rather than having to access a unique application for each respective device and / or each respective device manufacturer, the automated assistant can generate a similar interface with familiar graphical control elements that a user can interact with to modify the state of several devices. Also, by compiling the composite graphical assistant interface in such a way, network bandwidth can be conserved because the client device does not necessarily have to communicate with a remote server to process the voice utterance.

[0008] Moreover, in various implementations, as described above and elsewhere herein, a graphical assistant interface for controlling the functions of the connected device is rendered in response to a voice command for controlling the connected device. For example, in response to a voice input of "adjust the thermostat to 72 degrees," an interactive assistant interface may be rendered that reflects changing the setpoint of the thermostat to 72 degrees, allows touch interactions to further adjust the controlled setpoint function, and allows touch interactions to further adjust additional functions (e.g., humidity function, fan on / off function, etc.). In these and other manners, the presented assistant interface further assists the user in continuing to execute the control of the connected device. Moreover, the presented interface may enable further adjustment of the function that is more network and / or computational resource efficient, for example, as compared to further adjustment effected via further voice input.

[0009] In some implementations, the state of the device may be reflected in the composite graphical assistant interface presented by the automated assistant. For example, in response to receiving a voice utterance of "Assistant, turn on my kitchen lights," the automated assistant may cause the aforementioned composite graphical assistant interface to be presented on a display panel of the client device. Additionally, after the automated assistant turns on the kitchen lights, the automated assistant may identify the resulting state of the kitchen lights (e.g., state: [lights "on", ...]) and configure the graphical control elements of the composite graphical assistant interface to represent the state of the kitchen lights. For example, the graphical control element for controlling the kitchen lights may continue to present itself as being in an "on" state to reflect the change in state caused by the user uttering a voice utterance to the automated assistant. In some implementations, other states of the device may also be reflected in the composite graphical assistant interface. For example, when the brightness level of the kitchen lights is at 50%, the automated assistant may be notified when the user gives a voice utterance. The Tanto can cause the dial that controls the brightness level of the kitchen light to show 50%. In other words, in response to the automated assistant receiving a particular voice utterance that modifies the state of the IoT device, the automated assistant can retrieve (or receive) additional state information from the IoT device to precisely indicate the state of the IoT device to the user. Additionally, in response to the automated assistant receiving a particular voice utterance, the automated assistant can determine other functions that may be performed by the IoT device. This can enable the automated assistant to compile a more comprehensive composite graphical assistant interface that reflects the multiple controls that may be employed to modify the state of the IoT device.

[0010] Providing a graphical assistant interface that reflects the user's previous interactions with the automated assistant may facilitate improved efficiency of user interaction. In particular, related assistant tasks, such as controlling IoT devices, may be facilitated with a reduced set of interactions, thereby reducing the consumption of computer and network resources associated with those interactions. Moreover, in some examples, initial interactions may be accomplished through alternative modalities, such as through speech recognition, where speech is particularly appropriate for some controls (e.g., binary selection of on or off states) but is less efficient for related functions (such as precise control of desired levels or balance). Therefore, appropriate modalities for each function may be presented in an automated and efficient manner. Furthermore, providing a graphical interface may enhance the discoverability of functions related to the initial user input.

[0011] The above description is provided as an overview of some implementations of the present disclosure. Further descriptions of those and other implementations are described in more detail below.

[0012] In some implementations, a method implemented by one or more processors is described as including operations such as determining that a voice utterance detected via an automated assistant interface includes a request to modify a state of a connected device linked with the client device. The voice utterance is given by a user. The method may further include, in response to receiving the request, identifying a function corresponding to the received request, and, based on identifying the function, causing the connected device to execute the function corresponding to the received request, and determining a current state of the connected device. The current state of the connected device may be affected by the execution of the function. The method may also include identifying one or more other functions that may be executed by the connected device. The current state of the connected device may further be affected by the execution of the one or more other functions. The method may further include, based on identifying the function and the one or more other functions, determining a first graphical control element for controlling the function executed by the connected device and a second graphical control element for controlling the one or more other functions. The method may also include determining a configuration for each of the first and second graphical control elements based on a current state of the connected device, and, in response to receiving the request, causing a display panel associated with the client device to provide a composite graphical assistant interface for controlling the connected device. The composite graphical assistant interface may include the first and second graphical control elements arranged according to the configuration determined for each graphical control element.

[0013] In some implementations, the method may include determining that a user has selected a second graphical control element on the display panel corresponding to an additional function among the one or more other functions; in response to determining that the user has selected the second graphical control element, causing the connected device to perform the additional function, where execution of the additional function causes the connected device to indicate a modified state; and in response to the connected device executing the additional function, causing the second graphical control element to be reconfigured in accordance with the modified state.

[0014] In some implementations, the display panel includes a touch interface that allows a user to directly control the first and second graphical control elements using one or more gesture inputs. In some implementations, the method can include determining a third graphical control element for controlling an additional function of the one or more functions and a selectable link configured to link to the third graphical control element. Having the display panel associated with the first client device provide the composite graphical assistant interface can include having the selectable link provided in the composite graphical assistant interface.

[0015] In some implementations, the method may include determining that a user has selected a selectable link on the display panel, and causing the composite graphical assistant interface to provide a third graphical control element in response to determining that the user has selected the selectable link.

[0016] In some implementations, the client device lacks an integrated display panel, and the method may further include determining that the client device lacks an integrated display panel, and identifying an additional connected device incorporating a display panel and linked to the client device in response to determining that the client device lacks an integrated display panel. Having the display panel provide the composite graphical assistant interface may include having a display panel of the additional connected device render the composite graphical assistant interface. In some implementations, identifying the additional connected device includes selecting the additional connected device from the plurality of candidate devices based on a stored device topology indicating a proximity of the additional connected device to the client device.

[0017] In another implementation, a method implemented by one or more processors is described as including operations such as determining that the voice utterance detected via the automated assistant interface includes a request for the connected device to perform a function. The voice utterance may be provided by a user, and the connected device is linked to a client device. The method may also include, in response to receiving the request, identifying a function corresponding to the received request, and, based on identifying the function, causing the connected device to perform the function corresponding to the received request, and, based on historical interaction data accessible to the client device, determining one or more other functions previously performed by the connected device according to a previous request from the user. The method may further include, based on identifying the one or more other functions, determining a graphical control element for controlling the one or more other functions, and, in response to receiving the request, causing a display panel associated with the client device to provide a composite graphical assistant interface for controlling the connected device. The composite graphical assistant interface may include a graphical control element.

[0018] In some implementations, the method can include determining a profile for the user based on voice characteristics associated with the voice utterance. Providing a composite graphical assistant interface at the client device can include causing the composite graphical assistant interface to include a plurality of graphical control elements arranged based on the determined profile.

[0019] In some implementations, the plurality of graphical control elements omits at least one available graphical control element based on the user preferences indicated by the determined profile. In some implementations, the method may include requesting state data from the remote server device indicating a current state of the connected device. The current state of the connected device may be affected by one or more functions previously executed by the connected device. The method may further include receiving the state data from the remote server device and determining a configuration for the graphical control element based on the state data. The graphical control elements may be arranged in the composite graphical assistant interface according to the determined configuration.

[0020] In some implementations, the method can include receiving command data corresponding to additional functionality for modifying a current state of the connected device in response to requesting the state data. In some implementations, the method can include determining additional graphical control elements and selectable links for controlling the additional functionality. Having a display panel associated with the client device provide a composite graphical assistant interface can include causing the composite graphical assistant interface to include a selectable link. The selectable link can be configured to expose the additional graphical control element in response to receiving a user selection of the selectable link.

[0021] In yet another implementation, a method implemented by one or more processors is described as including operations such as determining that a user has selected a graphical control element in a first composite graphical assistant interface provided in a display panel associated with the client device. The first composite graphical assistant interface may be a manner in which the user can control a connected device in communication with the client device. The method may further include, in response to determining that the user has selected the graphical control element, causing the connected device to execute a function corresponding to the graphical control element. The connected device may be configured to execute a plurality of different functions. The method may also include, in response to determining that the user has selected the graphical control element, determining a user preference characterizing a historical propensity of the user to control one or more other functions of the plurality of different functions. The method may further include, based on determining the user preference, identifying one or more other functions of the plurality of different functions, and, based on identifying the one or more other functions, determining one or more graphical control elements for controlling the one or more other functions of the connected device. The method may also include, in response to determining that the user has selected a graphical control element, causing a display panel associated with the client device to replace the first composite graphical assistant interface with a second composite graphical assistant interface that includes one or more graphical control elements.

[0022] In some implementations, the method may include determining a state of the connected device. The state may be affected by the execution of the function and one or more other functions. The method may also include determining an arrangement of the one or more graphical control elements for each graphical control element based on the state of the connected device. In some implementations, the state of the connected device may be based on a resulting situation resulting from the execution or non-execution of each function of the one or more other functions.

[0023] In some implementations, the state of the connected device can be based on a resulting situation resulting from the execution of the function, and the second composite graphical assistant interface includes a graphical interface element that indicates the resulting situation. The second composite graphical assistant interface can exclude a graphical control element that corresponds to the function. In some implementations, the user preference characterizes a historical propensity of the user to control one or more other functions of the plurality of different functions based on at least a previous interaction between the user and the automated assistant accessible via the automated assistant interface of the client device. In some implementations, the previous interaction corresponds to an interaction session between the user and the automated assistant prior to the user selecting the graphical control element, during which the user provided a request to have the connected device execute the function.

[0024] Other implementations may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described above and / or elsewhere herein. Still other implementations may include one or more robotic systems including one or more computers and / or one or more processors operable to execute the stored instructions to perform methods such as one or more of the methods described above and / or elsewhere herein.

[0025] It should be appreciated that all combinations of the foregoing and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 illustrates a system for assembling a composite graphical assistant interface for interacting with connected devices. [Figure 2A] FIG. 13 is an example of a composite graphical assistant interface generated in response to a user providing a request for a connected device to perform a particular function. [Figure 2B] FIG. 13 is an example of a composite graphical assistant interface generated in response to a user providing a request for a connected device to perform a particular function. [Figure 3A] FIG. 13 is an example of a composite graphical assistant interface generated using suggestions for interacting with a connected device using a variety of different capabilities. [Figure 3B]FIG. 13 is an example of a composite graphical assistant interface generated using suggestions for interacting with a connected device using a variety of different capabilities. [Figure 4A] FIG. 1 illustrates a method for providing a composite graphical assistant interface for controlling connected devices. [Figure 4B] FIG. 1 illustrates a method for providing a composite graphical assistant interface for controlling connected devices. [Diagram 5] FIG. 1 illustrates a method for providing graphical control elements in a composite graphical assistant interface based on user preferences for a user to control a connected device via the composite graphical assistant interface. [Figure 6] FIG. 1 is a block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] FIG. 1 illustrates a system 100 for assembling a composite graphical assistant interface for interacting with a connected device. The composite graphical assistant interface may be generated in part by an automated assistant 118 provided at one or more computing devices, such as the client device 102 (e.g., tablet device 128) and / or a remote computing device 124, such as the server device 112. A user 120 may interact with the client automated assistant 104 via an assistant interface 110, which may be a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, the user 120 may initialize the client automated assistant 104 by providing linguistic, textual, and / or graphical input to the assistant interface 110 to cause the client automated assistant 104 to perform an action (e.g., provide data, control a peripheral device, access an agent, etc.). The client device 102 may include a display device 108, which may be a display panel including a touch interface for receiving touch inputs and / or gestures to allow a user to control an application of the client device 102 via a touch interface.

[0028] The client device 102 can communicate with a remote computing device 124 over a network 122, such as the Internet. The client device 102 can offload computational tasks to the remote computing device 124 to conserve computational resources at the client device 102. For example, the remote computing device 124 can host an automated assistant 118, and the client device 102 can send inputs received at one or more assistant interfaces to the remote computing device 124. However, in some implementations, the automated assistant 118 can be hosted entirely on the client device 102. In various implementations, all or less than all aspects of the client automated assistant 104 can be implemented on the client device 102. In some of those implementations, aspects of the automated assistant 118 are implemented via the client automated assistant 104 of the client device 102 and interface with a remote computing device 124 that implements other aspects of the client automated assistant 104. The remote computing device 124 can optionally serve multiple users and their associated assistant applications via multiple threads. In some implementations in which all or less than all aspects of the automated assistant 118 are implemented via a client automated assistant 104 of the client device 102, the client automated assistant 104 may be an application separate from (e.g., installed "on top of") the operating system of the client device 102 or, alternatively, may be implemented directly by (e.g., an operating system application but considered to be integral with) the operating system of the client device 102.

[0029] In some implementations, the remote computing device 124 can include a speech-to-text engine 116 that can process audio data received at the assistant interface 110 to identify text integrated within the audio data. The process for converting audio data to text can include speech recognition algorithms that can employ neural networks, word2vec algorithms, and / or statistical models to identify groups of audio data that correspond to words or phrases. The text converted from the audio data can be parsed by the text parser engine 114 and made available to the automated assistant 104 as text data that can be used to generate and / or identify command phrases from a user and / or third-party application.

[0030] A user 120 can interact with a connected device 132 using the user's client device 102 and / or the client automated assistant 104. The connected device 132 can be one or more IoT devices that can be connected or communicate with the client device 102 over a local area network and / or a wide area network. For example, both the client device 102 and the connected device 132 can be connected to a Wi-Fi network in a home or other environment. In order for the user 120 to interact with the connected device 132, the user 120 can initialize an application associated with the connected device 132. However, if the user 120 frequently interacts with multiple different connected devices, the user 120 may need to run a variety of different applications to control each connected device 132. To more efficiently use computational resources and prevent wasting network bandwidth, the client automated assistant 104 and / or the automated assistant 118 can provide a composite graphical assistant interface through which the user can interact with one or more connected devices 132.

[0031] The user 120 can provide a voice utterance 130 to the assistant interface 110 so that a synthetic graphical assistant interface is generated. The voice utterance can incorporate a request indicating that the user 120 prefers the connected device 132 to perform a particular function. Audio data corresponding to the voice utterance 130 can be transmitted from the client device 102 to the remote device 124. The audio data can then be processed by the speech-to-text engine 116 and by the text parser engine 114. Text obtained from processing the audio data can be used by the function identification engine 142 to identify one or more functions that the user 120 refers to in the user's voice utterance 130. For example, the function identification engine 142 can compare the text from the voice utterance 130 with function data 136 made available at the remote device 126, such as the connected device server 134. The function data 136 can provide a comprehensive list of functions that one or more connected devices can perform. A feature identification engine 142 may be executed by the connection device 132 and may compare the text with the feature data 136 to identify the feature or features to which the user is referring.

[0032] In response to determining that the text corresponds to one or more functions, the automated assistant 118 or the client automated assistant 104 may cause the connected device 132 to execute one or more identified functions. For example, when the connected device 132 is an IoT device such as, but not limited to, a thermostat, the function may include changing a temperature setting of the connected device 132. A command to cause the connected device 132 to execute a function may be provided from the server device 112 to the connected device 132. Alternatively, a command to cause the connected device 132 to execute a function may be provided from the server device 112 to the connected device server 134, which may communicate with the connected device 132 to cause the connected device 132 to execute the function. Alternatively, the client device 102 may communicate a command to the connected device 132, or the server device 112 may communicate a command to the client device 102, which may communicate with the connected device 132 to cause the connected device 132 to execute the function.

[0033] In response to receiving the voice utterance 130 and causing the connected device 132 to execute a function, the automated assistant can cause a composite graphical assistant interface to be provided on the display device 108 of the client device 102 or on a remote display device associated with the client device 102. To generate the composite graphical assistant interface, the automated assistant 118 can identify one or more other functions that can be executed by the connected device 132 from the function data 136. The identified other functions can be used as a basis for identifying a graphical control element 140 that can serve to control each of the identified other functions on the display device 108. For example, a function such as turning a fan on and off can be controlled by a graphical control element 140 that corresponds to a two-way switch. Additionally, a function such as adjusting humidity can be controlled using a graphical control element 140 that corresponds to a dial that can be turned to select one of a range of values. The automated assistant 118 or the client automated assistant 104 can map the identified other functions to a graphical control element 140, which can be stored on the server device 112 or, in some cases, can be accessible to the server device 112.

[0034] The composite interface engine 144 can map the graphical control elements 140 to the identified other functions to generate a composite graphical assistant interface. In some implementations, the automated assistant 118 can access status data 138 corresponding to a status or state of the connected device 132. The status data 138 can indicate a status that can be affected by the execution or non-execution of one or more functions of the connected device 132. For example, when a function of the connected device 132 is not being executed, the status of the connected device 132 can be "off". Alternatively, the status can correspond to a table or string of data indicating the status or parameters of a particular function of the connected device 132. The status of the connected device 132 can be used as a criterion for determining a configuration for each graphical control element 140 mapped to a function of the connected device 132. For example, if the connected device 132 includes a light that is in an off state and the function of the connected device 132 is to turn the light on and off, a graphical control element corresponding to a light switch can be provided in the composite graphical assistant interface and configured to indicate that the light switch is turned off. In other words, each graphical control element can reflect the current state of the connected device 132 when the composite graphical assistant interface is presented on the display device 108.

[0035] In some implementations, the arrangement and / or selection of graphical control elements for the composite graphical assistant interface can be based on data associated with interactions between the user 120 and the automated assistant and / or connected device 132. For example, a particular group of graphical control elements corresponding to a particular group of functions can be selected for the composite graphical assistant interface based on the user having previously performed such functions within a threshold period of time. Alternatively or additionally, a particular group of graphical control elements corresponding to a particular group of functions can be selected for the composite graphical assistant interface based on how the functions affect a particular user, a particular location, a particular device, and / or any other relationships that may be indicated by the device's functions. For example, a composite graphical assistant interface generated for a user can include graphical control elements corresponding to functions of a connected device that affect the home in which the user is currently located. Thus, the selection of graphical control elements can be based on the user's location, an identifier for the connected device, the user's calendar schedule, and / or any other data that may be used to estimate geolocation data for the user.

[0036] In some implementations, the user 120 can provide voice utterances 130 to an assistant device that lacks an integrated display panel or, in some cases, cannot easily present a composite graphical assistant interface at the assistant device. In such implementations, the automated assistant can receive the voice utterances 130 at the assistant device and cause the connected device 132 to perform a function requested by the user 120. However, in response to receiving a request to perform a function and determining that a display panel is not available at the assistant device, the automated assistant can identify one or more candidate devices. The candidate devices can be identified based on their ability to display the assistant interface, their proximity to the user 120, their proximity to the assistant device, their proximity to the connected device 132, and / or the user 120's preference for a particular candidate device. For example, when the automated assistant determines that a display panel is not available at the assistant device and that a tablet device or a television is the next closest display device to the assistant device, the automated assistant can cause the tablet device or the television to present a composite graphical assistant interface. In this manner, the automated assistant can still have the composite graphical assistant interface generated on a nearby display device, since the user 120 is not necessarily near the display panel when the user 120 requests that a particular function be performed by the connected device 132. In some implementations, the automated assistant can have the assistant device audibly notify the user 120 of the location of the composite graphical assistant interface in response to the user 120 providing a request for the connected device 132 to perform a function.For example, in response to user 120 uttering the voice utterance "Assistant, turn on my kitchen lights," the automated assistant can have the assistant device output the voice response "Okay, I've prepared an interface on your tablet device with additional controls." Thus, the automated assistant can inform user 120 that if user 120 goes to retrieve their tablet device, user 120 can find the additional controls for their connected device.

[0037] 2A and 2B show diagrams 200 and 210 of an example of a synthetic graphical assistant interface 212 generated in response to a user 202 providing a request for a connected device to perform a particular function. Specifically, the user 202 can interact with a client device 204, such as a tablet device, by providing a voice utterance 208 to an audio interface of the client device 204. The voice utterance can be, for example, "Assistant, turn up the temperature in my house." In response to receiving the voice utterance, an automated assistant available at the client device 204 can have the voice utterance processed to identify a request integrated into the voice utterance 208. The request can correspond to a function of the connected device, such as turning up the temperature at a thermostat. The automated assistant can have the function performed by the connected device, as well as have the synthetic graphical assistant interface 212 provided on the display panel 206 of the client device 204.

[0038] FIG. 2B shows a diagram 210 in which a composite graphical assistant interface 212 is provided on a display panel 206 of a client device 204. Specifically, because the user 202 requested that the temperature setting be increased, the composite graphical assistant interface 212 can include a notification 214 that the temperature setting has been increased to "72". Additionally, the composite graphical assistant interface 212 can include multiple different graphical control elements through which the user 202 can control other functions of the connected device. For example, the composite graphical assistant interface 212 can provide a dial 218 for controlling a humidity setting of the connected device. In addition, the composite graphical assistant interface 212 can include a switch 216 for controlling a fan setting and a switch 220 for controlling a security setting. In this manner, the user 202 does not necessarily have to provide different voice utterances to the client device 204 to cause the connected device to perform different functions. Rather, controls for the functions can be presented in a manner that makes it less computationally intensive to control those functions compared to providing multiple voice utterances. 2B , in various implementations, graphical control elements that allow for further adjustment of the adjusted functionality in response to the voice utterance 208 may also be presented in the synthetic graphical assistant interface 212. For example, the notification 214 may be replaced by a dial (e.g., similar to the dial 218) that allows for control of the temperature setting of the connected device. Such a dial may also reflect that the temperature setting has been increased to "72" (e.g., via a display of "72" provided on or adjacent the adjustment element of the dial).

[0039] 3A and 3B show diagrams 300 and 310 of an example of a synthetic graphical assistant interface 312 generated by suggestions for interacting with a connected device using a variety of different functions. For example, a user 302 can provide a voice utterance 308 to an audio interface of a client device 304 to interact with or control a connected device. The voice utterance can be, for example, "Assistant, I want you to change the temperature in my house." The automated assistant can receive the voice utterance 308 and have the voice utterance 308 processed to identify a particular function that the user 302 wants to affect or control.

[0040] In response to identifying a particular function (e.g., controlling the temperature), the automated assistant can compile a composite graphical assistant interface 312 including at least one graphical control element to control the identified particular function. The automated assistant can also select other elements from which to compile the composite graphical assistant interface. For example, the composite graphical assistant interface 312 can include a dial 318 for controlling the temperature, as well as a first link 314 for controlling another function of the connected device (e.g., adjusting the humidity) and a second link 316 for controlling yet another function of the connected device (e.g., adjusting the fan). When the user 302 selects, for example, the first link 314, the composite graphical assistant interface can be replaced with a different composite graphical assistant interface including a dial for adjusting the humidity (e.g., dial 218 of FIG. 2B). In this way, instead of providing a series of voice utterances to the automated assistant to determine other functions of the connected device, various functions can be suggested to the user 302 in response to the user 302 providing an initial voice utterance related to the connected device. In some implementations, the composite graphical assistant interface 312 can show the current state of a connected device. For example, the dial 318 can be configured to reflect the current temperature setting of the connected device.

[0041] 4A and 4B show methods 400 and 410 for providing a composite graphical assistant interface for controlling a connected device. Methods 400 and 410 may be performed by one or more of a computing device, an application, and / or any other device or module that may interact with a connected device and / or an automated assistant. Method 400 may include an operation 402 of determining that a voice utterance detected via an automated assistant interface of a client device includes a request to modify a state of a connected device linked to the client device. The automated assistant interface may include one or more of a microphone, a touch display, a camera, and / or any other device through which a user may communicate with an automated assistant and / or a computing device. In some implementations, the connected device may be one or more devices in communication with a client device and / or an automated assistant accessible via the client device. For example, the client device and the connected device (e.g., an IoT device) may be connected to a local area network (e.g., a Wi-Fi network) in a user's home, or the client device and the connected device may be connected over the Internet.

[0042] The method 400 may further include an operation 404 of identifying a function corresponding to the received request in response to receiving the request. The function may be an operation that the connected device can perform or accomplish. In some implementations, the function may correspond to a number of different operations that may be performed at one or more candidate connected devices. The function may be, for example, an operation of turning on the kitchen light in the user's home. The voice utterance may consolidate the request to perform the function. For example, the voice utterance may be "Assistant, turn on my kitchen light." In response to receiving the voice utterance, the client device may identify a function that best corresponds to the received voice utterance by comparing text derived from the voice utterance to available functions that may be initialized in the instructions of the automated assistant.

[0043] Method 400 may also include an operation 406 of causing the connected device to perform a function corresponding to the received request based on identifying the function. For example, in some implementations, the client device may receive the request and issue a command to the connected device to perform the identified function. In other implementations, the request may be captured at the client device and sent to a remote server for further processing. The remote server may then identify the function and send a command to the connected device or a server associated with the connected device to cause the connected device to perform the function. Alternatively or additionally, in some implementations, the remote server may identify the function and send a command to the client device, and in response, the client device may communicate with the connected device to cause the connected device to perform the function.

[0044] The method 400 may further include an operation 408 of determining a current state of the connected device. The current state of the connected device may be influenced or dependent on the execution of one or more functions. For example, when a function corresponds to a request to turn on a light, the execution of the function may cause a resulting status of the connected device (e.g., a smart light) to be "on". Thus, the current state may be based on one or more conditions that are dependent on the execution or non-execution of one or more functions, respectively. For example, in response to the connected device having a light that is on, the current state of the connected device may be "smart_light(state): ([light:on], [brightness:50%], [color:flat_white])". The current state of the connected device may be determined by the client device issuing a query to the connected device and receiving in response data indicative of the current state of the connected device. Alternatively or additionally, in response to the connected device receiving a command to cause the connected device to execute an identified function, the connected device may execute the function and provide confirmation that the function was executed along with data indicative of the current state of the connected device. Alternatively or additionally, the connected device may communicate the current state of the connected device to the server, and the client device may determine the current state of the connected device by communicating directly or indirectly with a remote server that has access to one or more current states of the one or more connected devices.

[0045] As indicated by the circled continuation element "A" in both Figures 4A and 4B, the method 400 of Figure 4A may continue to the method 410 of Figure 4B. The method 410 may include an operation 412 of identifying one or more other functions that may be performed by the connected device. The one or more other functions may be different from the previously identified function. For example, when the identified function corresponds to an operation of turning on a light, the one or more other functions may correspond to an operation of setting a brightness level of the light and / or an operation of setting a color of the light. Furthermore, the one or more other functions may affect a current state of the connected device depending on whether their execution or non-execution has resulted in a particular situation at the connected device.

[0046] The method 410 may further include an operation 414 of determining a first graphical control element for controlling the function executed by the connected device and a second graphical control element for controlling the one or more other functions based on identifying the function and the one or more other functions. Determining the first graphical control element and the second graphical control element may include selecting the graphical control elements from a local or remote memory storage device, including a plurality of different graphical control elements from which a composite graphical assistant interface may be compiled. Each graphical control element may be stored in association with metadata that may indicate a parameter that the respective graphical control element may control. For example, the first graphical control element may be associated with a binary or "on-off" parameter, thereby indicating that the first graphical control element may control a function having two states or conditions. Moreover, the second graphical control element may be stored in association with a range of the parameter (e.g., a range of two or more values ​​between 0 and 1 or between 0 and 100), thereby indicating that the second graphical control element may control an apparently analog-like function, such as changing a brightness level and / or changing a color.

[0047] The method 400 may further include an operation 416 of determining a configuration for each of the first and second graphical control elements based on a current state of the connected device. The configuration may refer to an appearance of each graphical control element that reflects the current state of the connected device. For example, when the initial request corresponds to a function for turning on a light and the light is turned on in response thereto, the first graphical control element may resemble a light switch configured in the "on" position, thereby reflecting the current state of the connected device. Furthermore, when the current state of the connected device corresponds to a brightness level of 90%, the second graphical control element may resemble an analog dial configured in the 90% position, with the knob or dial of the second graphical control element positioned within half the range of the analog dial. In some implementations, metadata associated with each graphical control element may indicate the configuration of each graphical control element, and thus the metadata may be modified or updated to reflect the current state of the connected device when the composite graphical assistant interface is generated.

[0048] The method 410 may also include an operation 418 of causing a display panel associated with the client device to provide a composite graphical assistant interface including the first and second graphical control elements in response to receiving the request. Each of the first and second graphical control elements may be arranged according to the determined configuration to reflect the current state of the connected device to the user. In this manner, the user may quickly recognize that the initial request has been satisfied and that the graphical control elements may also control other functions of the connected device without having to recite additional voice utterances. This may conserve computational resources because processing additional voice utterances may require higher processing intensity than receiving input at a graphical user interface. Additionally, by avoiding processing further additional voice utterances, the client device may conserve network bandwidth because audio data associated with the additional voice utterances does not need to be transmitted to a remote server for processing (e.g., speech-to-text processing).

[0049] FIG. 5 illustrates a method 500 for providing a graphical control element in a composite graphical assistant interface based on a user preference for a user to control a connected device via the composite graphical assistant interface. The method 500 may be performed by one or more computing devices, applications, and / or any other apparatus or module that can interact with a client device and / or an automated assistant. The method 500 may include an act of determining that a user has selected a graphical control element in a first composite graphical assistant interface provided in a display panel associated with the client device. The first composite graphical assistant interface may be provided in the client device for controlling a connected device that is in communication with the client device over a local area network or a wide area network. The display panel on which the first composite graphical assistant interface is provided may be a touch interface that allows a user to initiate the execution of a particular function on the connected device. The first composite graphical assistant interface may be provided by an automated assistant application that may be used to control multiple different connected devices. The display panel may be integrated into the client device or may be separate from the client device, thereby allowing a user to show some control on a connected device without necessarily having to directly contact the connected device. Rather, the user can interact with the connected devices through a composite graphical assistant interface that is generated by the automated assistant to provide a standard interface for interacting with different devices from different third parties.

[0050] The method 500 may further include an operation 504 of causing the connected device to execute a function corresponding to the graphical control element in response to determining that the user has selected the graphical control element. The graphical control element may be part of a first composite graphical assistant interface selectable via a touch display panel and may cause a particular process to occur in the client device and / or the connected device. For example, the first composite graphical assistant interface may be associated with an electromechanical system in the user's home, and selecting the graphical control element may cause the electromechanical system to execute a function. For example, the graphical control element may be a slideable element that may be used to adjust the temperature output of an HVAC system. In some implementations, the first composite graphical assistant interface may include multiple different graphical control elements that allow the user to control a particular function of the HVAC system. However, the first graphical control elements may be limited to the graphical control elements most frequently used by the user and / or the number of graphical control elements that fit appropriately within the boundaries of the first composite graphical assistant interface.

[0051] The method 500 may also include an operation 506 of determining user preferences that characterize a historical propensity of the user to control one or more functions of a plurality of different functions that may be performed by the connected device. The user preferences may be integrated as part of one or more files available at or accessible to the client device. The user preferences may be a subset of data that is generated based on historical interactions between the user and the automated assistant, the client device, the connected device, and / or any other application or apparatus that may be controlled at the direction of the user. For example, the user preferences may indicate that the user previously operated an HVAC system to modify a temperature output of the HVAC system (i.e., the connected device) and shortly thereafter operated a fan of the HVAC system on. Thus, the historical propensity may be that the user prefers to adjust the fan of the HVAC system when the user adjusts the temperature output of the HVAC system. In some implementations, the user preferences may characterize a plurality of historical propensities of the user to adjust a plurality of functions of the connected device.

[0052] The method 500 may further include an operation 510 of determining one or more graphical control elements for controlling one or more other functions of the connected device based on identifying the one or more other functions. For example, a connected device such as an HVAC system may perform multiple functions including changing temperature, humidity, airflow, and / or any other operation. The various functions may be identified by an automated assistant that may access a list of functions from a remote server and / or the connected device. Alternatively or additionally, the client device may include an application associated with the connected device and thus provide data corresponding to the functions of the connected device and / or any functions that may be controlled via the client device and / or the automated assistant.

[0053] In some implementations, the one or more other graphical control elements may be selected from a store of graphical control elements provided on the client device, a different client device, and / or a remote server accessible to the client device. The graphical control elements may be stored with metadata indicating the types of inputs and / or outputs they may be employed to receive and / or provide. Thus, the automated assistant may use the metadata to correlate the graphical control elements with functions that may be performed by the connected device. For example, if the connected device can perform a function that uses a single slot value with two possible parameters (e.g., on or off), the automated assistant may correlate the function with a binary switch graphical control element (e.g., a light switch, etc.).

[0054] The method 500 may also include an operation 512 of causing a display panel associated with the client device to replace the first composite graphical assistant interface with a second composite graphical assistant interface including one or more graphical control elements. Replacing the first composite graphical assistant interface with the second composite graphical assistant interface may include causing the first composite graphical assistant interface to be at least partially or completely replaced by the second composite graphical assistant interface. In some implementations, the second composite graphical assistant interface may indicate successful completion of a function associated with a graphical control element selected in the first composite graphical assistant interface. In some implementations, the second composite graphical assistant interface may include a graphical control element associated with a selectable link provided in the first composite graphical assistant interface. For example, the first composite graphical assistant interface may include a selectable link indicating an available function, and in response to a selection of the selectable link, the second composite graphical assistant interface may be provided with a graphical element for controlling the available function.

[0055] In some implementations, the second composite graphical assistant interface may be arranged to include multiple different graphical control elements corresponding to functions previously controlled by the user in the context of controlling a function associated with the selected graphical control element. In this manner, the user need not necessarily provide a voice utterance to affect the previously controlled function, but only interacts with the second composite graphical assistant interface on the display panel. As a result, computational resources and / or network bandwidth may be conserved by eliminating the amount of voice processing that needs to occur to control a particular function.

[0056] In some implementations, the graphical control elements provided in the second composite graphical assistant interface can be based on relationships between the functions corresponding to the graphical control elements. For example, when a connected device is associated with functions that affect different locations in a home and / or different geographic regions, the graphical control elements provided in each composite graphical assistant interface can be organized by the locations or geographic regions associated with their assigned functions. Alternatively or additionally, the graphical control elements presented in the second composite graphical assistant interface can be selected based on rankings of their corresponding functions. The rankings can be based on the frequency of use of each function by the user or people associated with the user, other devices or applications owned or employed by the user, and / or any other data that can be used to recommend a particular function to the user. Additionally, the second composite graphical assistant interface can be limited to graphical control elements corresponding to functions having rankings that satisfy a certain ranking threshold.

[0057] 6 is a block diagram of an exemplary computer system 610. The computer system 610 typically includes at least one processor 614 that communicates with several peripheral devices via a bus subsystem 612. These peripheral devices may include, for example, a storage subsystem 624 including memory 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices enable user interaction with the computer system 610. The network interface subsystem 616 provides an interface to external networks and is coupled to corresponding interface devices in other computer systems.

[0058] The user interface output devices 622 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad or a graphics tablet, a scanner, a touch screen integrated into a display, a voice input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computer system 610 or onto a communications network.

[0059] The user interface output devices 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for generating a visible image. The display subsystem may also provide a non-visual display, such as through an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 610 to a user or to another machine or computer system.

[0060] The storage subsystem 624 stores program and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include logic for performing selected aspects of the method 400, the method 500, and / or for implementing one or more of the connected device server 134, the server device 112, the automated assistant 118, the client device 102, the connected device 132, the capability identification engine 142, the composition interface engine 144, the client device 204, the client device 304, and / or any other operations or apparatus described herein.

[0061] These software modules are generally executed by the processor 614 alone or in combination with other processors. The memory 625 used within the storage subsystem 624 may include several memories including a main random access memory (RAM) 630 for storing instructions and data during program execution, and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage for program and data files and may include a hard disk drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules implementing the functionality of some implementations may be stored by the file storage subsystem 626 within the storage subsystem 624 or in other machines accessible by the processor 614.

[0062] The bus subsystem 612 provides a mechanism for allowing the various components and subsystems of the computer system 610 to communicate with each other as intended. Although the bus system 612 is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0063] The computer system 610 can be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computer system 610 shown in Figure 6 is intended only as a specific example to illustrate some implementations. Many other configurations of the computer system 610 can have more or fewer components than the computer system shown in Figure 6.

[0064] In situations where the systems described herein may collect or use personal information about users (or sometimes referred to herein as "participants"), the user may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social actions or activities, occupation, user preferences, or the user's current geographic location) or whether and / or how to receive content from a content server that may be more appropriate for the user. Also, some data may be processed in one or more ways before the data is stored or used, thereby removing personally identifiable information. For example, the user's identity may be processed such that personally identifiable information is not specific to the user, or the user's geographic location may be generalized (such as to the city, zip code, or state level) to where the geographic location information is obtained such that the user's specific geographic location cannot be identified. Thus, the user may have control over how information is collected about the user and / or used.

[0065] Although several implementations have been described and illustrated herein, a variety of other means and / or structures for performing the functions and / or obtaining one or more of the results and / or advantages described herein may be utilized, and each of such variations and / or modifications are deemed to be within the scope of the implementations described herein. More generally, it is intended that all parameters, dimensions, materials, and configurations described herein are exemplary, and that the actual parameters, dimensions, materials, and / or configurations will depend on the particular application in which the teachings are used. Those skilled in the art will recognize or be able to ascertain using no more than routine experimentation many equivalents to the specific implementations described herein. It is therefore to be understood that the foregoing implementations are presented by way of example only, and that within the scope of the appended claims and their equivalents, implementations may be performed differently from what is specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, item, material, tool, and / or method described herein. In addition, any combination of two or more of such features, systems, articles, materials, tools, and / or methods is included within the scope of the present disclosure, if such features, systems, articles, materials, tools, and / or methods are not mutually inconsistent. [Explanation of symbols]

[0066] 100 Systems 102 client devices 104 Client Automated Assistant 108 Display Devices 110 Assistant Interface 112 Server Device 114 Text Parser Engine 116 Speech to text engine 118 Automated Assistants 120 users 122 Network 124 Remote Computing Devices 126 Remote Devices 128 tablet devices 130 Voice utterances 132 Connected Devices 134 Connected Device Server 136 Functional Data 138 Status Data 140 Graphical Control Elements 142 Feature Identification Engine 144 Synthesis Interface Engine 200 Figures 202 users 204 Client Device 206 Display Panel 208 Speech 210 Figure 212 Synthesis Graphical Assistant Interface 214 Notification 216 Switch 218 Dial 220 Switch 300 Figures 302 users 304 Client Device 308 Voice Utterance 310 Figure 312 Synthesis Graphical Assistant Interface 314 First Link 316 Second Link 318 Dial 610 Computer Systems 612 Bus Subsystem 614 processor 616 Network Interface Subsystem 620 User Interface Output Device 622 User Interface Input Devices 624 Memory Subsystem 625 Memory 626 File Storage Subsystem 630 Main Random Access Memory (RAM) 632 Read-Only Memory (ROM)

Claims

1. 1. A method implemented by one or more processors, comprising: determining that a voice utterance detected via an automated assistant interface of a client device includes a request to change a state of a connected device linked to the client device, the voice utterance being provided by a user; in response to determining that the voice utterance includes a request to change a state of the connected device, causing the connected device to perform a function corresponding to the request, the connected device being configured to perform a plurality of different functions; determining a user preference characterizing a historical tendency of the user to control one or more other functions of the plurality of different functions in response to determining that the voice utterance includes a request to change a state of the connected device; identifying the one or more other features of the plurality of different features based on the determination of user preferences; determining, based on the identification of the one or more other functions, one or more graphical control elements for controlling the one or more other functions of the connected device; causing a display panel of a candidate device associated with the client device to provide a composite graphical assistant interface including the one or more graphical control elements, the composite graphical assistant interface being a manner in which a user may control the connected device, the candidate devices being determined based on at least one of proximity to the client device, proximity to the connected device, and user preferences for particular candidate devices; A method comprising:

2. determining a state of the connected device, the state changing depending on execution of the function and at least one of the one or more other functions; determining an arrangement of the one or more graphical control elements for each graphical control element based on the state of the connected device; The method of claim 1, further comprising:

3. The method of claim 2 , wherein the state of the connected device is based on a resulting situation resulting from performance or non-performance of each of the one or more other functions.

4. the state of the connected device is based on a resulting state resulting from execution of the function; The method of claim 2 , wherein the composite graphical assistant interface includes a graphical interface element that illustrates the resulting situation.

5. The method of claim 4 , wherein the composite graphical assistant interface does not display the graphical control element corresponding to the function.

6. 2. The method of claim 1, wherein the user preferences characterizing the historical propensities characterize the historical propensities of the user to control one or more other features of the plurality of different features based at least on previous interactions between the user and an automated assistant accessible via an automated assistant interface of the client device.

7. the previous interaction corresponds to an interaction session between the user and the automated assistant prior to the user selecting the graphical control element; The method of claim 6 , wherein during the interactive session, the user provides a request to cause the connected device to perform the function.

8. A computer program comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 7.

9. A computer-readable storage medium having instructions recorded thereon that, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 7.

10. A system comprising one or more processors for carrying out the method of any one of claims 1 to 7.

Images