Efficient control and / or linking of smart devices

Abstract

To provide a method for efficient control and / or linking of smart network connected devices.SOLUTION: A method includes receiving user input provided via one or more user input devices, such as a touch screen display, a microphone, or another input device, directed to an active application executed on a client device, determining that the user input is directed to rendering one or more interfaces that cause control of one or more smart devices, sending a request addressed to a third party address, addressed to a third party associated with the smart devices to which the user input is directed, receiving a deep link in response to the request, and rendering at least one graphical interface element that controls the smart devices when interacted with via user interface input.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] The present invention relates to the efficient control and / or linking of smart devices. [Background technology]

[0002] Applications and interfaces have been proposed that allow a user to control each of multiple smart devices associated with the user (e.g., associated with the user's account). For example, some automated assistant interfaces allow a user to provide verbal, typed, touch, and / or other input to control various smart devices. Also, for example, some home control applications allow a user to interact with graphical elements and / or provide verbal input to control various smart devices. For example, some automated assistants include companion home control applications that allow a user to configure various aspects of the automated assistant, control various smart devices linked to the automated assistant, etc.

[0003] In many situations, one or more of the smart devices controlled via the assistant interface and / or home control application are manufactured by a corresponding third party (3P). Third party or 3P, as used herein, refers to a party separate from the party controlling the referenced automation assistant interface and / or home control application. First party or 1P, as used herein, refers to the party controlling the referenced automation assistant and / or home control application. 3Ps often provide a 3P application dedicated to the control and management of one or more of the 3P's 3P smart devices (optionally, not allowing control of any smart devices not manufactured by the 3P). For example, a particular 3P that manufactures smart lights may provide a 3P application dedicated to the control and management of the 3P's smart lights. Additionally, 1P may manufacture smart devices and provide a separate 1P application dedicated to the control and management of one or more of the 1P's smart devices (optionally, not allowing control of any smart devices not manufactured by the 1P).

[0004] Controlling multiple smart devices from an automation assistant interface and / or a home control application provides various computational efficiencies. For example, a home control application running on a client device may enable control of three different smart devices from three different 3Ps, all from within the home control application. This may be more computationally efficient than, for example, launching and running three separate 3P applications from three different 3Ps (which may use significant memory and / or processor resources) to enable control of the three different smart devices. Additionally, a lesser amount of user interface input may be used within the home control application to control the three different smart devices, at least as compared to the user interface input that would be required to launch three different 3P apps and navigate within each of the three different 3P apps to control the three different smart devices.

[0005] Despite the computational efficiencies afforded by the automation assistant interface and home control application, it is often the case that one or more of a user's smart devices are not linked to the automation assistant and / or home control application and therefore are not controllable via the automation assistant and / or home control application, and therefore less efficient means are used in controlling those unlinked smart devices.

[0006] Additionally or alternatively, the automation assistant interface and / or home control application may allow a user to control 1P and / or 3P smart devices (including smart devices from multiple disparate 3Ps), although in many instances the control provided may be limited compared to the control provided by separate 1P and / or 3P applications dedicated to controlling and managing smart devices. For example, for a 3P smart thermostat, the home control application may present a graphical interface element that allows a user to adjust the set point of the thermostat. However, other features of the 3P thermostat may not be adjustable by the home control application, but may be adjustable by a 3P application that is primarily dedicated to controlling the 3P thermostat and, optionally, other smart devices of the 3P. For example, the thermostat's mode (e.g., cool / hot / cool / hot), fan control (e.g., run the fan for 4, 6, or 8 hours), thermostat schedule, and / or other features of the 3P thermostat may be controllable via the 3P application but not via the home control application. In a situation where a user desires the control afforded by a 3P application, the user must locate and launch the 3P application, and often must navigate the 3P application to identify the interface for the 3P thermostat and / or the interface for the desired control. Summary of the Invention [Means for solving the problem]

[0007] Implementations disclosed herein are directed to the efficient control and / or linking of smart network-connected devices (also referred to herein as smart devices or Internet of Things (IoT) devices), such as smart home alarms, smart door locks, smart cameras, smart lights, smart thermostats, smart scales, smart beds, smart cleaning systems, smart garage door openers, smart plugs, smart appliances, smart baby monitors, smart fire alarms, smart humidity detectors, and the like.

[0008] Some implementations disclosed herein are directed to rendering, within an active application (e.g., the home control application described above) that enables control of multiple smart devices from multiple parties, both (1) interactive graphical interface elements that, when interacted with, cause corresponding control of the 3P's smart device, and (2) selectable elements that, when selected, cause the execution of a deep link that causes the 3P application to open with additional information about the smart device. The 3P application may be primarily (or exclusively) dedicated to control of the 3P's smart device.

[0009] The additional information about the smart device presented when the application opens in that state is in addition to the interactive graphical interface elements and / or any other information presented by the active application. For example, the additional information may include at least one additional graphical element that, when interacted with, causes control of one or more properties of the smart device, where control of the one or more properties is not enabled through interaction with the at least one interactive graphical interface element rendered via the application. In these and other ways, selection of the selectable element causes the 3P application to open in a state that allows the 3P further control of the smart device. This is more computationally efficient and requires a reduced amount of user input compared to instead having the user locate an icon for the 3P application, select the icon to launch the 3P application, and then perform one or more user inputs within the 3P application to navigate to a state that allows further control of the smart device.

[0010] In some of those implementations, the deep link is generated by the 3P in response to a request addressed to a specific address for the 3P (e.g., an address for the 1P or the 3P's application programming interface (API)) and including an identifier of the smart device. For example, the 3P may generate a deep link to cause the 3P application to open with additional information about the smart device indicated by the identifier. By having the 3P generate the deep link rather than relying on a deep link stored by the 1P of the active application, the 3P can generate the deep link to conform to the current schema of the 3P application, thus preventing the use of an outdated (or "broken") stored link that no longer conforms to the current schema. Furthermore, the memory and / or storage resources of the 1P may be saved by not having to store a large number of deep links, but instead relying on deep links dynamically provided by the 3P. Still further, the 3P may optionally generate the deep link in light of recent events related to the smart device that are accessible to the 3P but may not be accessible to the 1P of the active application. This can allow the 3P application to be opened in a state tailored to the recent event, thus providing relevant additional information with reduced latency. As one non-limiting example, the 3P smart device can include a 3P smart camera, and a deep link generated at a given time can link to a state of the 3P application that causes the 3P application to provide a video clip of a recent activity detection that occurred via the 3P camera. Such a deep link can be generated based on the 3P determining that the recent activity detection occurred within a threshold amount of time prior to the request. For example, the 3P can generate such a deep link if the recent activity detection occurred less than five minutes after the request.If no recent activity detection occurs, the 3P may generate an alternative deep link, such as a deep link that causes the 3P application to open showing a current activity feed and allows adjustment of smart camera features (e.g., zoom, pan, and / or tilt) that are not adjustable by the active application.

[0011] In various implementations in which a deep link is generated by a 3P in response to a request addressed to a specific address for the 3P, the request is generated by an active application in response to a user interface input determined to be a request to render a graphical interface element that allows control of the 3P's smart device. For example, the active application may first present icons for multiple heterogeneous smart devices that can be controlled, including the 3P's specific smart device and other smart devices (of the 3P and additional 3Ps). In response to the selection of the icon for the 3P's specific smart device, the active application may generate a request identifying the specific smart device and send the request to the specific 3P address. As another example, the active application may receive an utterance that is a request to render a graphical interface element that allows control of the 3P's smart device, and in response thereto, generate and send a request to the specific 3P address. For example, the utterance may be "turn on the couch light" and may be determined to be a request to render a graphical interface element that allows control of the smart device. For example, an utterance may be determined to be a request to send a command to turn a "couch light" "on" and render a graphical interface element that allows control of a dim level and / or other property of the "couch light." The request may be generated and / or transmitted in response to determining that an utterance is a request to render a graphical interface element. In these and other ways, requests to the 3P are only generated and / or transmitted in response to input that is a request to render a graphical interface element that allows control of the 3P's smart device.Thus, in various implementations, the computational resources used in generating the request and / or the network bandwidth used in transmitting the request are only used when certain conditions are met (e.g., only when the input is determined to be a request for rendering a graphical interface element that allows control of the smart device). Moreover, in some of these and / or other implementations, the request is generated and / or transmitted in further response to determining that the 3P application is installed on the client device that executes the active application. In these and other ways, generating and / or transmitting the request to the 3P may be bypassed when it is determined that the 3P application is not installed. In some of those situations, a default deep link for the 3P may be used instead, such as a default deep link that links to a web page or a "Play Store" application state that either enables installation of the 3P application.

[0012] Some implementations disclosed herein are additionally or alternatively directed to determining that a 3P application is installed on a client device associated with a user account, the 3P application enabling the 3P to control the 3P smart devices. The implementations further determine whether the smart device topology includes any entries for any of the 3P smart devices of the 3P. The smart device topology is associated with the user account and is used by the automation assistant in controlling the smart devices through interactions with the automation assistant. The smart device topology may define, for example, the smart devices, the automation assistant devices, the manufacturers of each, the locations of each (e.g., "living room", "kitchen"), the semantic identifiers of each (e.g., "couch light", "living room assistant device"), and / or other characteristics of the smart devices and / or the automation assistant devices.

[0013] In response to determining that the 3P application is installed on the client device but the smart device topology lacks any entry for the 3P smart device, an output may be rendered on the client device associated with the user account or an additional client device, the output including a suggestion to add the 3P to the smart device topology. In response to receiving a positive user interface input in response to rendering the output, one or more specific smart devices of the 3P may be efficiently added to the smart device topology. For example, the user account may be used to interface with an application programming interface (API) of the 3P in automatically adding at least some of the specific devices of the one or more 3P smart devices to the smart device topology. Also, for example, a sign-in interface may be presented that allows signing in to a 3P account specific to the 3P and that, when used to sign in to the 3P account, causes an interface with the API of the 3P, or an additional API, in automatically adding the specific device to the smart device topology. As yet another example, the specific smart device may be added to the smart device topology by interfacing with a third-party application installed on the client device, as described in more detail herein. In these and other ways, a particular smart device is added to the device topology in a more efficient manner than navigating, for example, through a home control application to a screen where 3P devices can be added, identifying the 3P from a long list of available 3Ps (e.g., scrolling through the list), and then selecting the identified 3P from the list to be presented with a sign-in interface that allows the user to sign in to a 3P account specific to the 3P.

[0014] Furthermore, once 3P smart devices are added to the smart device topology in response to a prior suggestion to add 3P smart devices to the smart device topology, control of those 3P smart devices is enabled through the automation assistant interface and / or the home control application. Such control provides various computational efficiencies. For example, a home control application running on a client device may enable control of the 3P smart device, as well as smart devices from other 3Ps, all from a single application. This may be more computationally efficient than launching and running multiple separate 3P applications (which may use significant memory and / or processor resources), for example, to enable control of different smart devices.

[0015] Some implementations disclosed herein are directed to directly interfacing with a third-party application in automatically adding a 3P smart device to a smart device topology in response to determining that a 3P application is installed on a client device and that the associated smart device topology lacks any entries for any of the 3P smart devices of the 3P. In some of those implementations, a particular application (e.g., an automation assistant application or a home control application) of the client device may authenticate the 3P application. If the 3P application is authenticated, the particular application may provide an intent to execute a third-party application with a linking function. For example, an intent may be provided to an operating system of the client device to cause the operating system client device to execute the third-party application by invoking the intent. The intent may be specific to the linking function of the third-party application, which, when executed by the third-party application, causes the third-party application to return at least one access token (e.g., a long-term and / or short-term access token) if one or more conditions are satisfied. The one or more conditions may include, for example, validation of the application providing the intent, as described in more detail below. At least one access token may be received from the third-party application in response to providing the intent. Furthermore, the at least one access token may be used to obtain additional information related to the 3P smart device, and at least a portion of the obtained additional information may be added to the smart device topology. The access token may also enable control of the 3P smart device by the automation assistant (e.g., by including the device in a request from the automation assistant and / or in obtaining an updated short-term access token).Adding 3P smart devices to the smart device topology can enable control of those smart devices from a single application, which can be resource efficient as described above. Moreover, the local interaction at the client device between the automation assistant application or home control application and the 3P application when adding the smart device can use minimal network resources when adding the smart device.

[0016] The above is provided as a summary of some implementations disclosed herein. Further descriptions of these and other implementations are provided below.

[0017] Further, some implementations include one or more processors (e.g., a central processing unit (CPU)), a graphics processing unit (GPU, and / or a tensor processing unit (TPU) of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, where the instructions are configured to cause performance of any of the methods described herein. Some implementations also include one or more non-transitory computer-readable storage media that store computer instructions executable by the one or more processors to perform any of the methods described herein.

[0018] It should be appreciated that all combinations of the above concepts and additional concepts described in more detail herein are contemplated as being part of the subject matter disclosed herein, for example, all combinations of claimed subject matter listed at the end of this disclosure are contemplated as being part of the subject matter disclosed herein. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented. [Diagram 2]FIG. 2 illustrates an example interface for a home control application. [Diagram 3] FIG. 1 illustrates an example interface for a third-party application that is primarily dedicated to controlling one or more third-party smart devices. [Figure 4] 3 illustrates an example interface of the home control application of FIG. 2, the example interface including a selectable graphical element associated with the generated deep link. [Diagram 5] FIG. 1 illustrates an example interface that includes graphical interface elements for controlling various smart devices and includes suggestions for adding third-party smart devices to a smart device topology. [Figure 6] 1 is a flowchart illustrating an example method according to various implementations disclosed herein. [Figure 7A] 1 is a flowchart illustrating another example method according to various implementations disclosed herein. [Figure 7B] 10 is a flowchart illustrating yet another example method according to various implementations disclosed herein. [Figure 8] FIG. 1 illustrates an example architecture of a computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] Some implementations described herein are directed to rendering a graphical interface of an active application with selectable graphical elements that, when selected, cause the execution of a deep link to the 3P application. The deep link may be generated in response to a request from the active application indicating the 3P smart device with which the user is currently interacting through the active application. The deep link may be generated "on the fly" by the 3P, and allows the user to access the 3P application directly from the active application and to access the state of the 3P application that is specific to the 3P smart device currently interacting through the active application.

[0021] In some implementations, a user interface input directed to an active application executing on the client device is received. The input may be related to a particular functionality the user is accessing for controlling one or more 3P smart devices. For example, the input may be a request to render one or more graphical interface elements associated with controlling the 3P smart devices. Also, for example, the input may be a request to render a graphical interface that allows the user to control the brightness of one or more 3P smart lights and / or turn one or more 3P smart lights on / off. The one or more graphical interface elements rendered in response to the request may include one or more sliders for adjusting the brightness of the 3P smart lights and / or one or more toggle buttons for turning the 3P smart lights on or off.

[0022] In response to the input on the 3P smart device, a request is generated and sent by the active application to a 3P system corresponding to the 3P smart device. The 3P system is associated with a 3P application that is specifically designed to control smart lights manufactured by 3P, and the 3P application may be installed on the same client device on which the active application (e.g., a home control application) is running.

[0023] In some instances, the 3P application includes additional functionality for the 3P smart device that is not available via the active application. Continuing with the smart light example, the 3P application may include functionality for programming the 3P smart light to automatically turn on at a particular time, change the color temperature of the smart light, and / or control other functionality of the smart light apart from the functionality available via the active application. Thus, the user may be required to access the 3P application to provide input related to the additional functionality.

[0024] The request to the 3P system may include, for example, an identifier of the particular smart device that the user is controlling through the active application, the particular interface that the user is accessing, the functionality available through the particular interface, and / or additional information regarding the user's current interaction with the active application.

[0025] The 3P system can determine the relevant interfaces and / or functionality of the targeted smart device that is available through a third-party application. For example, a user may be accessing an interface that allows control of a smart light named "Living Room Light." The request can include an identifier for the smart light, allowing the 3P to identify the relevant interfaces for that smart device.

[0026] In response to the request, the 3P system may send a deep link, which is a deep link to the 3P application. The active application may render interactive graphical interface elements that allow the user to control the targeted smart device. Additionally, the active application may render selectable graphical elements that correspond to the provided deep link (e.g., selectable icons and / or buttons) simultaneously with the interactive graphical interface elements. Thus, when the user is presented with interactive graphical interface elements via the active application, selectable graphical elements may also be presented that, when selected, may cause the opening of the 3P application in a smart device-specific state.

[0027] Some implementations described herein are additionally or alternatively directed to configuring one or more 3P smart devices for control via an automation assistant and / or a home control application. Some of those implementations determine that a 3P application is installed on a client device associated with a user account, where the 3P application enables control of the 3P smart devices of the 3P. Those implementations further determine whether a smart device topology includes any entries for any of the 3P smart devices of the 3P. The smart device topology is associated with the user account and is used by the automation assistant in controlling the smart devices by interacting with the automation assistant. In response to determining that the 3P application is installed on the client device but the smart device topology lacks any entries for the 3P smart devices, an output may be rendered at the client device associated with the user account or an additional client device, where the output includes a suggestion to add the 3P to the smart device topology. In response to receiving a positive user interface input in response to rendering the output, one or more specific smart devices of the 3P may be efficiently added to the smart device topology.

[0028] 1, an example environment in which the techniques disclosed herein may be implemented is shown. The example environment includes a client device 105, smart devices 145A and 145B, a remote home control server 120, and a third-party (3P) remote server 110. 1-N As described herein, the 3P remote server 110 1-NEach of the 3Ps may be associated with a different corresponding 3P of the plurality of 3Ps. For example, the 3P1 remote server 1101 may be associated with a first 3P that manufactures one or more corresponding smart lights, the 3P2 remote server 1102 may be associated with a second 3P that manufactures one or more corresponding smart lights, and the 3PN remote server 110 N may be associated with a third 3P that manufactures one or more corresponding smart cameras, smart thermostats, and smart alarm systems. 1-N 1 as a single component, it will be understood that any one or more modules and / or aspects may be implemented in whole or in part by multiple devices. For example, in some implementations, one or more aspects described as being implemented by a remote server may be implemented by a cluster of high performance remote server devices that handle requests from one or more client devices as well as requests from additional devices. Also, one or more components described as components of a client device 105 may be implemented by one or more remote components that interface with the client device 105 over one or more networks.

[0029] The client device 105 may be a mobile phone computing device, a tablet computing device, a standalone device primarily dedicated to assistant functionality, and / or a user wearable device including a computing device (e.g., the user's watch with a computing device, the user's glasses with a computing device, a virtual or augmented reality computing device). Additional and / or alternative client devices may be provided. Furthermore, one or more components of the client device 105 may be implemented on separate devices. For example, the graphical display 107 may be implemented on one or more alternative computing devices that communicate with the client device 105. The client device 105, the remote home control server 120, the third party (3P) remote server 110, the remote home control server 12 ... 1-N, and / or smart devices 145A and 145B can communicate via one or more communication networks. The communication networks may include, for example, a local area network (LAN) and / or a wide area network (WAN) (e.g., the Internet). Smart device 145A is manufactured by a first 3P associated with 3P1 remote server 1101, and can be controlled by commands sent from 3P1 remote server 1101 (optionally in response to corresponding commands communicated via a corresponding one of home control application 115, third party application 125, and / or automation assistant application), and / or can be directly controlled by client device 105. The smart device 145B is manufactured by a second 3P associated with the 3P2 remote server 1102 and can be controlled by commands sent from the 3P2 remote server 1102 (optionally in response to a corresponding command being communicated via the home control application 115, a corresponding one of the third party applications 125, and / or an automation assistant application) and / or can be controlled directly by the client device 105.

[0030] The client device 105 includes a display 107 that can render content to a user of the client device 105 and is optionally touch-sensitive to allow touch input to be received via the display 107. For example, a user can use the display 107 to interact with a home control application 115 to control one or more smart devices of the user. The user can view one or more interfaces via the display 107, which are rendered by the rendering engine 117 or other rendering engines. The interfaces allow the user to provide input and / or view output via the display 107. The user can interact with the display 107 via one or more input devices, such as a keyboard, a mouse, and / or other input devices that can select areas of the display 107, voice control, touch screen control, and / or other input methods that allow the user to submit input and select content to be rendered. Input received via the display 107 can be provided to one or more applications running on the client device 105, such as the home control application 115 and / or third party applications 125.

[0031] The home control application 115, when executed and active on the client device 105, can be used by a user to control one or more of the user's smart devices, such as smart devices 145A and 145B. The home control application 115 can optionally interface with a remote home control server 120 in performing some or all of the functions described herein. A smart device may be an intelligent, multi-sensing, network-connected device configured to communicate with a central server, a cloud computing system, and / or multiple additional smart devices to provide any of one or a variety of useful smart home services. For example, a smart device may be or may be otherwise associated with an air conditioning system, a lighting device, a home theater and entertainment system, a security system, an automatic door locking system, a thermostat device, a home automation system, a sound speaker, a camera device, a treadmill, a weighing scale, a smart bed, a washing system, a garage door opener, an appliance, a baby monitor, a fire alarm, and / or other suitable devices or systems.

[0032] One of the third party applications 125 installed on the client device 105 may optionally be provided by a first 3P that manufactured the smart device 145A and may be used to control the smart device 145A. Such a third party application may optionally enable control of the smart device of the first 3P, but may not enable control of the smart device of the other 3P (e.g., any other 3P). Furthermore, another one of the third party applications 125 installed on the client device 105 may optionally be provided by a second 3P that manufactured the smart device 145B and may be used to control the smart device 145B. Such a third party application may optionally enable control of the smart device of the second 3P, but may not enable control of the smart device of the other 3P (e.g., any other 3P). For example, one of the third party applications 125 may render, via the display 107, a graphical user interface that provides one or more graphical elements that allow a user to control the smart device 145A.

[0033] A home control application 115 is also installed on the client device 105 and enables control of smart devices from multiple 3Ps, including smart devices 145A and 145B, as well as potentially other smart devices (not shown) configured for the home control application 115 (e.g., assigned in a smart device topology used by the home control application 115). The home control application 115 includes a rendering engine 117, which can render an interface including one or more graphical elements that allow a user to control smart device 145A and / or smart device 145B. User input provided via the display 107 and / or other input devices may be provided to the home control application 115, which can provide instructions to the rendering engine 117 to process the input and render one or more additional graphical interfaces and / or update the active interface.

[0034] The home control application 115 may include some or all of the same controls as one or more of the third party applications 125. For example, the smart device 145A may be a smart lighting unit manufactured by a first 3P that also provides one of the installed third party applications 125. The first 3P's third party application enables control of the smart lighting unit 145A. Additionally, the home control application 115 may enable control of the smart lighting unit 145A, for example, through one or more graphical interface elements rendered by the home control application 115 (via the rendering engine 117) and by communicating corresponding control commands to the lighting unit 145A directly or to the 3P's application program interface ("API") (e.g., via the 3P1 remote server 1101), which then generates and provides the corresponding control commands to the lighting unit 145A. Additionally or alternatively, the smart device 145B may be another device manufactured by a second 3P that also provides another one of the installed third party applications 125. The smart device 145B is also controllable via the home control application 115.

[0035] In some instances, the home control application 115 allows a user to control only a subset of the functionality of the smart device 145A and / or the smart device 145B, as compared to the functionality that may be controlled via the respective third-party application 125. For example, referring to FIG. 2, an exemplary interface 200 of the home control application 215 is shown. The interface 200 includes an “on / off” toggle button 205 that is an interactive graphical element and allows a user to turn a lighting unit on or off with an input directed to the toggle button 205. Additionally, the interface 200 includes a slider element 210 that is another interactive graphical element and allows a user to adjust the brightness of the same lighting unit with an input directed to the slider element 210. In contrast, referring to FIG. 3, an exemplary interface 300 of one of the third-party applications 125 is shown. The interface 300 includes a toggle button 305 that allows a user to turn the same lighting unit on and off. Additionally, the interface 300 includes a slider 310 for adjusting the brightness of the lighting unit. However, interface 300 further includes a slider 315 for adjusting the color temperature of the lighting unit, as well as a scene button 320 that, when selected, allows a user to select one of a number of configured scenes for the lighting unit. This functionality is not included with interface 200. Thus, third party application 125 includes more options for controlling the lighting unit than does home control application 115.

[0036] In some implementations, a user may be interested in accessing functionality for a smart device that is included in the interface of one of the 3P applications 125 but not included in an active application, such as the home control application 115. For example, a user may be interested in adjusting the color of a lighting unit, but upon opening the interface of the home control application 115, notice that there is no graphical element for color control. Furthermore, in some cases, a user may access a control interface of a home control application 115 that includes less functionality than a corresponding one of the third party applications 125, but may not realize that additional functionality is available for the smart device. In these cases, to adjust the color of a lighting unit, a user would be required to navigate away from the active home control application 115, locate an icon for a corresponding one of the third party applications 125, tap the icon to launch the third party application 125, and then navigate within the 3P application 125 until a state of the 3P application 125 is reached that allows for the adjustment of the color of the lighting unit.

[0037] In some implementations, rather than requiring the user to perform multiple inputs to locate the 3P application, launch the 3P application, and navigate within the 3P application to a desired state of the 3P application, selectable interface elements may instead be presented within the home control application 115 that, when selected, execute a deep link that launches the corresponding 3P application in the desired state. In various implementations, the deep link is directed to a 3P corresponding server (e.g., the 3P remote server 110 1-NIn this manner, the deep link may be generated in view of the smart device being viewed and / or otherwise interacted with via the home control application 115. Additionally, the deep link may be generated in view of any updates made to the corresponding 3P application and / or the corresponding 3P backend interface, such as updates to a schema used by the 3P application and / or the 3P backend interface to cause the 3P application to open in a desired state.

[0038] For example, referring again to FIG. 1 , smart device 145A may be controlled by one of the 3P applications 125, and the user may be accessing an interface of the home control application 115 that is directed to control of only smart device 145A. Rather than providing only a link in the interface of the home control application 115 to an interface of one of the 3P applications 125 that is not specific to smart device 145A (e.g., directed to the “Home” interface of that third party application 125), implementations disclosed herein may additionally or alternatively provide a deep link that is specific to smart device 145A. For example, the deep link, when executed, causes the corresponding one of the 3P applications to open with controls specific to smart device 145A, including controls not provided via the home control application 115.

[0039] In some implementations, the request engine 119 of the home control application 115 may generate the request, which may include a reference to a particular smart device. The request engine 119 may generate the request in response to received input (received via the home control application 115) that includes a request for rendering one or more interactive graphical interface elements that enable control of the particular smart device. The request may include a reference to a particular smart device and may be generated by the 3P remote server 110 based on the 3P being associated with a 3P that is also the manufacturer of the particular smart device. 1-N The request engine 119 can communicate with the 3P remote server directly or via a remote home control server 120. The request engine 119 or the remote home control server 120 communicates with the home control application 115 and the 3P remote server 110. 1-N The request may be sent to the 3P remote server using an API, such as the API used for communication between one or more of the 3P remote servers.

[0040] In some implementations, the home control application 115 may determine that the client device 105 does not include a corresponding 3P application for a particular smart device being interacted with via the home control application. For example, the 3P application associated with the smart device 145A may not be installed on the client device 105. In some of those cases, the home control application 115 may bypass generating a request to minimize network traffic in light of any generated deep links for the 3P application that are not executable on the client device 105 (because the 3P application is not installed). Furthermore, in some of those cases, a default link for the 3P may be used instead, such as a link to a particular web page for the 3P, such as to an application store state that allows the 3P application to be downloaded.

[0041] Based on the information contained in the request, the 3P remote server 110 1-N The corresponding one of the 3P remote servers 110 generates a deep link directed to the specific state of the 3P application associated with the same smart device associated with the request. For example, the request may include an identifier for the smart device on which the interactive graphical element is being rendered via the home control application 115. The request may be forwarded to the 3P remote server 110. 1-N , and the request can generate a deep link to provide to the home control application 115 based on the identifier for the smart device, which, when executed, causes the client device 105 to open a corresponding one of the third party applications specialized for the same identified smart device.

[0042] In some implementations, the identifier of the smart device included in the request may be different from the unique identifier used by the corresponding 3P remote server and / or the corresponding 3P application. The identifier may be a key used in communication between the home control application 115 and the 3P remote server, and may be mapped to the corresponding unique identifier by the 3P remote server. For example, the home control application 115 may have an identifier for the smart device of "12345". However, a third party may have a unique identifier for the same device of "67890" and may maintain a local mapping between the "12345" identifier and the unique identifier of "67890".

[0043] As an example, a user, via the home control application 115, may provide input that causes the home control application 115 to render the interface shown in FIG. 2. The interface allows for control of a particular smart device. In response to the input, the request engine 119 generates a request that identifies the particular smart device and transmits the request to the 3P remote server 110. 1-N 3. In response to receiving the request, the 3P remote server may generate a deep link to the interface shown in FIG. 3 and provide the deep link to the rendering engine 117. The rendering engine 117 may then use the deep link to generate graphical elements that may be included in the accessed interface shown in FIG. 2. For example, referring to FIG. 4, an interface 400 is shown that includes elements 405 and 410 that are the same as elements 205 and 210 included in the interface shown in FIG. 2, but also includes a button 415. The button 415 may be associated with a deep link, and when the button 415 is selected, the user may then be directed to the interface of FIG. 3. For example, selection of the button 415 may open a corresponding one of the 3P applications 125, causing the corresponding one of the 3P applications to be opened along with the interface of FIG. 3. Thus, the user may be directed to the interface 300 that allows user-added functionality not available via the interface of FIG. 4 (i.e., via the color control 315 and / or scene button 320 included in the interface 300).

[0044] In some implementations, the provided deep link may incorporate or be sent along with additional information indicating the particular selectable graphical element that should be presented in FIG. 4. For example, if the deep link is directly to a state that allows only color control of the smart light, the additional information may indicate a particular selectable graphical element having a color wheel or other indicia indicating color control. Also, for example, if the deep link is directly to a state that allows configuration of the smart light's schedule, the additional information may indicate a particular selectable graphical element that includes a calendar or other indicia indicating schedule control. In these and other ways, the 3P may be given control over what deep links are provided (and, as a result, the state in which the 3P application is opened), as well as what graphical elements are used to indicate the functionality of the deep link.

[0045] As another example, a user may access an interface of a home control application 115 targeted to a camera smart device. Only through the home control application 115 can the user view a live video feed from the camera smart device. However, additional functionality may be provided via the 3P application for the camera smart device, such as functionality to allow the user to view video footage captured by the camera smart device from one or more past events, functionality to allow the user to zoom / pan / tilt the camera, etc. In such an example, the 3P remote server 110 1-NA corresponding one of the 3P applications 125 may generate a deep link that, when executed, opens the corresponding one of the 3P applications 125 with footage from the past event and / or allowing zoom / pan / tilt adjustments. The 3P remote server may optionally determine to generate a deep link that causes the 3P application to open with footage from the past event in response to determining that one or more criteria are satisfied. For example, such a deep link may be generated by the 3P remote server using information specific to the 3P, such as information indicating that a past event has occurred and / or has occurred within the last X minutes.

[0046] In some implementations, the home control application 115 or the automation assistant application can identify one or more third parties whose corresponding devices the user has not yet configured to be controlled via the home control application 115 and / or via the automation assistant application. The automation assistant application can be optionally associated with the home control application 115 but separate from it. For example, the automation assistant application and the home control application 115 can both be controlled by the same 1P. For example, a third party application can be installed on the client device 105, but the user has not gone through one or more steps to integrate the home control application 115 and / or the automation assistant application into communication with the user's smart device associated with the 3P application 125. Because the user may not be aware of the protocols for setting up the smart device and / or may not realize that the smart device is not yet controllable by the home control application, the user may be helped by one or more notifications from the 3P that the smart device is ready to be configured.

[0047] In some implementations, the home control application 115 may use the device topology representation to identify that smart devices from a particular 3P are not included in the device topology. The device topology representation may be assembled using various suitable techniques and may be any suitable format and / or structure capable of representing the topology and organization of one or more smart devices within one or more structures. The device topology representation may define and / or organize one or more structures associated with a user (e.g., a user account of a user). For example, the structures associated with a user may be a home, a business, a vehicle, a workplace, and / or any other suitable structure associated with the user. The user may organize each structure into one or more rooms (e.g., a bedroom, a living room, a kitchen) and may assign one or more smart devices to the one or more rooms. For example, a user may assign one or more smart lights to a first room of a particular structure and one or more door locking systems to a second room of the structure. The user may then assign one or more different smart devices to one or more rooms in the second structure.

[0048] A user may assign one or more identifiers to the smart devices in the device topology representation. An identifier may be chosen by the user and associated with each smart device in the device topology representation. The identifier may include a nickname and / or alias chosen by the user for the smart device (e.g., couch lamp, front door lock, bedroom speaker, etc.). In this manner, the identifier may be the name or alias of each smart device that the user is likely to use when providing voice utterances to control the smart device.

[0049] The device topology representation may further specify one or more device attributes associated with each device. The device attributes may be associated with one or more actions that the smart device is configured to perform. For example, device attributes associated with a lighting device may include a turn on capability, a turn off capability, a dim capability, and / or other suitable attributes associated with the lighting device. As another example, an automatic door locking device may include a lock capability, an unlock capability, and / or other suitable capabilities.

[0050] Rather than providing a user with a suggestion to configure a smart device for any 3P that lacks any smart devices in the device topology representation, implementations disclosed herein provide a suggestion to configure a 3P's smart device based in part on determining that the 3P's 3P application is installed on a client device associated with the user's user account, where the 3P application enables the 3P to control one or more 3P smart devices. In these and other ways, a suggestion to configure a 3P's smart device may only be provided in response to determining that the user has installed a 3P's application that enables the 3P to control the 3P's smart devices. This ensures that the provided suggestion is for a 3P that the user is likely to have an associated smart device.

[0051] In some implementations, suggestions for configuring smart devices may be provided through the home control application 115. For example, referring to FIG. 5, an interface for a home control application is provided. The interface 500 includes graphical elements 505, 510, and 515 that allow a user to change the status of "Light 1," "Light 2," and "Light 3." Additionally, the interface 500 includes a suggestion 520 indicating that no smart devices from a particular 3P ("3P ABC") are configured, along with a prompt to add the device to the device topology representation. Upon positive input from the user (e.g., speaking "YES" or selecting the interface element "YES"), the device configuration engine 160 contacts the 3P remote server 110 to request the configuration of the smart devices. 1-N Any of the user's devices can be added to the device topology representation by interacting with the corresponding one of

[0052] In some implementations, in response to receiving an affirmative input from the user, the device configuration engine 160 may configure the 3P remote server 110 using a user account associated with the home control application 115 and / or the automation assistant application. 1-N For example, the user account may be used by the user for the corresponding 3P, allowing the device configuration engine 160 to use the user account to validate the user at the corresponding 3P without having to prompt the user for additional sign-in information. The device configuration engine 160 may interface with an application programming interface (API) of the 3P in automatically adding the particular device and a third-party application associated with the unconfigured smart device to a smart device topology consisting of at least some of the one or more 3P smart devices to configure the device to be controllable by the home control application.

[0053] In some implementations, in response to receiving affirmative input from the user, the device configuration engine 160 presents a sign-in interface that allows for signing in to a 3P-specific 3P account and that, when used to sign in to the 3P account, triggers an interface with 3P's API or additional APIs in automatically adding the particular device to the smart device topology. In some of those implementations, the device configuration engine first attempts to sign in with a user account associated with the home control application 115 and / or the automation assistant application, and then presents the sign-in interface only if such attempt fails.

[0054] In some implementations, in response to receiving a positive input from the user, the device configuration engine 160 interfaces with one of the third party applications 125 (e.g., via the operating system of the client device 105) in automatically adding the 3P smart device to the smart device topology. In some of those implementations, a particular application (e.g., the home control application 115) can incorporate all or aspects of the device configuration engine 160 and can authenticate the third party application. If the third party application is authenticated, the device configuration engine 160 can provide an intent to execute the third party application with a linking function. For example, the intent can be provided to the operating system of the client device 105 to cause the operating system client device to execute the third party application by invoking the intent. The intent can be specific to the linking function of the third party application, which, when executed by the third party application, causes the third party application to return at least one access token (e.g., a long-term and / or short-term access token) if one or more conditions are satisfied. The one or more conditions can include, for example, validation of the application granting the intent, as described in more detail below. At least one access token may be received from the third-party application in response to granting the intent. Additionally, the device configuration engine 160 may use the at least one access token to obtain additional information related to the 3P smart device, at least a portion of which may be added to the smart device topology. The access token may also enable control of the 3P smart device (e.g., by including the device in a request from an automation assistant and / or upon obtaining an updated short-term access token).

[0055] In some implementations, suggestions for configuring the smart device may be provided to the user only when the user is accessing the home control application, as shown in FIG 5. However, in some implementations, the suggestions may additionally or alternatively be provided regardless of whether the user is accessing the home control application and / or any other active applications. For example, the suggestions may appear on the home screen of the client device, through one or more notification messages via an automated assistant and / or any other interface elements executing on the client device, regardless of whether any applications are active.

[0056]

[0023] Figure 6 shows a flow chart of an exemplary method. The steps of Figure 6 may be performed by one or more processors, such as one or more processors of a client device. Other implementations may include additional steps other than those shown in Figure 6, may perform the steps of Figure 6 in a different order and / or in parallel, and / or may omit one or more of the steps of Figure 6.

[0057] In step 605, user input directed to an active application executing on the client device is received. The user input is provided via one or more user input devices, such as a touch screen display, a microphone, or other input device. In some implementations, the active application may be a home control application, such as home control application 115 of FIG.

[0058] In step 610, one or more components determine that the user input is directed to rendering one or more interfaces that cause control of one or more smart devices. The smart devices may share one or more characteristics with smart devices 145A and / or 145B as illustrated in FIG. 1. For example, the smart devices may include thermostats, smart lighting units, cameras, and / or other devices manufactured by third parties associated with one or more software components executing on the client device and / or one or more other devices. The active application's interface provides one or more inputs via one or more components to enable the user to control, via the inputs, one or more of the third-party smart devices.

[0059] In step 615, a request is sent to a third party associated with the smart device to which the user input of step 605 is directed, the request being addressed to the third party's address. The request includes at least an identifier of one or more particular smart devices. For example, a user can provide input to control smart lighting unit "123" associated with third party "ABC", and a request can be generated that is addressed to the system address associated with "ABC" and includes the identifier of lighting unit "123".

[0060] In step 620, in response to providing the request, a deep link is received, the deep link to a third party application installed on the client device. The deep link is generated (optionally in real time) by the third party system in response to the request of step 615 and is configured to cause the third party application to open in a state presenting additional information about one or more particular devices. For example, third party "ABC" may be associated with an application that, when installed on the client device, allows a user to control lighting unit "123" through one or more interfaces directed to lighting unit "123". When accessed using the deep link, the third party application presents additional information that is not included with the interface of the active application. For example, the active application may allow a user to turn a lighting unit on or off, but does not allow the user to control the color emitted by the lighting unit; however, the interface associated with the deep link may include such functionality.

[0061] At step 625, at least one graphical interface element is rendered that, when interacted with via further user interface input, allows for control of the smart device. Additional graphical elements are simultaneously rendered that, when interacted with (e.g., selected), trigger the execution of a deep link. For example, the execution of a deep link may open a third party application to present additional information about the particular smart device. For example, the additional graphical element may be a selectable button provided with an active application's interface directed to the lighting unit "123". Interacting with the button may execute a deep link to cause the third party application to open an interface associated with the lighting unit "123" and including additional information related to the lighting unit "123". The additional information may include, for example, one or more graphical elements that allow the user to change the emitted color temperature of the lighting unit "123".

[0062] 7A shows a flowchart of an example method. The steps of FIG. 7A may be performed by one or more processors, such as one or more processors of a client device. Other implementations may include additional steps other than those shown in FIG. 7A, may perform the steps of FIG. 7A in a different order and / or in parallel, and / or may omit one or more of the steps of FIG. 7A.

[0063] At step 705, it is determined that a third party application is installed on the client device. The third party application may enable control of one or more smart devices of a third party and may share one or more characteristics with the third party application 125 as described herein with respect to FIGURE 1. In some implementations, the third party application may be determined based on a package name, or other identifier, of the third party application and may be identified based on a stored association of the third party application with a smart device control.

[0064] In step 710, it is determined that the user account of the client device and / or the smart device topology associated with the client device does not include any entries for third-party smart devices. The user account may include, for example, a user account associated with an automation assistant accessible via the client device (e.g., via an automation assistant application on the client device). The smart device topology may be used by the automation assistant in controlling various smart devices.

[0065] In step 715, in response to determining that the smart device topology does not include an entry for any of the third-party smart devices, an output is rendered that includes a suggestion to the user for adding the third-party smart device to the smart device topology. The suggestion may share one or more characteristics with the suggestion shown in Figure 5. For example, the suggestion may include an identifier of the third party and one or more graphical elements that allow the user to select whether to add the third-party smart device to the smart device topology.

[0066] At step 720, a positive user input is received in response to rendering the output at step 715. For example, the output may include a selectable interface element to confirm that the user wants to add a smart device, and the positive user input may be a selection of the interface element. Also, for example, the positive user input may be a verbal "yes" or "Yes, add smart device" provided in response to rendering the output at step 715. If a positive user input is not received (e.g., within a threshold amount of time of rendering the output at step 715) or a negative user input is instead received (e.g., a verbal "no", a selection of a no interface element, or a rejection of the output), the method of FIG. 7 may end.

[0067] In response to receiving a positive input in step 720, one of steps 725, 730, and 735 may be performed. In some implementations, more than one of steps 725, 730, and 735 are attempted (in any order and / or in parallel) and only one is successful. In some of those implementations, step 730 is only performed upon determining that step 725 and / or step 735 cannot be successfully performed. As described herein, step 725 and / or step 735 may eliminate the need for the user to provide any further input (or at least reduce the duration of input compared to step 730), which may be advantageous for less dexterous users and / or other users. Thus, user input is further reduced at least when step 725 and / or step 735 are performed.

[0068] In step 725, the user account is used to interface with the third-party API to automatically add at least some of the smart devices to the topology. For example, the user account used for the assistant on the client device may also be accepted by the third party. For example, the user account may be a single sign-on account, used by the user as a user account for the assistant and one (only one, or one of many) account for the third party. When interfacing with the third-party API, the single sign-on account may be used as authorization, and an access token and / or additional information may be provided by the third-party system during interfacing. The access token may be used to enable control (directly or indirectly) of the third-party smart device by the assistant, and the additional information may include, for example, an identifier of the third-party smart device associated with the user account, other attributes of the smart device, and / or other information related to the smart device. In response to interfacing with the third-party API and obtaining the token and / or other information, the smart device topology may be updated to include an identifier of the smart device and one or more nodes associated with the smart device, so that the automated assistant can subsequently control the smart device.

[0069] At step 730, a sign-in interface specific to the third-party account is presented. The interface may be used to sign in to the third-party account to interface with the third-party's API using the third-party account separate from the account associated with the automated assistant. Once the third-party account is signed in, the smart device may be added to the smart device topology. For example, a user may use the interface to enter a username and / or password, and the third-party's API may be accessible only after confirmation that the user has an account with the third party. Once confirmed, one or more components may receive an access token and other information to interact with the third party via the API to enable control of the smart device and / or to add the smart device to the topology.

[0070] In step 735, one or more steps of FIG. 7B are performed. FIG. 7B illustrates a flowchart of an exemplary method. The steps of FIG. 7B may be performed by one or more processors, such as one or more processors of a client device. Other implementations may include additional steps other than those shown in FIG. 7B, may perform the steps of FIG. 7B in a different order and / or in parallel, and / or may omit one or more of the steps of FIG. 7B. In various implementations, the steps of FIG. 7B are performed following steps 705, 710, 715, and 720 of FIG. 7A. However, it should be noted that in various alternative implementations, one or more steps of FIG. 7B may be performed without the performance of one or more (e.g., any) steps of FIG. 7A. For example, the steps of FIG. 7B may be performed in response to a user manually navigating to and selecting an option for adding a third-party device to an assistant account within a home application and / or other application. Also, for example, the steps of FIG. 7B may be performed automatically (ie, without any further user input) in response to performing steps 705 and 710 of FIG. 7A.

[0071] In step 740, the third-party application is authenticated. For example, the third-party application may be initially (e.g., in step 705) identified based on the package name of the third-party application, but may be authenticated using a pre-assigned signature of the third-party application. For example, the third-party application may be authenticated by obtaining a signing signature of the third-party application and comparing a hashed signature (e.g., hashed using SHA-256 or other cryptographic hash algorithm) with one provided by the third party as part of the on-boarding process. For example, the third-party system may have pre-assigned a valid hashed signature for the third-party application to a first-party system that operates the automated assistant and / or the operating system of the client device. The signing signature of the third-party application installed on the client device may be compared to such pre-assigned hash signature to verify that the installed third-party application is indeed the third-party application (and not a "spoofed" application).

[0072] At step 745, an intent to execute a third-party application with a linking function is provided. For example, an intent may be provided to an operating system of a client device (e.g., by an automation assistant application or a home control application) to cause the operating system client device to execute the third-party application by invoking the intent. The intent may be specific to the linking function of the third-party application, which, when executed by the third-party application, causes the third-party application to return at least one access token (e.g., a long-term and / or short-term access token) if one or more conditions are satisfied. The one or more conditions may include, for example, validation of the application providing the intent, as described in more detail below. The provided intent may optionally include a client identifier, such as an identifier of a user account associated with the automation assistant application or the home control application, and / or may optionally include an indication of the scope of access and / or token requested. In some implementations, if the operating system attempts to resolve a target for the intent in the third-party application but no resolution is found, it may be assumed that the third-party application does not support the linking function, and the method of FIG. 7B may end (and an error message is optionally presented).

[0073] At step 750, an access token is received from a third-party application. In some implementations, the third-party application interfaces with one or more remote systems of the third party in obtaining the access token. In some other implementations, the access token may be pre-provided to the third-party application by one or more remote systems of the third party for local storage on the access-constrained client device. In implementations where the access token is stored locally on the client device, latency in adding a third-party smart device to the topology may be reduced.

[0074] In some implementations, when a third-party application is executed with a linking function in response to a given intent, the third-party application first verifies the intent request before granting an access token. For example, the third-party application may verify the request by authenticating that the calling application (that granted the intent) is an automation assistant application or other authenticated application. For example, the third-party application may obtain the signature of the calling application and compare it to a pre-given signature that was given to the third-party system (e.g., during the on-boarding process) as the actual signature of the automation assistant application, home control application, or another verified application. In some implementations, the third-party application may give an error if it cannot return an access token. In some of those implementations, the error may indicate that it is a recoverable error and a new intent (e.g., with an alternative linking function and / or other parameters) given in response to the error indicating that it is recoverable. In some other implementations, the error may indicate that it is not a recoverable error and no further intents given in response to the error indicating that it is not a recoverable error.

[0075] In step 755, the access token is used to obtain additional third-party smart device data. For example, the access token may be used to interface with a remote system of a third-party application over a network to obtain additional information related to the particular smart device. At least a portion of the obtained additional information may be added to the smart device topology. The access token may also enable control of the 3P smart device by the automation assistant (e.g., by including the device in a request from the automation assistant and / or upon obtaining an updated short-term access token that may be included in the request).

[0076] 8 is a block diagram of an exemplary computing device 810 that may optionally be used to implement one or more aspects of the techniques described herein. The computing device 810 typically includes at least one processor 814 that communicates with a number of peripheral devices via a bus subsystem 812. These peripheral devices may include, for example, a storage subsystem 824 including a memory subsystem 825 and a file storage subsystem 826, a user interface output device 820, a user interface input device 822, and a network interface subsystem 816. The input and output devices enable user interaction with the computing device 810. The network interface subsystem 816 provides an interface to an outside network and is coupled to corresponding interface devices in other computing devices.

[0077] The user interface input devices 822 may include a keyboard, a pointing device such as a mouse, a trackball, a touchpad, or a graphics tablet, a scanner, a touch screen integrated into a display, a voice recognition system, an audio input device such as a microphone, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and methods for inputting information into the computing device 810 or onto a communications network.

[0078] The user interface output devices 820 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, use of the term "output device" is intended to include all possible types of devices and methods that output information from the computing device 810 to a user or to another machine or computing device.

[0079] Storage subsystem 824 stores the programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 824 may contain logic for performing selected aspects of the methods described herein, as well as for implementing the various components illustrated in FIG.

[0080] These software modules are generally executed by the processor 814 alone or in combination with other processors. The memory 825 used in the storage subsystem 824 may include several memories including a main random access memory (RAM) 830 for storing instructions and data during program execution, and a read only memory (ROM) 832 in which fixed instructions are stored. A file storage subsystem 826 may provide persistent storage for program and data files and may include hard disk drives, floppy disk drives, with associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing the functionality of some implementations may be stored in the storage subsystem 824 by the file storage subsystem 826, or in other machines accessible by the processor 814.

[0081] The bus subsystem 812 provides a mechanism for allowing various components and subsystems of the computing device 810 to communicate with each other as intended. Although the bus subsystem 812 is shown generally as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0082] The computing device 810 may be of various types, including a workstation, a server, a computing cluster, a blade server, a server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 810 shown in FIG. 8 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of the computing device 810 may have more or fewer components than the computing device shown in FIG.

[0083] In situations where some implementations discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, and user activity and demographic information, relationships between users, etc.), the user is given one or more opportunities to control whether information is collected, whether the personal information is stored, whether the personal information is used, and how information is collected, stored, and used about the user. That is, the systems and methods discussed herein collect, store, and / or use user personal information only upon receiving explicit authorization to do so from the associated user.

[0084] For example, a user is given control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. Each user for whom personal information is to be collected is presented with one or more options to give permission or authorization as to whether information is collected and what portions of information are to be collected to allow control over the collection of information appropriate to that user. For example, a user may be given one or more such control options over a communications network. Additionally, certain data may be treated in one or more ways such that personally identifiable information is removed before it is stored or used. As one example, a user's identity may be treated such that no personally identifiable information can be determined. As another example, a user's geographic location may be generalized to a larger area such that the user's specific location cannot be determined.

[0085] In some implementations, a method is provided that is implemented by one or more processors, the method including receiving a user interface input directed to an active application executing on a client device. The user interface input is provided via a user interface input device of the client device. The method further includes determining that the user interface input is a request to render, via the active application, at least one interactive graphical interface element that, when interacted with via a further user interface input, causes corresponding control of one or more specific smart devices of a specific third party (3P). The active application enables control of multiple smart devices of multiple third parties (3Ps), including one or more specific smart devices of the specific 3P. The method further includes, in response to determining that the user interface input is a request to render at least one interactive graphical interface element that, when interacted with via a further user interface input, causes corresponding control of one or more specific smart devices of the specific 3P, transmitting, via one or more networks, a request addressed to a specific 3P address for the specific 3P and including an identifier of the one or more specific smart devices. The method further includes, in response to transmitting the request, and receiving from the specific 3P, a deep link to a 3P application installed on the client device and controlled by the 3P. The received deep link is configured to cause the client device to open a 3P application in a state presenting additional information about the one or more particular smart devices based on a request that includes identifiers of the one or more particular smart devices.The method further includes, in response to the user interface input, causing the active application to simultaneously render at least one interactive graphical interface element that, when interacted with via a further user interface input, causes corresponding control of one or more particular smart devices of the particular 3P, and a selectable 3P element that, when selected, causes the client device to execute a deep link to open the 3P application in a state presenting additional information about the one or more particular smart devices.

[0086] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0087] In some implementations, the additional information about the one or more specific smart devices includes at least one additional graphical element that, when interacted with, causes control of one or more properties of the one or more specific smart devices. The control of the one or more properties of the one or more specific smart devices is not enabled via interaction with the at least one interactive graphical interface element rendered via the application. In some implementations, the identifier of the one or more specific smart devices is a unique identifier of the one or more specific smart devices, the unique identifier is used in the interaction between the application and the 3P, and the unique identifier is locally mapped by the 3P to a unique identifier of the one or more specific smart devices that is unique to the 3P.

[0088] In some implementations, the method further includes determining that a 3P application is installed on the client device prior to receiving user interface input directed to an active application executing on the client device. In some of those implementations, sending the request is conditioned on determining that the 3P application is installed on the client device. Determining that the 3P application is installed on the client device may include interfacing with an application programming interface (API) of an operating system of the client device to determine that the 3P application is installed on the client device.

[0089] In some implementations, one or more smart devices include a camera, and the additional information about the one or more particular smart devices includes at least a video clip recently captured by the camera.

[0090] In some implementations, the method further includes receiving, in response to sending the request and from the particular 3P, an indication of a selectable 3P element, and selecting a selectable 3P element for rendering from a plurality of candidate 3P elements for the 3P, wherein selecting the selectable 3P element is based on receiving the indication of the selectable 3P element in response to sending the request.

[0091] In some implementations, the method includes, following causing the active application to simultaneously render at least one interactive graphical interface element and a selectable 3P element, receiving an additional user interface input directed to the active application; determining that the additional user interface input is a request via the active application to render at least one interactive graphical interface element that, when interacted with via the further user interface input, causes corresponding control of one or more specific smart devices of the specific 3P; and, in response to determining that the user interface input is a request to render at least one interactive graphical interface element that, when interacted with via the further user interface input, causes corresponding control of one or more specific smart devices of the specific 3P, receiving, via one or more networks, a specific 3P address for the specific 3P. sending another instance of the request including identifiers of the plurality of specific smart devices; in response to sending the request and receiving from the particular 3P a different deep link to the 3P application installed on the client device, the received different deep link being different from the deep link and configured to cause the client device to open the 3P application in a different state presenting different information about the one or more specific smart devices based on the request including the identifiers of the one or more specific smart devices; and in response to the additional user interface input, providing to the active application at least one interactive graphical interface element which, when interacted with via the further user interface input, causes corresponding control of one or more specific smart devices of the particular 3P, and a selectable 3P element, or a selectable 3P element, when selected, for opening the 3P application in a different state presenting different information about the one or more specific smart devices.and causing additional selectable 3P elements to be simultaneously rendered that trigger execution of different deep links to the client device.

[0092] In some implementations, the method includes, following causing the active application to simultaneously render at least one interactive graphical interface element and a selectable 3P element, receiving an additional user interface input directed to the active application; determining that the additional user interface input is a request via the active application to render at least one interactive graphical interface element that, when interacted with via the further user interface input, causes corresponding control of one or more specific smart devices of the specific 3P; and, in response to determining that the user interface input is a request to render at least one interactive graphical interface element that, when interacted with via the further user interface input, causes corresponding control of one or more specific smart devices of the specific 3P, addressing the user interface input via one or more networks to a specific 3P address for the specific 3P. sending another instance of the request including an identifier of the one or more specific smart devices; and in response to sending the request and receiving from the particular 3P a different deep link to a 3P application installed on the client device, the received different deep link being different from the deep link and configured to cause the client device to open the 3P application in the same state as the deep link based on the request including the identifier of the one or more specific smart devices; and in response to a further user interface input, providing the active application with at least one interactive graphical interface element which, when interacted with via the further user interface input, causes corresponding control of one or more specific smart devices of the particular 3P, and a selectable 3P element, or elements, which, when selected, cause the client device to open the 3P application in the same state presenting the same information about the one or more specific smart devices.and causing additional selectable 3P elements to be simultaneously rendered that trigger execution of different deep links.

[0093] In some implementations, the method further includes receiving a further additional user interface input directed to the active application, determining, via the active application, that the further user interface input is a request to render at least one further interactive graphical interface element that, when interacted with via the further user interface input, causes corresponding control of one or more further smart devices of the further 3P separate from the particular 3P, determining that the client device lacks a further 3P application controlled by the further 3P, and in response to determining that the further user interface input is a request to render at least one further interactive graphical interface element and that the client device lacks a further 3P application, causing the active application to simultaneously render at least one further interactive graphical interface element that, when interacted with via the further user interface input, causes corresponding control of one or more further smart devices of the further 3P and a further selectable 3P element that, when selected, causes the client device to present options for installation of the further 3P application.

[0094] In some implementations, a method implemented by one or more processors is provided, comprising determining that a third party (3P) application is installed on a client device associated with a user account. The 3P application enables the 3P to control one or more 3P smart devices. The method further comprises determining that a smart device topology associated with the user account and used by an automation assistant to control the smart devices through interaction with the automation assistant lacks any entry for any of the one or more 3P smart devices of the 3P. In response to determining that the 3P application is installed on the client device and that the smart device topology lacks any entry for any of the one or more 3P smart devices, the method further comprises causing an output to be rendered on the client device or an additional client device associated with the user account, the output including a suggestion for adding the 3P to the smart device topology. The method further includes receiving a positive user interface input in response to rendering the output, and in response to receiving the positive user interface input, using the user account to interface with an application programming interface (API) of the 3P in automatically adding at least some of the specific devices of the one or more 3P smart devices to the smart device topology, interfacing with a third party application installed on the client device in automatically adding the specific smart device to the smart device topology, or presenting a sign-in interface that allows signing in to a 3P account specific to the 3P and that, when used to sign in to the 3P account, causes interfacing with the 3P API, or additional APIs, in automatically adding the specific device to the smart device topology.

[0095] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0096] In some implementations, the method includes using a user account to interface with the 3P's API in automatically adding at least some of the specific devices of the one or more 3P smart devices to the smart device topology, in some of those implementations, no addition prompt is presented to the user following receipt of a positive user interface input and before automatically adding the specific devices to the smart device topology is completed.

[0097] In some implementations, the method includes presenting a sign-in interface that is specific to 3P and enables signing in to the 3P account that, when used to sign in to the 3P account, triggers interfacing with 3P's API or additional APIs in automatically adding the particular device to the smart device topology. In some of those implementations, presenting the sign-in interface occurs in response to a failed attempt to use the user account to interface with 3P's API.

[0098] In some implementations, the output includes a graphical output rendered by a home control application.

[0099] In some implementations, the output includes graphical and / or audible output rendered by an automated assistant application.

[0100] In some implementations, in response to receiving the positive user interface input, the method includes interfacing with a third-party application installed on the client device in automatically adding the specific smart device to the smart device topology. In some versions of those implementations, interfacing with the third-party application in automatically adding the specific smart device to the smart device topology includes authenticating the third-party application using a pre-given signature for the third-party application, in response to authenticating the third-party application, granting an intent to cause the third-party application to run with the linking function, and in response to granting the intent, receiving an access token from the third-party application. In some of those versions, when the third-party application runs with the linking function, the third-party application authenticates that the intent was granted by the verified application using a pre-given signature of the verified application, and returns an access token in response to authenticating that the intent was granted by the verified application. In some implementations, interfacing with the third party application when automatically adding the particular smart device to the smart device topology includes interfacing with a remote system of the third party application over a network and using an access token to obtain additional information associated with the particular smart device, and adding at least a portion of the additional information to the smart device topology.

[0101] In some implementations, a method is provided that is implemented by one or more processors of a client device, and includes determining that a third-party (3P) application is installed on the client device. The 3P application enables control of one or more 3P smart devices of the 3P. The method further includes determining that a smart device topology associated with the client device and used by an automation assistant to control the smart devices through interaction with the automation assistant lacks any entry for any of the one or more 3P smart devices of the 3P. The method further includes authenticating the third-party application using a pre-given signature for the third-party application. The method further includes interfacing with the third-party application installed on the client device in automatically adding the specific smart device to the smart device topology in response to authenticating the third-party application and in response to determining that the 3P application is installed on the client device and that the smart device topology lacks any entry for any of the one or more 3P smart devices.

[0102] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0103] In some implementations, interfacing with the third-party application when automatically adding the specific smart device to the smart device topology includes providing an intent to cause the third-party application to run with the linking function, and receiving an access token from the third-party application in response to providing the intent. In some of those implementations, the method is performed by a specific application of the client device, and when the third-party application runs with the linking function, the third-party application authenticates the specific application based on the specific application providing the intent, and returns the access token in response to authenticating the specific application. In some implementations, interfacing with the third-party application when automatically adding the specific smart device to the smart device topology includes interfacing with a remote system of the third-party application over a network, using the access token to obtain additional information related to the specific smart device, and adding at least a portion of the additional information to the smart device topology. [Explanation of symbols]

[0104] 105 client devices 107 Graphical display, display 110 1-N Third Party (3P) Remote Server 11013P1 Remote Server 11023P2 Remote Server 110 N 3PN Remote Server 115 Home Control Applications 117 Rendering Engine 119 Request Engine 120 Remote Home Control Server 125 Third Party Applications 145A Smart Device, Smart Lighting Unit, Lighting Unit 145B Smart Devices 160 Device Configuration Engine 200 Interface 205 Toggle Button, Element 210 Slider element, element 215 Home Control Applications 300 Interface 305 Toggle Button 310 Slider 315 Slider, Color Control 320 Scene Button 400 Interface 405 Elements 410 Elements 415 Button 500 Interface 505 Graphical Elements 510 Graphical Elements 515 Graphical Elements 810 Computing Devices 812 Bus Subsystem 814 Processor 816 Network Interface Subsystem 820 User Interface Output Device 822 User Interface Input Devices 824 Memory Subsystem 825 Memory Subsystem, Memory 826 File Storage Subsystem 830 Main Random Access Memory (RAM) 832 Read-Only Memory (ROM)

Claims

1. 1. A method implemented by one or more processors, comprising: receiving user interface input directed to an active application executing on a client device, said user interface input being provided via a user interface input device of the client device; determining that the user interface input is a request for rendering, via the active application, at least one interactive graphical interface element that, when interacted with via a further user interface input, causes corresponding control of one or more specific smart devices of a specific third party (3P); the active application enables control of multiple smart devices of multiple third parties (3Ps), including the one or more specific smart devices of the specific 3P; in response to determining that the user interface input is the request for rendering the at least one interactive graphical interface element that, when interacted with via a further user interface input, causes a corresponding control of the one or more particular smart devices of the particular 3P; transmitting a request over one or more networks, said request comprising: addressed to a specific 3P address for said specific 3P; including an identifier for the one or more particular smart devices; receiving a deep link to a 3P application from the particular 3P in response to transmitting the request; the 3P application is installed on the client device and controlled by the 3P; the received deep link is configured to cause the client device to open the 3P application with additional information about the one or more specific smart devices based on the request including the identifier of the one or more specific smart devices; in response to the user interface input, causing the active application to simultaneously render the at least one interactive graphical interface element and a selectable 3P element; the at least one interactive graphical interface element, when interacted with via a further user interface input, causes a corresponding control of the one or more specific smart devices of the specific 3P; the selectable 3P element, when selected, causes execution of the deep link to cause the client device to open the 3P application in the state presenting additional information about the one or more particular smart devices.

2. 2. The method of claim 1, wherein the additional information about the one or more particular smart devices includes at least one additional graphical element that, when interacted with, causes control of one or more properties of the one or more particular smart devices, and the control of the one or more properties of the one or more particular smart devices is not enabled via interaction with the at least one interactive graphical interface element rendered via the application.

3. A method according to any one of claims 1 to 2, wherein the identifier of the one or more particular smart devices is a unique identifier of the one or more particular smart devices, the unique identifier being used in an interaction between the application and the 3P, and the unique identifier being locally mapped by the 3P to a unique identifier of the one or more particular smart devices that is specific to the 3P.

4. prior to receiving the user interface input directed to the active application executing on the client device; determining that the 3P application is installed on the client device; The method of claim 1 , wherein sending the request is conditioned on determining that the 3P application is installed on the client device.

5. The step of determining that a 3P application is installed on the client device includes: The method of claim 4 , comprising interfacing with an application programming interface (API) of an operating system of the client device to determine that the 3P application is installed on the client device.

6. The one or more smart devices include a camera, and the additional information about the one or more specific smart devices includes at least: The method of claim 1 , including a video clip recently captured by the camera.

7. receiving, in response to transmitting the request and from the particular 3P, an indication of the selectable 3P elements; 7. The method of claim 1, further comprising: selecting the selectable 3P element for rendering from a plurality of candidate 3P elements for the 3P, the selecting the selectable 3P element being based on receiving the indication of the selectable 3P element in response to sending the request.

8. causing the active application to simultaneously render the at least one interactive graphical interface element and the selectable 3P element; receiving additional user interface input directed to the active application; determining that the additional user interface input is a request to render, via the active application, the at least one interactive graphical interface element that, when interacted with via a further user interface input, causes a corresponding control of the one or more specific smart devices of the specific 3P; in response to determining that the user interface input is the request for rendering the at least one interactive graphical interface element that, when interacted with via a further user interface input, causes a corresponding control of the one or more particular smart devices of the particular 3P; transmitting another instance of the request over one or more networks, the instance comprising: addressed to the specific 3P address for the specific 3P; including the identifier of the one or more particular smart devices; In response to sending said request, and from said particular 3P, receiving a different deep link to the 3P application installed on the client device, the received different deep link being different from the deep link and configured to cause the client device to open the 3P application in a different state presenting different information about the one or more particular smart devices based on the request including the identifier of the one or more particular smart devices; In response to the additional user interface input, the active application the at least one interactive graphical interface element that, when interacted with via a further user interface input, causes a corresponding control of the one or more specific smart devices of the specific 3P; and causing simultaneously rendering the selectable 3P element or additional selectable 3P elements that, when selected, cause execution of the different deep links to the client device to open the 3P application in the different states presenting the different information for the one or more particular smart devices.

9. causing the active application to simultaneously render the at least one interactive graphical interface element and the selectable 3P element; receiving additional user interface input directed to the active application; determining that the additional user interface input is a request to render, via the active application, the at least one interactive graphical interface element that, when interacted with via a further user interface input, causes a corresponding control of the one or more specific smart devices of the specific 3P; in response to determining that the user interface input is the request for rendering the at least one interactive graphical interface element that, when interacted with via a further user interface input, causes a corresponding control of the one or more particular smart devices of the particular 3P; transmitting another instance of the request over one or more networks, the instance comprising: addressed to the specific 3P address for the specific 3P; including the identifier of the one or more particular smart devices; In response to sending said request, and from said particular 3P, receiving a different deep link to the 3P application installed on the client device, the different deep link being different from the deep link and configured to cause the client device to open the 3P application in a state identical to the deep link based on the request including the identifier of the one or more particular smart devices; In response to the additional user interface input, the active application the at least one interactive graphical interface element that, when interacted with via a further user interface input, causes a corresponding control of the one or more specific smart devices of the specific 3P; the selectable 3P element or an additional selectable 3P element that, when selected, causes the client device to execute the different deep link to open the 3P application in the same state presenting the same information for the one or more particular smart devices; Rendering steps simultaneously 9. The method of claim 1, further comprising:

10. receiving a further additional user interface input directed to the active application; determining, via the active application, that the further user interface input is a request to render the at least one further interactive graphical interface element, which when interacted with via the further user interface input causes a corresponding control of one or more further smart devices of a further 3P distinct from the particular 3P; determining that the client device lacks a further 3P application controlled by the further 3P; in response to determining that the further user interface input is the request for rendering the at least one further interactive graphical interface element and that the client device lacks the further 3P application, The active application: the at least one further interactive graphical interface element, when interacted with via a further user interface input, causing a corresponding control of the one or more further smart devices of the further 3P; and causing the client device to simultaneously render a further selectable 3P element that, when selected, causes the client device to present an option for installation of the further 3P application.

11. 1. A method implemented by one or more processors, comprising: determining that a third party (3P) application is installed on a client device associated with a user account, the 3P application enabling the 3P to control one or more 3P smart devices; determining that a smart device topology associated with the user account and used by an automation assistant to control smart devices through interactions with the automation assistant lacks any entry for any of the one or more 3P smart devices of the 3P; in response to determining that the 3P application is installed on the client device and that the smart device topology lacks any entry for any of the one or more 3P smart devices, rendering an output at the client device or an additional client device associated with the user account, the output including a proposal for adding the 3P to the smart device topology; receiving a positive user interface input in response to rendering the output; in response to receiving the positive user interface input; using the user account to interface with an application programming interface (API) of the 3P in automatically adding at least some of the specific devices of the one or more 3P smart devices to the smart device topology; interfacing with the third-party application installed on the client device in automatically adding the particular smart device to the smart device topology; or presenting a sign-in interface that allows the user to sign in to a 3P account specific to the 3P and that, when used to sign in to the 3P account, triggers an interface with the API or additional APIs of the 3P in automatically adding the specific device to the smart device topology; The method includes:

12. 12. The method of claim 11, wherein in response to receiving the positive user interface input, the method includes using the user account to interface with the API of the 3P in automatically adding at least some of the specific devices of the one or more 3P smart devices to the smart device topology.

13. 13. The method of claim 12, wherein following the receipt of the positive user interface input and before the automatic adding of the particular device to the smart device topology is completed, no adding prompt is presented to a user.

14. 12. The method of claim 11, wherein in response to receiving the positive user interface input, the method includes presenting a sign-in interface that is specific to the 3P and enables signing in to the 3P account, which when used to sign in to the 3P account, causes an interface with the API or additional APIs of the 3P in automatically adding the particular device to the smart device topology.

15. 15. The method of claim 14, wherein the step of presenting the sign-in interface occurs in response to a failed attempt to use the user account to interface with the application programming interface (API) of the 3P.

16. The method of claim 11 , wherein the output comprises a graphical output rendered by a home control application.

17. The method of claim 11 , wherein the output comprises an audible output rendered by an automated assistant application.

18. 12. The method of claim 11, wherein in response to receiving the positive user interface input, the method includes interfacing with the third party application installed on the client device in automatically adding the particular smart device to the smart device topology.

19. Interfacing with the third-party application in automatically adding the particular smart device to the smart device topology includes: authenticating the third party application using a pre-assigned signature for the third party application; in response to authenticating the third party application, providing an intent to cause the third party application to execute with a linking function; and receiving an access token from the third party application in response to granting the intent.

20. When the third party application is executed with the linking functionality, the third party application: authenticating that the intent was provided by a verified application using a pre-provided signature of the verified application; 20. The method of claim 19, further comprising returning the access token in response to authenticating that the intent was granted by the verified application.

21. Interfacing with the third-party application in automatically adding the particular smart device to the smart device topology includes: using the access token to interface with a remote system of the third party application over a network to obtain additional information associated with the particular smart device; and adding at least a portion of the additional information to the smart device topology.

22. 1. A method implemented by one or more processors of a client device, comprising: determining that a third party (3P) application is installed on the client device, the 3P application enabling the 3P to control one or more 3P smart devices; determining that a smart device topology associated with said client device and used by an automation assistant to control smart devices through interaction with said automation assistant lacks any entry for any of said one or more 3P smart devices of said 3P; authenticating the third party application using a pre-assigned signature for the third party application; in response to authenticating the third party application and in response to determining that the 3P application is installed on the client device and that the smart device topology lacks any entry for any of the one or more 3P smart devices, interfacing with the third-party application installed on the client device in automatically adding the particular smart device to the smart device topology; The method includes:

23. Interfacing with the third-party application in automatically adding the particular smart device to the smart device topology includes: providing an intent to execute the third party application with a linking function; and receiving an access token from the third-party application in response to granting the intent.

24. The method is implemented by a particular application on the client device, and when the third party application is executed with the linking functionality, the third party application: authenticating the particular application based on the particular application having granted the intent; 24. The method of claim 23, further comprising returning the access token in response to authenticating the particular application.

25. Interfacing with the third-party application in automatically adding the particular smart device to the smart device topology includes: using the access token to interface with a remote system of the third party application over a network to obtain additional information associated with the particular smart device; and adding at least a portion of the additional information to the smart device topology.

26. 26. A device comprising a memory storing instructions and one or more processors operable to execute the instructions to cause the processor to perform the method of any one of claims 1 to 25.

Images