Proximity-Based Control of a Second Device

Abstract

To provide a method and a user device for controlling an assistant-enabled device.SOLUTION: A method includes obtaining proximity information for each of a plurality of assistant-enabled devices within an environment of a user device. Each assistant-enabled device is controlled by an assistant application. The method also includes: determining a proximity score on the basis of the proximity information indicating a proximity estimation of each assistant-enabled device; generating, by using the proximity scores, a ranked list of candidate assistant-enabled devices, and displaying, in a graphical user interface, control for executing respective set of actions associated with corresponding assistant-enabled device.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to proximity-based control over a second device. [Background technology]

[0002] An individual may have numerous connected devices in their environment that can be controlled by various applications. Such connected devices may include, for example, smart light bulbs, smart TVs, smart speakers, smart thermostats, smart security systems, or any smart appliance (e.g., a smart oven) configured to perform respective sets of actions. In some instances, a user may be provided with a specific application to control a particular connected device. However, to control a connected device, the user must identify that application from the numerous applications running on the user device, select the application to control the specific connected device, and then navigate within the specific application to control the connected device. Managing numerous controls for each connected device from a single user device can become disorganized and cumbersome. Furthermore, many connected devices may perform respective sets of overlapping actions, making it difficult to discern which connected device a user's command is directed to. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a method for controlling assistant-enabled devices. The method includes acquiring, by data processing hardware of a user device, proximity information for each of a plurality of assistant-enabled devices in an environment of the user device. Each assistant-enabled device of the plurality of assistant-enabled devices is controllable by an assistant application to perform a respective set of available actions associated with the assistant-enabled device. For each assistant-enabled device of the plurality of assistant-enabled devices, the method also includes determining, by the data processing hardware, a proximity score based on the proximity information acquired for the corresponding assistant-enabled device. The proximity score indicates an estimation of the proximity of the corresponding assistant-enabled device relative to the user device in the environment. The method further includes generating, by the data processing hardware, a ranked list of candidate assistant-enabled devices from the plurality of assistant-enabled devices using the determined proximity scores for the plurality of assistant-enabled devices, and, for each of one or more corresponding assistant-enabled devices in the ranked list of candidate assistant-enabled devices, displaying, by the data processing hardware, a respective set of controls for performing the respective set of actions associated with the corresponding assistant-enabled device in a graphical user interface (GUI) displayed on a screen communicating with the data processing hardware.

[0004] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the method further includes receiving, at the data processing hardware, a user request from a user of a user device requesting to launch an assistant application to run on the data processing hardware. In these implementations, acquiring proximity information for each of the multiple assistant-enabled devices occurs during execution of the assistant application on the data processing hardware. In further implementations, receiving the user request includes one of receiving a user input indication indicating a selection of a graphical element representing the assistant application in a GUI displayed on the screen, receiving a voice input from a user including a call command to launch the assistant application to run on the data processing hardware, or detecting a predefined movement / pose of a user device configured to launch the assistant application to run on the data processing hardware.

[0005] In some examples, acquiring proximity information for at least one assistant-enabled device among a plurality of assistant-enabled devices in the environment of the user device includes receiving, at a sensor of the user device, a wireless communication signal transmitted from the at least one assistant-enabled device among the plurality of assistant-enabled devices, and determining the proximity information for the at least one assistant-enabled device based on a signal strength of the wireless communication signal received at the sensor of the user device. In some implementations, the method further includes acquiring, by data processing hardware, directionality information for at least one assistant-enabled device among a plurality of assistant-enabled devices in the environment of the user device by receiving, at each sensor in an array of sensors of the user device, a wireless communication signal transmitted from the at least one assistant-enabled device among the plurality of assistant-enabled devices, and determining the directionality information for the at least one assistant-enabled device based on a respective signal strength of the wireless communication signal received at each sensor in the array of sensors of the user device relative to a respective signal strength of the wireless communication signal received at other sensors in the array of sensors of the user device. Here, determining a proximity score for the at least one assistant-enabled device is further based on the directionality information for the at least one assistant-enabled device.

[0006] In some embodiments, receiving proximity information for at least one assistant-enabled device among a plurality of assistant-enabled devices in an environment of the user device includes receiving, at the user device, an audible or inaudible signal output from the at least one assistant-enabled device among the plurality of assistant-enabled devices and determining proximity information for the at least one assistant-enabled device based on the energy and / or frequency of the audible or inaudible signal output from the at least one assistant-enabled device. In some implementations, the method also includes receiving, at the data processing hardware, from each assistant-enabled device of the plurality of assistant-enabled devices, a respective set of available actions associated with the corresponding assistant-enabled device, and, for each corresponding assistant-enabled device in the ranked list of candidate assistant-enabled devices, determining, by the data processing hardware, a respective set of controls for executing the respective set of actions associated with the corresponding assistant-enabled device. In these implementations, at least one available action in the respective set of available actions received from the at least one assistant-enabled device of the plurality of assistant-enabled devices may include a suggested action for the corresponding assistant-enabled device to perform based on a current context. In a further embodiment, the method may include, in the data processing hardware, receiving, from at least one assistant-enabled device of the plurality of assistant-enabled devices, device state information associated with the corresponding assistant-enabled device. Wherein, determining a respective set of controls for performing a respective set of actions associated with the corresponding assistant-enabled device is further based on the device state information associated with the corresponding assistant-enabled device.

[0007] In some implementations, generating a ranked list of candidate assistant-enabled devices from the plurality of assistant-enabled devices includes ordering the assistant-enabled devices from an assistant-enabled device having a proximity that is relatively closest to the user device to an assistant-enabled device having a proximity that is relatively farthest from the user device. In some examples, generating a ranked list of candidate assistant-enabled devices from the plurality of assistant-enabled devices includes discarding any assistant-enabled device that includes a proximity score that indicates a proximity estimate that meets a maximum distance threshold from the ranked list of candidate assistant-enabled devices. Generating a ranked list of candidate assistant-enabled devices from the plurality of assistant-enabled devices may also or alternatively include discarding any assistant-enabled device that includes a proximity score that indicates a proximity estimate that meets a minimum distance threshold from the ranked list of candidate assistant-enabled devices.

[0008] In some embodiments, the method further includes, in the data processing hardware, receiving device state information associated with at least one assistant-enabled device from at least one assistant-enabled device among the plurality of assistant-enabled devices, and generating a ranked list of candidate assistant-enabled devices. After generating a ranked list of candidate assistant-enabled devices, the method further includes, using the device state information associated with the at least one assistant-enabled device by the data processing hardware, re-ranking the candidate assistant-enabled devices in the ranked list of candidate assistant-enabled devices. Wherein, displaying respective sets of controls for performing respective sets of actions associated with the corresponding assistant-enabled devices is based on re-ranking the candidate assistant-enabled devices in the ranked list of candidate assistant-enabled devices.

[0009] The ranked list of candidate assistant-enabled devices from the plurality of assistant-enabled devices may include assistant-enabled devices selected from the plurality of assistant-enabled devices having N highest proximity scores among the plurality of proximity scores determined for the plurality of assistant-enabled devices. In some examples, displaying the respective sets of controls for performing the respective sets of actions associated with the corresponding assistant-enabled devices includes displaying the respective sets of controls for one of the assistant-enabled devices in the ranked list of candidate assistant-enabled devices in a GUI differently from the respective sets of controls displayed in the GUI for at least another one of the assistant-enabled devices in the ranked list of candidate assistant-enabled devices.

[0010] Another aspect of the present disclosure provides a user device for controlling assistant-enabled devices located within the user device's environment. The user device includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including acquiring proximity information for each of a plurality of assistant-enabled devices within the user device's environment. Each of the plurality of assistant-enabled devices is controllable by an assistant application to perform a respective set of available actions associated with the assistant-enabled device. The operations also include, for each of the plurality of assistant-enabled devices, determining a proximity score based on the proximity information acquired for the corresponding assistant-enabled device. Here, the proximity score indicates an estimated proximity of the corresponding assistant-enabled device relative to the user device in the environment. The operations further include generating a ranked list of candidate Assistant-enabled devices from the plurality of Assistant-enabled devices using the plurality of proximity scores determined for the plurality of Assistant-enabled devices, and, for each of one or more corresponding Assistant-enabled devices in the ranked list of candidate Assistant-enabled devices, displaying, in a graphical user interface (GUI) displayed on a screen in communication with the data processing hardware, a respective set of controls for performing a respective set of actions associated with the corresponding Assistant-enabled device.

[0011] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, the operations further include receiving a user request from a user of a user device requesting to launch an assistant application. In these implementations, acquiring proximity information for each of the multiple assistant-enabled devices occurs while the assistant application is running. In further implementations, receiving the user request includes one of receiving a user input indication indicating a selection of a graphical element representing the assistant application in a GUI displayed on the screen, receiving a voice input from a user including a call command to launch the assistant application to run on data processing hardware, or detecting a predefined movement / pose of a user device configured to launch the assistant application to run on data processing hardware.

[0012] In some examples, acquiring proximity information for at least one assistant-enabled device among a plurality of assistant-enabled devices in the environment of the user device includes receiving, at a sensor of the user device, a wireless communication signal transmitted from the at least one assistant-enabled device among the plurality of assistant-enabled devices, and determining the proximity information for the at least one assistant-enabled device based on a signal strength of the wireless communication signal received at the sensor of the user device. In some implementations, the operations further include acquiring directionality information for at least one assistant-enabled device among a plurality of assistant-enabled devices in the environment of the user device by receiving, at each sensor in an array of sensors of the user device, a wireless communication signal transmitted from the at least one assistant-enabled device among the plurality of assistant-enabled devices, and determining the directionality information for the at least one assistant-enabled device based on a respective signal strength of the wireless communication signal received at each sensor in the array of sensors of the user device relative to a respective signal strength of the wireless communication signal received at other sensors in the array of sensors of the user device. Here, determining a proximity score for the at least one assistant-enabled device is further based on the directionality information for the at least one assistant-enabled device.

[0013] In some examples, receiving proximity information for at least one assistant-enabled device of a plurality of assistant-enabled devices in the environment of the user device includes receiving an audible or inaudible signal output from the at least one assistant-enabled device of the plurality of assistant-enabled devices and determining proximity information for the at least one assistant-enabled device based on the energy and / or frequency of the audible or inaudible signal output from the at least one assistant-enabled device. In some implementations, the operations also include receiving, from each assistant-enabled device of the plurality of assistant-enabled devices, a respective set of available actions associated with the corresponding assistant-enabled device, and determining, for each corresponding assistant-enabled device in the ranked list of candidate assistant-enabled devices, a respective set of controls for executing the respective set of actions associated with the corresponding assistant-enabled device. In these implementations, at least one available action in the respective set of available actions received from at least one assistant-enabled device of the plurality of assistant-enabled devices may include a suggested action for the corresponding assistant-enabled device to perform based on a current context. In further examples, the operations may include receiving, from at least one assistant-enabled device of the plurality of assistant-enabled devices, device state information associated with the corresponding assistant-enabled device. Wherein, determining the respective set of controls for performing the respective set of actions associated with the corresponding assistant-enabled device is further based on device state information associated with the corresponding assistant-enabled device.

[0014] In some implementations, generating a ranked list of candidate assistant-enabled devices from the plurality of assistant-enabled devices includes ordering the assistant-enabled devices from an assistant-enabled device having a proximity that is relatively closest to the user device to an assistant-enabled device having a proximity that is relatively farthest from the user device. In some examples, generating a ranked list of candidate assistant-enabled devices from the plurality of assistant-enabled devices includes discarding any assistant-enabled device that includes a proximity score that indicates a proximity estimate that meets a maximum distance threshold from the ranked list of candidate assistant-enabled devices. Generating a ranked list of candidate assistant-enabled devices from the plurality of assistant-enabled devices may also or alternatively include discarding any assistant-enabled device that includes a proximity score that indicates a proximity estimate that meets a minimum distance threshold from the ranked list of candidate assistant-enabled devices.

[0015] In some embodiments, the operations further include receiving, from at least one assistant-enabled device among the plurality of assistant-enabled devices, device state information associated with the at least one assistant-enabled device; and, after generating a ranked list of candidate assistant-enabled devices, using the device state information associated with the at least one assistant-enabled device to re-rank the candidate assistant-enabled devices in the ranked list of candidate assistant-enabled devices. Wherein, displaying respective sets of controls for performing respective sets of actions associated with the corresponding assistant-enabled devices is based on re-ranking the candidate assistant-enabled devices in the ranked list of candidate assistant-enabled devices.

[0016] The ranked list of candidate assistant-enabled devices from the plurality of assistant-enabled devices may include assistant-enabled devices selected from the plurality of assistant-enabled devices having N highest proximity scores among the plurality of proximity scores determined for the plurality of assistant-enabled devices. In some examples, displaying the respective sets of controls for performing the respective sets of actions associated with the corresponding assistant-enabled devices includes displaying the respective sets of controls for one of the assistant-enabled devices in the ranked list of candidate assistant-enabled devices in a GUI differently from the respective sets of controls displayed in the GUI for at least another one of the assistant-enabled devices in the ranked list of candidate assistant-enabled devices.

[0017] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description that follows. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a schematic diagram illustrating an example system that includes controlling assistant-enabled devices located within an environment of a user device. [Figure 2] FIG. 1 is a schematic diagram illustrating an example user device communicating with an assistant-enabled device. [Figure 3A] FIG. 300a is a schematic diagram illustrating a user device controlling an assistant-enabled device within the user device's environment. [Figure 3B] FIG. 300b is a schematic diagram illustrating a user device controlling an assistant-enabled device within the user device's environment. [Figure 3C] 300c is a schematic diagram illustrating a user device controlling an assistant-enabled device within the user device's environment. [Figure 4A]FIG. 4 illustrates an exemplary GUI 400a rendered on a screen of a user device to display respective sets of controls for performing respective sets of actions associated with respective corresponding assistant-enabled devices. [Figure 4B] FIG. 4 illustrates an example GUI 400b rendered on a screen of a user device to display respective sets of controls for performing respective sets of actions associated with respective corresponding assistant-enabled devices. [Figure 4C] FIG. 4 illustrates an exemplary GUI 400c rendered on a screen of a user device to display respective sets of controls for performing respective sets of actions associated with respective corresponding assistant-enabled devices. [Figure 5] 10 is a flowchart illustrating an example arrangement of operations for a method for controlling an assistant-enabled device. [Figure 6] FIG. 1 is a schematic diagram illustrating an example computing device that may be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0019] Like reference symbols in the various drawings indicate like elements.

[0020] The present disclosure generally relates to a user device that executes an assistant application for controlling one or more assistant-enabled devices through a portable client device interface. Specifically, the user device can automatically detect which assistant-enabled device, among multiple assistant-enabled devices in the user device's environment, a user of the user device is likely to want to control from the user device. The assistant-enabled device can include any Internet of Things (IoT) device, such as a smart light, a smart TV, a smart speaker, a smart thermostat, a smart security system, or any smart appliance (e.g., a smart oven) configured to perform a respective set of actions. That is, a user device (e.g., a smartphone, tablet, smartwatch, smart display, etc.) that executes the assistant application can provide a respective set of controls for controlling the assistant-enabled device to perform one or more of the respective set of actions. For example, a user device may display in a graphical user interface a respective set of controls for any available actions that the assistant-enabled device can perform at a given time, such that the graphical user interface can receive a user input instruction (e.g., touch input and / or voice input) indicating selection of one of the controls to cause the assistant-enabled device to perform the respective action. As the number of assistant-enabled devices with which a user interacts in their environment (e.g., home or office) increases, managing the numerous controls for controlling each of those devices from a single user device can become chaotic and cumbersome. Furthermore, assistant-enabled devices may share overlapping capabilities and may share similar device names, creating a need to disambiguate the distinction between the respective controls among those devices.

[0021] Implementations herein are directed to automatically detecting, among multiple assistant-enabled devices, which assistant-enabled device a user is likely to want to control through an interface displayed on a screen of a user device associated with the user. This detection is based on the proximity of each assistant-enabled device relative to the user device. For example, each assistant-enabled device in the user device's environment may broadcast proximity information receivable by the user device, which the user device may use to determine the proximity of each assistant-enabled device relative to the user device. The proximity information may include wireless communication signals, such as WiFi, Bluetooth, or Ultrasonic, and the signal strength of the wireless communication signal received at the user device may correlate with the proximity (e.g., distance) of the assistant-enabled device relative to the user device. The proximity information may also include a user input indication indicating that the user has interacted with one of the assistant-enabled devices, for example, by touching the device to adjust the volume. By utilizing proximity-based detection, a user can reach their intended control in fewer steps and in less time. The user device can acquire proximity information for each of the assistant-enabled devices in the user's environment that may be controlled by the user device. That is, based on the proximity information acquired from each of these assistant-enabled devices, the user device can determine a respective proximity score for each assistant-enabled device that indicates a proximity estimate of each assistant-enabled device relative to the user device. Furthermore, the user device can collect direction information from one or more of the assistant-enabled devices that indicates the direction / orientation of the user device relative to each of the one or more assistant-enabled devices.That is, the directionality information may indicate whether the user device is facing or pointing toward a given assistant-enabled device to serve as an indicator that the user intends to use the user device to control the assistant-enabled device to perform the respective set of available actions. In these scenarios, the directionality information may be used in combination to determine the respective proximity scores, or the directionality information may be used to bias the proximity scores determined from the proximity information alone. After obtaining the proximity information and / or directionality information, the user device can determine a proximity score for each of the assistant-enabled devices in the environment. After determining the proximity scores for each of the assistant-enabled devices in the environment, the user device (e.g., via execution of an assistant application) ranks the assistant-enabled devices based on the proximity scores to prioritize which assistant-enabled devices the user is more likely to intend to control at a given time. The user device then displays the respective sets of controls for one or more of the assistant-enabled devices in a graphical user interface (GUI) displayed on the screen of the user device for use in controlling one or more of the assistant-enabled devices to perform the respective currently available actions. For example, the controls for the assistant-enabled device with the highest-rated proximity score may be displayed at the top of the screen in a larger font than other controls associated with other assistant-enabled devices in the user's environment with lower-rated proximity scores. Furthermore, assistant-enabled devices with very low-rated proximity scores may be omitted from displaying in the GUI entirely, since the user is unlikely to intend to control such devices at a given time.

[0022] 1 , in some implementations, a proximity-based controller system 100 includes a user device 200 that communicates with multiple assistant-enabled devices 210, 210a-n within an environment 32 of the user device 200 via a network 10. The environment 32 may include a premises of a user 30 associated with the user device 200. In some examples, the environment 32 includes a portion of the premises of the user 30, such as a floor or other section of the premises. The network 10 may include a local area network (LAN) (e.g., a home area network HAN) that facilitates communication and interoperability between the user devices 200 within the environment 32, such as the user's home, school, or office. In the illustrated example, assistant-enabled devices 210 are located throughout a home 32 of a user 30 having a first floor and a second floor, with a smart speaker 210a, a smart light 210b, a smart TV 210c, and a smart thermostat 210d located on the first floor and a second smart TV 210e and a second smart speaker 210f located on the second floor, e.g., in a bedroom of the home 32 of the user 30. The user devices 200 may communicate with each of the assistant-enabled devices 210 via a wireless connection using standard communication technologies and / or protocols. To this end, the network 10 may include Wireless Fidelity (WiFi) (e.g., 802.11), Worldwide Interoperability for Microwave Access (WiMAX), 3G, 4G, Long Term Evolution (LTE), 5G, Digital Subscriber Line (DSL), Bluetooth, Near Field Communications (NFC), or any other wireless standard. The premises 32 may include one or more access points (APs) (not shown) configured to facilitate wireless communication between the user device 200 and one or more of the assistant-enabled devices 210, 210a-f.

[0023] User device 200 can be any computing device capable of wireless communication with assistant-enabled device 210. While user device 200 includes a tablet in the illustrated example, user device 200 may include, without limitation, a smartphone, a smartwatch, a laptop, a desktop, or a smart display. User device 200 may use a variety of different operating systems 104. In examples in which user device 200 is a mobile device, user device 200 may run an operating system including, but not limited to, ANDROID® developed by Google, Inc., IOS® developed by Apple, Inc., or WINDOWS PHONE® developed by Microsoft Corporation. Thus, operating system 104 running on user device 200 may include, but is not limited to, one of ANDROID®, IOS®, or WINDOWS PHONE®. In some embodiments, user device 200 may run an operating system, including, but not limited to, MICROSOFT WINDOWS® by Microsoft Corporation, MAC OS® by Apple Inc., or Linux®. User device 200 may communicate with assistant-enabled devices 210, 210a-f while running an operating system 104 other than the operating systems 104 described above, whether currently available or developed in the future. Operating system 104 may run one or more software applications 106.

[0024] A software application 106 may refer to computer software that, when executed by a computing device, causes the computing device to perform a task. In some embodiments, a software application 106 is referred to as an "application," "app," or "program." Exemplary software applications 106 include, but are not limited to, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and games. An application 106 may be executed on a variety of different user devices 200. In some embodiments, an application 106 is installed on a user device 200 prior to a user 30 purchasing the user device 200. In other embodiments, a user 30 downloads and installs the application 106 on the user device 200.

[0025] In some implementations, user device 200 executes assistant application 106, which initiates communication with each assistant-enabled device 210 and provides a respective set of controls 220 for executing a respective set of actions 120 associated with each assistant-enabled device 210. That is, each assistant-enabled device 210 of multiple assistant-enabled devices 210 is controllable by assistant application 106 to execute a respective set of available actions 120 associated with the assistant-enabled device 210. Advantageously, assistant application 106 allows user 30 to control and / or configure each assistant-enabled device 210 using an interface 400, such as a graphical user interface (GUI) 400, which application 106 may render for display on the screen of user device 200. In this manner, user 30 can execute assistant application 106 to indicate to user device 200 which assistant-enabled device 210 user 30 wishes to control, rather than necessarily having to rely on various individual applications to control various devices in the user's home. Having a single application 106 that provides a respective set of controls 220 for performing a respective set of available actions 120 associated with each of multiple assistant-enabled devices 210 frees up memory and processing bandwidth in user device 200 as a result of user 30 not necessarily being required to install, open, or switch between all applications provided by manufacturers of various assistant-enabled devices 210 to control the devices 210 to perform the respective sets of actions 120 of the assistant-enabled devices 210.The GUI 400 used in this specification may receive user input instructions via any one or more of touch, voice, gesture, gaze, and / or input device (e.g., a mouse or stylus) to launch the application 106 and to control the functionality of the application 106 running on the user device 200.

[0026] The assistant application 106 executing on the user device 200 may render a ranked list 310 of candidate assistant-enabled devices 210 that the user 30 may control via the user device 200 for display on the GUI 400. The ranked list 310 rendered on the GUI 400 may include, for each candidate assistant-enabled device 210, a respective graphic 402 identifying the candidate assistant-enabled device 210 and a respective set of controls 220 for performing a respective set of actions 120 associated with the corresponding assistant-enabled device 210. For example, the smart speakers 210a, 210f may each include the same respective set of actions 120 associated with each of the smart speakers 210a, 210f, including a play action for audibly playing media content (e.g., a song), a stop / pause action for stopping / pausing the audible output of the media content, and a volume action for increasing / decreasing the volume of audio content output from the corresponding speakers 210a, 210f. In this example, the assistant application 106 may display, for each smart speaker 210a, 210f displayed in the list 310, the same respective set of controls 220 for performing the respective set of actions 120 associated with the corresponding speaker 210a, 210f. That is, each set of controls 220 may include a control 220 for performing an action 120 associated with a play operation, a control 220 for performing an action 120 associated with a stop / pause operation, a control 220 for performing an action 120 associated with increasing a volume level, and a control 220 for performing an action 120 associated with decreasing a volume level.

[0027] In some examples, the ranked list 310 of candidate assistant-enabled devices 210 rendered for display in GUI 400 further includes a graphic showing respective device state information 215 for at least one of the assistant-enabled devices 210 in the ranked list 310. In the illustrated example, device state information 215 for smart light 210b is displayed to indicate that smart light 210b is currently off and therefore not providing illumination. Similarly, device state information 215 for smart thermostat 210d is displayed to indicate the current temperature setting of smart thermostat 210d (e.g., the temperature is currently 21 degrees Celsius).

[0028] The respective sets of available actions 120 associated with each assistant-enabled device 210 may vary depending on the device state information 215 of the assistant-enabled device 210. For example, if the smart light 210b's device state information 215 indicates that it is currently off and therefore not providing illumination, the only action 120 available for the smart light 210b to perform is to turn it on, thereby causing the assistant application 106 to provide only the corresponding control 220 for causing the smart light 210b to perform an operation to switch from the off state to the on state in which the smart light 210b provides illumination. On the other hand, if the smart light 210b's device state information 215 indicates that it is currently on and therefore providing illumination, the smart light 210b may perform at least the turn it off action 120 and, optionally, an increase / decrease illumination setting operation, which may increase or decrease the illumination provided by the smart light 210b. In this scenario, assistant application 106 may provide a set of controls 220 that each enable user 30 to control smart lighting 210b to perform one of the following actions: switching from an on state to an off state, increasing a light setting, or decreasing a light setting. Thus, at least one available action 120 in each set of available actions 120 received from at least one assistant-enabled device 210 may include a suggested action for the corresponding assistant-enabled device 210 to perform based on a current context. Here, the current context may include device state information 215 of the corresponding assistant-enabled device 210.The current context may additionally or alternatively include past behavior of user 30 indicating which assistant-enabled devices 210 user 30 actually controlled under the same or similar context, including, without limitation, the same or similar proximity information 250, directional information 255, and / or device state information 215 associated with one or more of assistant-enabled devices 210 in environment 32 of user 30.

[0029] The assistant application 106 may further provide settings that enable a user 30 of the user device 200 to customize the active controls 220 for performing one or more specific actions 120 associated with each assistant-enabled device 210. The user 30 may provide a user request 108 requesting that the assistant application 106 be launched for execution on the user device 200. For example, a graphical user interface (GUI) 400 displayed on a screen of the user device 200 may receive the user request 108 as a user input instruction indicating a selection of a graphical element representing the assistant application 106, or the user 30 may provide the user request 108 via voice input including an invocation command that launches the assistant application 106 for execution on the user device 200. In other examples, the user device 200 receives the user request 108 in response to detecting a predefined movement / pose of the user device 200 configured to launch the assistant application 106 for execution on the user device 200. For example, the user device 200 may include an accelerometer that detects the user 30 moving the user device from a first position to a second position.

[0030] Presenting each set of controls 220 in GUI 400 can become unwieldy as the number of assistant-enabled devices 210 in environment 32 increases. For example, simply displaying each set of controls 220 for performing each set of actions 120 associated with each of the assistant-enabled devices 210 requires the user 30 to navigate GUI 400 to find the target assistant-enabled device 210 that the user 30 wants to control. For obvious reasons, this is a tedious and cumbersome task for the user 30 when there are many assistant-enabled devices 210 available to control. By using proximity information 250 for each of the multiple assistant-enabled devices 210 in the user's environment 32, implementations herein are directed to the assistant application 106 automatically detecting one or more candidate assistant-enabled devices 210 among the multiple assistant-enabled devices 210 in environment 32 as the target assistant-enabled device 210 that the user 30 wants to control by commanding / instructing the target assistant-enabled device 210 to perform the respective actions 120. For example, the user 30 may provide a user input instruction indicating a selection of one of the controls 220 displayed in the GUI 400 associated with performing the respective action 120 associated with the target assistant-enabled device 210, thereby causing the assistant application 106 to send a command 80 corresponding to the target assistant-enabled device 210 to perform the respective action 120.

[0031] In the illustrated example, during execution of assistant application 106, user device 200 receives proximity information 250, 250a-f for each of a plurality of assistant-enabled devices 210, 210a-f in environment 32. For each assistant-enabled device 210, user device 200 determines a proximity score 260, 260a-f (i.e., using assistant application 106) based on the proximity information 250 obtained for the corresponding assistant-enabled device 210. Here, each proximity score 260 indicates a proximity estimate of the corresponding assistant-enabled device 210 relative to user device 200 in environment 32. As used herein, a proximity estimate may include a distance from user device 200 at which the corresponding device 210 is located. The proximity estimate may further include a location of the corresponding device 210 relative to the location of user device 200. Using the determined proximity scores 260 for the assistant-enabled devices 210, the user device 200 generates a ranked list 310 of candidate assistant-enabled devices 210 from the assistant-enabled devices 210 in the environment 32 (i.e., using the assistant application 106). Here, the ranked list 310 is displayed in a GUI 400, and for each assistant-enabled device 210 in the ranked list 310, the user device 200 displays in the GUI 400 a respective set of controls 220 for performing a respective set of actions 120 associated with the corresponding assistant-enabled device 210.

[0032] In some implementations, the assistant application 106 generates a ranked list 310 of candidate assistant-enabled devices 210 by ordering the assistant-enabled devices 210 from the assistant-enabled device 210 having the closest relative proximity to the user device 200 to the assistant-enabled device 210 having the furthest relative proximity to the user device 200. For example, FIG. 1 shows a ranked list 310 of candidate assistant-enabled devices 210 displayed in a GUI 400 including, in order from top to bottom, a smart speaker 210a, a smart light 210b, a smart TV 210c, a smart thermostat 210d, a second smart TV 210e, and a second smart speaker 210f. Here, the smart speaker 210a displayed at the top of the ranked list 310 in the GUI 400 includes the closest proximity to the user device 200, while the second smart speaker 210f displayed at the bottom of the ranked list 310 in the GUI 400 includes the furthest proximity to the user device 200.

[0033] In some embodiments, the assistant application 106 generates a ranked list 310 of candidate assistant-enabled devices 210 by selecting assistant-enabled devices 210 having the highest N proximity scores 260 from among the proximity scores 260 determined for the assistant-enabled devices 210 in the environment 32 of the user device 200. In a further embodiment, the ranked list 310 of candidate assistant-enabled devices 210 includes only assistant-enabled devices 210 having proximity scores 260 that satisfy a maximum distance threshold. Here, the maximum distance threshold may be configurable and may be associated with a large distance between the assistant-enabled device 210 and the user device 200 that indicates a low likelihood that the user 30 intends to control the assistant-enabled device 210. In this manner, assistant-enabled devices 210 that are farther away from the user device 200 than the maximum distance threshold can be effectively filtered out from the ranked list 310 of candidate assistant-enabled devices 210. For example, when user device 200 is on a first floor, one or more assistant-enabled devices 210 located on a second floor may be excluded from ranked list 310 of candidate assistant-enabled devices 210. Additionally or alternatively, ranked list 310 of candidate assistant-enabled devices 210 may exclude assistant-enabled devices 210 having proximity scores 260 indicating proximity estimates that meet a minimum distance threshold. Here, the minimum distance threshold may be configurable and may be associated with a short distance between assistant-enabled device 210 and user device 200 that indicates the user 30 is close enough to directly manually control the assistant-enabled device 210 without using application 106. In this manner, assistant-enabled devices 210 within reach of user 30 can be effectively filtered out from ranked list 310 of candidate assistant-enabled devices 210.

[0034] FIG. 2 shows an example user device 200 in communication with an assistant-enabled device (e.g., a smart speaker) 210. The user device 200 includes data processing hardware 202 and memory hardware 204 that communicates with the data processing hardware 202 and stores instructions that, when executed on the data processing hardware 202, cause the data processing hardware 202 to execute an assistant application 106. The user device 200 may also include at least one sensor 206, such as an antenna, configured to receive wireless communication signals transmitted by the assistant-enabled device 210. The user device 200 may also include an array of one or more microphones 208. The assistant-enabled device 210 may include one of multiple assistant-enabled devices 210 in the environment 32 of the user device 200 that are controllable by the assistant application 106 to perform a respective set of available actions 120 associated with the assistant-enabled device 210. In some implementations, the assistant-enabled device 210 discloses a respective set of available actions 120 that the assistant-enabled device 210 can perform. In the illustrated embodiment, each set of available actions 120 includes media player actions 120, including a play action, a volume control action, a stop / pause action, and a next / previous track action. User device 200 may receive set of available actions 120 directly from assistant-enabled device 210 or indirectly through an access point (not shown). User device 200 may then determine each set of controls 220 to display in GUI 400 based on each set of available actions 120 received from assistant-enabled device 210.The user 30 may provide a user input instruction indicating the selection of one of the controls 220 displayed in the GUI 400 associated with performing the respective action 120 associated with the target assistant-enabled device 210 (e.g., a speaker) that the user 30 actually desires to control, thereby causing the assistant application 106 to send a corresponding command 80 to the target assistant-enabled device 210 to perform the respective action 120.

[0035] User device 200 may also receive device state information 215 from assistant-enabled device 210. For example, device state information 215 may indicate that assistant-enabled device 210 is currently on and playing a song playlist. User device 200 also receives proximity information 250 from assistant-enabled device 210. User device 200 may receive proximity information 250, and optionally, device state information 215, from each assistant-enabled device 210 in environment 32 of user device 200 continuously or at least during periodic intervals. As used herein, proximity information 250 includes any information or data that user device 200 may use to determine a proximity score 260 for a corresponding assistant-enabled device 210, where proximity score 260 indicates an estimate of the proximity of the corresponding assistant-enabled device 210 relative to user device 200 in environment 32. Thus, although many embodiments include user device 200 determining proximity score 260 for corresponding assistant-enabled device 210 based on proximity information 250 received from assistant-enabled device 210, user device 200 may receive proximity information 250 associated with corresponding assistant-enabled device 210 from another device. For example, another device including an image capture device may provide image data indicative of proximity information 250 associated with assistant-enabled device 210 in relation to user device 200.

[0036] In some embodiments, user device 200 receives wireless communication signals transmitted by assistant-enabled device 210 at sensor 206 of user device 200 and determines proximity information 250 based on the signal strength of the wireless communication signals received at sensor 206 of user device 200. Here, the wireless communication signals may include, without limitation, Bluetooth signals, infrared signals, NFC signals, or ultrasonic signals. In other embodiments, user device 200 receives proximity information 250 from an access point (not shown) indicating the signal strength of wireless communication signals received at the access point from assistant-enabled device 210. In these embodiments, user device 200 may determine proximity score 260 indicating a proximity estimate based on the signal strength of wireless communication signals received at the access point from user device 200. In a further embodiment, user device 200 receives audible or inaudible signals output from assistant-enabled devices 210 at an array of one or more microphones 208, and determines proximity information 250 about at least one assistant-enabled device 210 based on the energy and / or frequency of the audible or inaudible signals output from assistant-enabled devices 210.

[0037] In some implementations, the user device 200 further acquires directionality information 255 about the assistant-enabled device 210 by receiving wireless communication signals transmitted by the assistant-enabled device 210 at each microphone (e.g., sensor) in the array of microphones 208 and determining directionality information 255 based on the respective signal strengths of the wireless communication signals received at each microphone (e.g., sensor) in the array of sensors 208 relative to the respective signal strengths of the wireless communication signals received at the other microphones in the array of microphones 208. In these implementations, the proximity score 260 determined about the assistant-enabled device 210 is further based on the directionality information 255. For example, the directionality information 255 may indicate that the user device 200 is not facing the assistant-enabled device 210 or is facing in a different direction from the assistant-enabled device 210. This can serve as a strong indicator that the user 30 does not intend to control the assistant-enabled device 210. Thus, the directionality information 255 may bias the proximity score 260 by increasing the proximity score 260 when the directionality information 255 indicates that the user device 200 is facing the direction of the assistant-enabled device 210, or by decreasing the proximity score 260 when the directionality information 255 indicates that the user device 200 is facing away from the assistant-enabled device 210. As used herein, the proximity score 260 increases as the proximity relative to the user device 200 decreases and decreases as the proximity relative to the user device 200 increases. Thus, the magnitude of the proximity score 260 is inversely proportional to the proximity of the corresponding assistant-enabled device 210 relative to the user device 200.

[0038] In some implementations, the assistant application 106 trains the user-specific control model on historical data including, without limitation, past proximity-directional information 250, 255, resulting proximity score 260, available set of actions 120, device state information 215, and the target assistant-enabled device 210 associated with the control 220 that the user 30 actually controlled to perform each action 120 associated with the target assistant-enabled device 210. For example, the user 30 may provide a user input instruction indicating the selection of one of the controls 220 displayed in the GUI 400 associated with performing each action 120 associated with the target assistant-enabled device 210, thereby causing the assistant application 106 to send a command 80 corresponding to the target assistant-enabled device 210 to perform each action 120.

[0039] 3A-3C show schematic diagrams 300a-c of user device 200 executing assistant application 106 for controlling multiple assistant-enabled devices 210 within user device 200's environment (e.g., room 32) while user 30 carries user device 200 and moves around the room. The room may include a living room on a first floor of building 32 of FIG. 1 , with assistant-enabled devices 210, 210a-d, arranged throughout the living room. Specifically, schematic diagrams 300a-c show multiple assistant-enabled devices 210, including a smart speaker 210a, a smart light 210b, a smart TV 210c, and a smart thermostat 210d. FIGS. 4A-4C show exemplary GUIs 400a-c rendered on the screen of user device 200 to display respective sets of controls 220 for performing respective sets of actions 120 associated with each corresponding assistant-enabled device 210. Specifically, each GUI 400a-c shows a ranked list 310 of candidate assistant-enabled devices 210 and a respective set of controls 220 for each device 210 based on proximity information 250 obtained for each of the plurality of assistant-enabled devices 210 of Figures 3A-3C. Each candidate assistant-enabled device 210 in the ranked list 310 may be rendered in the GUI 400 as a respective graphic 402 representing the candidate assistant-enabled device 210. That is, the GUI 400a of Figure 4A shows a ranked list 310 of candidate assistant-enabled devices 210 and corresponding controls 220 generated based on proximity estimates of the assistant-enabled devices 210 relative to the user device 200 in the environment 32 of Figure 3A. GUI 400b of FIG. 4B shows a ranked list 310 of candidate assistant-enabled devices 210 generated based on proximity estimates of the assistant-enabled devices 210 relative to user device 200 in environment 32 of FIG. 3B, and corresponding controls 220.4C shows a ranked list 310 of candidate assistant-enabled devices 210 and corresponding controls 220 generated based on proximity estimates of the assistant-enabled devices 210 relative to the user device 200 in the environment 32 of FIG. 3C. As is apparent, the ranked list 310 of candidate assistant-enabled devices 210 rendered in each of the GUIs 400a-c changes based on the proximity score 260 determined for each of the assistant-enabled devices 210 as the user device 200 moves to each of the different locations in the room 32 shown in the schematic diagrams 300a-c of FIGS. 3A-3C.

[0040] 3A and 4A, a user 30 is positioned across a room near a smart speaker 210a placed on a table, while other assistant-enabled devices 210, including a smart light 210b, a smart TV 210c, and a smart thermostat 210d, are positioned on the other side of the room. Furthermore, the user device 200 and the user 30 are facing toward the smart speaker 210a, indicating a high likelihood that the user 30 intends to control the smart speaker 210a via the assistant application 106. Furthermore, the user device 200 and the user 30 are facing away from the smart light 210b, the smart TV 210c, and the smart thermostat 210d, indicating a high likelihood that the user 30 does not intend to control the smart light 210b, the smart TV 210c, or the smart thermostat 210d via the assistant application 106.

[0041] 1 and 2, the user device 200 may continuously (or at periodic intervals) acquire proximity information 250 for each of the assistant-enabled devices 210 in the environment 32 to determine a corresponding proximity score 260, with each proximity score 260 indicating an estimate of the proximity of the corresponding assistant-enabled device 210 relative to the user device 200 in the environment 32. As shown in FIG. 3A, the smart speaker 210a is in closest proximity to the user 30 holding the user device 200, resulting in a higher proximity score 260. As the candidate assistant-enabled device 210 with the highest proximity score 260, it is a strong indication that the user 30 desires to control the smart speaker 210a. Conversely, the smart light 210b, the smart TV 210c, and the smart thermostat 210d are farther away from the user 30 holding the user device 200, resulting in lower proximity scores 260. Thus, as a magnitude, the proximity score 260 can serve as a strong indicator of whether the user 30 intends or does not intend to control the corresponding candidate assistant-enabled device 210. The user device 200 (i.e., using the assistant application 106) may identify the smart speaker 210a as having the highest proximity score 260 among the multiple assistant-enabled devices 210a-d as the target assistant-enabled device 210 that the user 30 intends to control.

[0042] Additionally, assistant-enabled devices 210 located in other rooms on a second floor of user premises 32, such as second smart TV 210e and second smart speaker 210f shown in FIG. 1, may have even lower proximity scores 260 that cause user device 200 (i.e., using assistant application 106) to disqualify those assistant-enabled devices 210 from consideration as possible candidates that user 30 intends to control. As a result, user device 200 may discard second smart TV 210e and second smart speaker 210f from ranked list 310 of candidate assistant-enabled devices 210 rendered in GUIs 400a-c of FIGS. 4A-4C.

[0043] 3A and 4A, after determining proximity scores 260 for each of assistant-enabled devices 210, user device 200 generates (i.e., using assistant application 106) a ranked list 310 of candidate assistant-enabled devices 210 and corresponding controls 220 based on proximity scores 260 indicating proximity estimates of assistant-enabled devices 210 relative to user device 200. More specifically, user device 200 generates ranked list 310 by ordering assistant-enabled devices 210 from assistant-enabled device 210a having closest proximity relative to user device 200 to assistant-enabled device 210d having furthest proximity relative to user device 200. In a further embodiment, proximity scores 260 are combined with additional context information, such as a confidence level for each set of available actions associated with the corresponding assistant-enabled device 210. For example, there may be a higher confidence level for turning off lights later in the night. In short, such contextual information can be used to bias proximity scores 260 to influence how assistant-enabled devices are positioned in a ranked list 310 personalized for a user.

[0044] 4A displays a ranked list 310 of candidate assistant-enabled devices 210 that user device 200 generates using proximity scores 260 determined for each of assistant-enabled devices 210. As previously described, user device 200 may determine that proximity scores 260 associated with second smart TV 210e and second smart speaker 210f located on a second floor of user premises 32 of FIG. 1 indicate proximity estimates that satisfy a maximum distance threshold, resulting in user device 200 discarding these assistant-enabled devices 210e, 210f from the ranked list 310 of candidate assistant-enabled devices 210. As a result of having the highest proximity score 260, the user device 200 ranks the smart speaker 210a higher in the ranked list 310, which results in the user device 200 rendering the corresponding graphical elements 402 representing the smart speaker 210a, the corresponding controls 220, and the device state information 215 more prominently in the GUI 400a compared to the graphical elements 402 representing each of the assistant-enabled devices 210b-d that are ranked lower in the ranked list 310. In the illustrated example, the user device 200 renders / displays the ranked list 310 of the candidate assistant-enabled devices 210 and the graphical elements 402 of the corresponding controls 220 in the GUI 400a in a larger font, with the smart speaker 210a at the top. GUI 400a is not limiting, and in other embodiments, user device 200 may render graphical element 402 representing smart speaker 210a associated with highest proximity score 260 in another location in GUI 400a, such as the center, and / or display that graphical element 402 differently from the graphical elements 402 representing other assistant-enabled devices 210b-d.This display format makes it easier for the user 30 to quickly find each set of controls 220 for performing each set of actions 120 associated with the smart speaker 210a.

[0045] 4A , each set of controls 220 for the smart speaker 210a displayed in the GUI 400a includes controls 220 for causing the smart speaker 210a to perform any action 120 from a respective set of actions 120 associated with a song playlist (e.g., a streaming song service) currently available for the smart speaker 210a to perform. Each set of controls 220 includes controls 220 that, when selected, cause the smart speaker 210a to perform the respective action 120: returning to a previous song track in the playlist, playing the song playlist, skipping to the next song track in the playlist, and adjusting the volume level of the smart speaker 210a. Continuing with this example, a graphical element 402 representing a smart speaker 210a located in the living room of the user's premises 32 displays device status information 215 associated with the smart speaker 210a indicating the battery level of the smart speaker 210a, as well as the fact that the song playlist is currently paused and, therefore, no songs are currently being output from the smart speaker 210a. Notably, the control 220 for performing the action 120 to pause the playlist is not displayed in the GUI 400a because the playlist is currently paused. Now, the user 30 may provide a user input indication indicating selection of the “Play” control 220 (e.g., by touching a graphical button in the GUI 400a generically representative of “Play”) to cause the smart speaker 210a to perform the respective action 120 of audibly playing the current song track in the playlist. Optionally, the user input indication indicating selection of the “play” control 220 may include a voice input, such as the user 30 uttering the word “play,” which may be captured by a microphone on the user device 200. Notably, the voice recognition capabilities for controlling the devices 210 in the ranked list 310 may be biased based on the proximity score 260.This follows from the idea that a user may be more likely to issue a voice-based command. Thus, a proximity score 260 that meets a threshold for a particular device 210 may cause the device 210 to open the microphone for capturing and perform voice recognition on the captured voice. Additionally or alternatively, the voice recognition capabilities for a particular device 210 may be biased to recognize the respective set of available actions 120 for that device 210. As shown in FIG. 3A, a user input instruction indicating selection of the “Play” control 220 displayed in the GUI 400a causes the user device 200 (i.e., the assistant application 106) to send a corresponding command 80 to the smart speaker 210a to perform the respective action 120 of audibly playing the current song track in the playlist.

[0046] 4A , user device 200 further displays in GUI 400a graphical elements 402 representing other assistant-enabled devices 210b-d in ranked list 310 of candidate assistant-enabled devices 210. Specifically, smart thermostat 210d is second in ranked list 310 because smart thermostat 210d has the second-highest proximity score 260, smart TV 210c is third in ranked list 310 because smart TV 210c has the third-highest proximity score 260, and smart light 210b is last in ranked list 310 because smart light 210b has the lowest proximity score 260. In the illustrated example, the graphical element 402 representing the smart thermostat 210d provides device status information 215 indicating the current temperature setting of the smart thermostat 210d (e.g., 21°C) and a respective set of controls 220 for controlling the smart thermostat 210d to perform a respective action 120 of increasing or decreasing the current temperature setting. Similarly, the graphical element 402 representing the smart light 210b provides device status information 215 indicating that the smart light 210b is currently off and a single control 220 for controlling the smart light 210b to perform a respective action 120 of turning on.

[0047] In some embodiments, for candidate assistant-enabled devices 210 in the ranked list 310 associated with lower ranking proximity scores 260, the GUI 400a may display only the device state information 215 associated with those assistant-enabled devices 210 without rendering the respective controls 220. However, if the user 30 desires to control any of these devices 210 to perform the respective actions 120, the user 30 may provide an input prompt to select the corresponding graphical element 402 that causes the GUI 400a to render the respective set of controls 220 in the GUI 400a. In the illustrated embodiment, the user device 200 (i.e., using the application 106) suppresses the display of the respective controls 220 for the smart TV 210c in the ranked list 310 due to the smart TV 210c having a low proximity score 260 (e.g., failing to meet the minimum proximity score threshold). This decision to suppress display of control 220 may be based on corresponding directionality information 255 indicating that user device 200 is facing away from smart TV 210c. Here, directionality information 255 may bias proximity score 260 for smart TV 210c by reducing the proximity score 260. Furthermore, user device 200 may consider context, such as the type of assistant-enabled device 210, when considering whether to bias proximity score 260 based on directionality information 255. In this example, the context indicates that assistant-enabled device 210c is a smart TV and that a user typically faces the smart TV when issuing a command. Therefore, directionality information 255 indicating that user device 200 is facing away from smart TV 210c can serve as a strong indicator that user 30 does not intend to control smart TV 210c.As a result, the user device 200 may optionally suppress the display of a control 220 in the GUI 400a to cause the smart TV 210c to perform an action 120, even if there is an action 120 available for the smart TV 210c to perform. On the other hand, directional information 255 associated with the smart light 210b may not be used to bias the proximity score 260 because the user 30 typically does not orient in any particular manner when controlling the operation of the smart light. Conversely, directional information 255 indicating that the user device 200 is facing the smart speaker 210a may further bias the proximity score 260 for the smart speaker 210a by increasing the proximity score 260. However, the amount of biasing may be lighter weighted because users often control smart speakers to perform actions even when facing other directions.

[0048] 3B and 4B, in response to a user 30 providing a user input indication indicating a selection of a "Play" control 220 displayed in GUI 400a of FIG. 4A, which causes the user device 200 to send a command 80 to the smart speaker 210a to perform the respective "Play" action 120 shown in FIG. 3A, the smart speaker 210a is audibly playing the current song track in the song playlist. Additionally, the user device 200 and user 30 are pointing in the direction of other assistant-enabled devices 210, including a smart light 210b, a smart TV 210c, and a smart thermostat 210d, thereby indicating that the user 30 likely intends to control one of the smart light 210b, the smart TV 210c, or the smart thermostat 210d via the assistant application 106.

[0049] 3B, smart speaker 210a remains in closest proximity to user 30 holding user device 200, resulting in a higher proximity score 260. As the candidate assistant-enabled device 210 with the highest proximity score 260, this is a strong indication that user 30 desires to control smart speaker 210a. Conversely, smart light 210b, smart TV 210c, and smart thermostat 210d remain farther away from user 30 holding user device 200 than smart speaker 210a, resulting in these assistant-enabled devices 210b-d having lower proximity scores 260 than smart speaker 210a.

[0050] 3B and 4B, after determining proximity scores 260 for each of assistant-enabled devices 210, user device 200 generates (i.e., using assistant application 106) a ranked list 310 of candidate assistant-enabled devices 210 and corresponding controls 220 based on proximity scores 260 indicating proximity estimates of assistant-enabled devices 210 relative to user device 200. More specifically, user device 200 generates ranked list 310 by ordering assistant-enabled devices 210 from assistant-enabled device 210a having closest proximity relative to user device 200 to assistant-enabled device 210d having furthest proximity relative to user device 200. In some implementations, one or more of proximity scores 260 are further determined or biased based on directionality information 255 associated with one or more of assistant-enabled devices 210.

[0051] 4B displays a ranked list 310 of candidate assistant-enabled devices 210 that the user device 200 generates using the proximity score 260 determined for each of the assistant-enabled devices 210. As previously described, the user device 200 may determine that the proximity score 260 associated with the second smart TV 210e and the second smart speaker 210f located on the second floor of the user's premises 32 of FIG. 1 may indicate a proximity estimate that still meets the maximum distance threshold, resulting in the user device 200 discarding these assistant-enabled devices 210e, 210f from the ranked list 310 of candidate assistant-enabled devices 210. As a result of maintaining the highest proximity score 260, the user device 200 ranks the smart speaker 210a higher in the ranked list 310, which results in the user device 200 rendering the corresponding graphical elements 402 representing the smart speaker 210a, the corresponding controls 220, and the device state information 215 more prominently in the GUI 400b compared to the graphical elements 402 representing each of the lower-ranked assistant-enabled devices 210b-d in the ranked list 310. Furthermore, the corresponding controls 220 for the higher-ranked assistant-enabled devices 210 in the ranked list may be displayed more prominently (e.g., larger, more controls, etc.). In the illustrated example, the user device 200 renders / displays the ranked list 310 of the candidate assistant-enabled devices 210 associated with the smart speaker 210a and the graphical elements 402 of the corresponding controls 220 at the top and in a larger font in the GUI 400b.GUI 400b is not limiting, and in other embodiments, user device 200 may render graphical element 402 representing smart speaker 210a associated with the highest proximity score 260 in another location in GUI 400b, such as the center of GUI 400b, and / or display that graphical element 402 differently from the graphical elements 402 representing the other assistant-enabled devices 210b-d. This display format makes it easier for user 30 to quickly find the respective set of controls 220 for performing the respective set of actions 120 associated with smart speaker 210a.

[0052] Continuing with reference to the exemplary GUI 400b of Figure 4B, the respective sets of controls 220 for the smart speaker 210a displayed in the GUI 400b include controls 220 that, when selected, cause the smart speaker 210a to perform respective actions 120: returning to a previous song track in the playlist, pausing the song playlist, skipping to the next song track in the playlist, and adjusting the volume level of the smart speaker 210a. Thus, because the device state information 215 for the smart speaker 210a indicates that the smart speaker 210a is currently playing a song from the song playlist (i.e., in response to a command 80 being sent from the user device 200 to the smart speaker 210a to perform the respective "play" action 120 shown in Figure 3A), the action 120 to pause the song playlist, which was not available as a control 220 in the GUI 400a of Figure 4A, is now available for selection in the GUI 400a of Figure 4B. Continuing with this example, a graphical element 402 representing a smart speaker 210a located in the living room of the user's premises 32 also displays device status information 215 associated with the smart speaker 210a indicating the battery level of the smart speaker 210a, as well as the fact that a song playlist is currently playing and, therefore, songs are currently being output from the smart speaker 210a. Notably, because the user 30 provided an input indication indicating selection of the "Play" control 220 in Figure 3A, the control 220 for performing the action 120 to play the playlist is not displayed in GUI 400b because the playlist is currently playing.

[0053] 4B , user device 200 further displays in GUI 400b graphical elements 402 representing other assistant-enabled devices 210b-d in ranked list 310 of candidate assistant-enabled devices 210. Specifically, smart thermostat 210d is second in ranked list 310 because smart thermostat 210d has the second-highest proximity score 260, smart TV 210c is third in ranked list 310 because smart TV 210c has the third-highest proximity score 260, and smart light 210b is last in ranked list 310 because smart light 210b has the lowest proximity score 260. In the illustrated example, the graphical element 402 representing the smart thermostat 210d provides device status information 215 indicating the current temperature setting of the smart thermostat 210d (e.g., 21°C) and a respective set of controls 220 for controlling the smart thermostat 210d to perform a respective action 120 of increasing or decreasing the current temperature setting. Similarly, the graphical element 402 representing the smart light 210b provides device status information 215 indicating that the smart light 210b is currently off and a single control 220 for controlling the smart light 210b to perform a respective action 120 of turning on.

[0054] In the illustrated example, the user device 200 (i.e., using the application 106) no longer suppresses the display of the respective control 220 for the smart TV 210c in the ranked list 310. This decision to now render the display of the respective control 220 for the smart TV 210c may be based on the corresponding directionality information 255 indicating that the user device 200 is now facing away from the smart speaker 210a and is now facing the smart TV 210c, as shown in the schematic diagram 300b of FIG. 3B. Here, the directionality information 255 indicating that the user device 200 is facing the smart TV 210c may cause the assistant application 106 to bias the proximity score 260 for the smart TV 210c by simply increasing the proximity score 260, and / or by simply choosing not to apply a bias that reduces the proximity score 260 as in FIG. 3A when the directionality information 255 indicates that the user device 200 is facing away from the smart TV 210c. In this example, the directionality information 255 indicating that the user device 200 is facing the smart TV 210c may serve as a strong indicator that the user 30 intends to control the smart TV 210c. As a result, the user device 200 may optionally render the display of controls 220 for "off" and "mute" in the GUI 400b to cause the smart TV 210c to perform actions 120 of powering off the smart TV 210c and muting the audio output of the smart TV 210c. Furthermore, the directionality information 255 indicating that the user device 200 is facing the smart TV 210c may cause the smart TV 210c to provide an indication that the user device 200 is facing the smart TV 210c, thereby serving as confirmation that the assistant application intends to control the TV.This indication can be a change in color on the screen, a change in color on a panel of the television 210c, an audible alert, or any other indication that confirms that the user device 200 is aligned with the television 200c.

[0055] While many embodiments described herein describe the assistant application 106 using received directional information 255 to bias the proximity score 260, other embodiments may include the assistant application 106 determining the proximity score 260 as a function of both the proximity information 250 and the directional information 255. In these embodiments, the type of the particular assistant-enabled device 210 providing the information 250, 255 may be further taken into consideration when determining the corresponding proximity score 260. For example, the assistant application 106 may adjust the weight of the proximity information 250 and / or the weight of the directional information 255 based on the type of the particular assistant-enabled device 210 to affect how the proximity information 250 and the directional information 255 affect the magnitude of the resulting proximity score 260 for that particular assistant-enabled device 210. In some implementations, the assistant application 106 uses proximity information 250 and directional information 255 received continuously over a period of time to determine whether the user device 200 is moving. In these implementations, when user device 200 is moving, assistant application 106 may further derive a current heading of user device 200 indicating which direction user device 200 is moving toward and, therefore, which assistant-enabled device 210 user device 200 is moving toward. The current heading may bias proximity score 260 determined for one or more assistant-enabled devices 210, or proximity score 260 determined for one or more assistant-enabled devices 210 may be determined as a function of the current heading of user device 200 derived from proximity information 250 and directionality information 255.

[0056] Furthermore, when considering whether to include in GUI 400 each control 220 for performing available actions 120 associated with one or more assistant-enabled devices 210 in environment 32, user device 200 may take into account the context of past behavior of user 30, who provides an input indication indicating selection of control 220 to cause assistant-enabled device 210 to perform each action 120 based on device state information 215 of one or more of assistant-enabled devices 210. For example, if the device state information 215 for the smart television 210c indicates that the smart television 210c is currently on, but the user 30 immediately mutes or turns off the smart television 210c every time the user 30 issues a command 80 to cause the smart speaker 210a to play a song, the user device 200 may make a decision to currently render display of the controls 220 for “off” and “mute” in the GUI 400b for the smart television 210c in response to the command 80 (FIG. 3A) being issued to cause the smart speaker 210a to play a song and the current device state information 215 for the smart television 210c indicating that the smart television 210c is currently on.

[0057] 3B and 4B , a user 30 desires to switch a smart light 210b from off to on and uses the assistant application 106 to provide a corresponding user input instruction (e.g., touch input, voice input, gaze, or gesture) indicating selection of an “on” control 220 displayed in GUI 400b for the smart light 210b. Here, the user device 200 receives the user input instruction indicating selection of the “on” control 220 displayed in GUI 400b for the smart light 210b, causing the user device 200 (i.e., the assistant application 106) to send a command 80 corresponding to the smart light 210b to perform a respective turn-on action 120.

[0058] 3C and 4C, the smart light 210b is currently turned on, and the user 30 has moved from across the room near the smart speaker 210a toward the opposite side of the room closest to the smart thermostat 210d, so that the smart light 210b and the smart TV 210c are now in closer proximity to the user device 200 compared to the example shown in FIG. 3B. The GUI 400c currently shows device state information 215 for the smart light 210b indicating that the smart light 210b is currently on, such that only one control 220 for the smart light 210b is rendered in the GUI 400c to control the smart light 210b to perform the only available action 120, which is to turn it off. Furthermore, FIG. 3C shows that the user device 200 is pointing toward the smart thermostat 210d, indicating that the user 30 likely intends to control the smart thermostat 210d via the assistant application 106. Conversely, user device 200 is not facing toward either smart light 210b or smart TV 210c, even though smart light 210b and smart TV 210c are in close proximity to user device 200. Directional information 255 indicating that user device 200 is not facing toward assistant-enabled device 210 may indicate a high likelihood that user 30 does not intend to control assistant-enabled device 210, e.g., smart light 210b, smart TV 210c, and smart speaker 210a in the illustrated example.

[0059] 1 and 2, user device 200 may continuously (or at periodic intervals) acquire proximity information 250, and optionally, directional information 255, for each of assistant-enabled devices 210 to determine a proximity score 260 for each of assistant-enabled devices 210 to generate a ranked list 310 of candidate assistant-enabled devices 210. As shown in FIG. 3C, smart thermostat 210d is in closest proximity to user 30 holding user device 200, while smart speaker 210a has the furthest proximity relative to user device 200, resulting in user device 200 initially determining a higher proximity score 260 for smart thermostat 210d compared to smart speaker 210a. Because smart thermostat 210d is the candidate assistant-enabled device 210 in ranked list 310 with the highest proximity score 260, there is a strong indication that user 30 intends to control smart thermostat 210d first. Smart light 210b and smart TV 210c include proximity closer to user device 200 than smart speaker 210a, but relatively farther from user device 200 than smart thermostat 210d, resulting in user device 200 first determining proximity scores 260 for smart TV 210c and smart light 210b that are lower than smart thermostat 210d, but higher than smart speaker 210a. Thus, the magnitude of proximity score 260 can serve as a strong indicator of whether user 30 intends or does not intend to control the corresponding candidate assistant-enabled device 210.Using only the proximity scores 260 determined for the assistant-enabled devices 210a-d in the environment 32, the user device 200 (i.e., using the assistant application 106) may identify the smart thermostat 210d having the highest proximity score 260 among the assistant-enabled devices 210a-d as the target assistant-enabled device 210 that the user 30 intends to control.

[0060] Additionally, user device 200 (i.e., via assistant application 106) may bias one or more of proximity scores 260 based on an underlying suggestion confidence associated with one or more of assistant-enabled devices 210. For example, if user 30 has recently interacted with assistant-enabled device 210, there may be a high suggestion confidence that user 30 wants to continue controlling assistant-enabled device 210. In this example, user device 200 may apply the suggestion confidence to bias proximity score 260 by increasing proximity score 260 associated with the corresponding assistant-enabled device 210. Conversely, if user 30 has not interacted with assistant-enabled device 210 for a threshold period, there may be a low suggestion confidence that user 30 wants to continue controlling assistant-enabled device 210, which may cause user device 200 to bias proximity score 260 associated with the corresponding assistant-enabled device 210 by decreasing / lowering / smaller proximity score 260. In the illustrated example, by selecting the “Play” control 220 in the GUI 400a to issue a command 80 for the smart speaker 210a from the user device 200, causing the smart speaker 210a to perform the respective action 120 of playing a song playlist, the user device 200 acquires a high suggestion confidence associated with the smart speaker 210a, indicating that the user 30 may seek to continue controlling the smart speaker 210a because the user device 200 has recently interacted with the smart speaker 210a in Figures 3A and 4A. As a result, even though the smart speaker 210a has the furthest proximity relative to the user device 200 and the directionality information 255 indicates that the user device 200 is facing a different direction than the smart speaker 210a, the user device 200 may apply the high suggestion confidence by increasing the proximity score 260 associated with the smart speaker 210a.Here, user device 200 (i.e., using assistant application 106) keeps smart speaker 210a and each set of controls 220 highest in ranked list 310 of candidate assistant-enabled devices rendered in GUI 400c of FIG. 4C. In another example, a high suggestion confidence indicating that user 30 intends to control a particular assistant-enabled device 210 independently of proximity information 250 and directional information 255 of that particular assistant-enabled device 210 relative to user device 200 causes assistant application 106 to simply override proximity score 260 determined for that particular assistant-enabled device 210 so that the particular assistant-enabled device 210 is ranked highest in ranked list 310. Similarly, a low suggestion confidence that the user 30 intends to control a particular assistant-enabled device 210 may cause the assistant application 106 to simply override the associated proximity score 260 of that particular assistant-enabled device 210, causing those devices 210 to be at least ranked lower in the ranked list 310 or rendered less prominently for display in the GUI 400. In fact, a low suggestion confidence may simply cause the assistant-enabled device 210 to be discarded entirely from the ranked list 310 of candidate assistant-enabled devices 210.

[0061] Additionally, user device 200 may discard assistant-enabled device 210 after determining that proximity score 260 indicates a proximity estimate that meets (e.g., is less than) a minimum distance threshold, where the minimum distance threshold is configured to filter out assistant-enabled devices 210 having proximity scores 260 that indicate the assistant-enabled devices 210 are sufficiently close to user device 200 such that user 30 directly manually controls assistant-enabled device 210 without using application 106. In the example shown in FIG. 3C , proximity score 260 associated with smart thermostat 210d indicates a proximity estimate that meets (e.g., is less than) the minimum distance threshold, resulting in user device 200 discarding smart thermostat 210d from ranked list 310 of candidate assistant-enabled devices 210 rendered in GUI 400c of FIG. 4C .

[0062] 4C displays a ranked list 310 of candidate assistant-enabled devices 210 that user device 200 generates using proximity scores 260 determined for each of the assistant-enabled devices 210. As previously described, user device 200 may determine that proximity score 260 associated with smart thermostat 210d indicates a proximity estimate that cannot exceed a minimum distance threshold, resulting in user device 200 discarding this assistant-enabled device 210d from the ranked list 310 of candidate assistant-enabled devices 210. On the other hand, a recent interaction with smart speaker 210a in FIG. 3A may increase proximity score 260 associated with smart speaker 210a, causing smart speaker 210a to have the highest ranking in the ranked list 310. Thus, user device 200 may continue to render graphical element 402 representing smart speaker 210a and its respective set of controls 220 in GUI 400c more prominently (e.g., at the top of ranked list 310, in a larger font) than graphical elements 402 representing other assistant-enabled devices 210 in ranked list 310. GUI 400c is non-limiting, and in other examples, user device 200 may render graphical element 402 representing smart speaker 210a associated with the highest proximity score 260 in another position in GUI 400c, such as the center, and / or display that graphical element 402 differently from graphical elements 402 representing other assistant-enabled devices 210b, 210c. This display format makes it easier for user 30 to quickly find the respective set of controls 220 for performing the respective set of actions 120 associated with smart speaker 210a.

[0063] The user device 200 also displays corresponding graphical elements 402 in the GUI 400c that represent the other assistant-enabled devices 210b, 210c in the ranked list 310 of candidate assistant-enabled devices 210. Specifically, the smart TV 210c is second in the ranked list 310 because it has the second-highest proximity score 260 (e.g., after increasing the proximity score 260 associated with the smart speaker 210a and discarding the smart thermostat 210d from the ranked list 310). The smart lighting 210b is third in the ranked list 310 because it has the third-highest proximity score 260. In the example GUI 400c of FIG. 4C, a graphical element 402 representing the smart TV 210c provides the same device state information 215 as in GUI 400b of FIG. 4B by indicating that the smart TV 210c is currently on, as well as the same respective set of controls 220 for controlling the smart TV 210c to perform the respective action 120 of turning off or muting the smart TV 210c.

[0064] 5 is a flowchart of an exemplary configuration of operations for a method 500 of controlling an assistant-enabled device. At operation 510, method 500 includes, by data processing hardware 202 of user device 200, acquiring proximity information 250 for each of a plurality of assistant-enabled devices 210 in environment 32 of user device 200. Examples of proximity information 250 for at least one assistant-enabled device 210 include any information or data that user device 200 may use to determine a proximity score 260 for the corresponding assistant-enabled device 210, where proximity score 260 indicates an estimate of the proximity of the corresponding assistant-enabled device 210 relative to user device 200 in environment 32. Each assistant-enabled device 210 of the plurality of assistant-enabled devices 210 is controllable by assistant application 106 to perform a respective set of available actions 120 associated with assistant-enabled device 210. For example, each set of available actions 120 for the smart speaker 210a may include one or more of: returning to a previous song track in a playlist, playing a song playlist, pausing a playlist, jumping to a next song track in a playlist, and adjusting the volume level of the smart speaker 210a.

[0065] At operation 520, method 500 includes, for each assistant-enabled device 210 of the plurality of assistant-enabled devices 210, determining, by data processing hardware 202, proximity score 260 based on proximity information 250 obtained for the corresponding assistant-enabled device 210. Proximity score 260 indicates a proximity estimate of the corresponding assistant-enabled device 210 relative to user device 200 in environment 32. The proximity estimate may indicate a distance from user device 200 at which the corresponding assistant-enabled device 210 is located and / or may include a position of the corresponding assistant-enabled device 210 relative to the position of user device 200.

[0066] At operation 530, method 500 also includes generating, by data processing hardware 202, a ranked list 310 of candidate assistant-enabled devices 210 from the plurality of assistant-enabled devices 210 using the plurality of proximity scores 260 determined for the plurality of assistant-enabled devices 210. In some embodiments, method 500 generates the ranked list 310 of candidate assistant-enabled devices 210 by ordering the assistant-enabled devices 210 from the assistant-enabled device 210 having the closest proximity to the user device 200 to the assistant-enabled device 210 having the furthest proximity to the user device 200. The ranked list 310 of candidate assistant-enabled devices 210 may include the assistant-enabled devices 210 having the highest N-th proximity scores 260 among the plurality of proximity scores 260 determined from the plurality of assistant-enabled devices 210. Additionally, generating the ranked list 310 of candidate assistant-enabled devices 210 may further include discarding from the ranked list 310 of candidate assistant-enabled devices 210 those assistant-enabled devices 210 that include proximity scores 260 indicating proximity estimates that meet a maximum distance threshold (i.e., are too far from the user device 200) and / or that meet a minimum distance threshold (i.e., are too close to the user device 200).

[0067] At operation 540, method 500 also includes, for each of one or more corresponding assistant-enabled devices 210 in ranked list 310 of candidate assistant-enabled devices 210, displaying, by data processing hardware 202, a respective set of controls 220 for performing the respective set of actions 120 associated with the corresponding assistant-enabled device 210 in graphical user interface (GUI) 400 displayed on a screen of user device 200 communicating with data processing hardware 202. For each candidate assistant-enabled device 210, user device 200 may render a corresponding graphical element 402 in GUI 400 representing the candidate assistant-enabled device 210, the corresponding control 220 of the device 210, and / or device state information 215. Furthermore, method 500 for displaying in GUI 400 respective sets of controls 220 for performing respective sets of actions 120 associated with corresponding assistant-enabled devices 210 may include re-ranking candidate assistant-enabled devices 210 in ranked list 310 of candidate assistant-enabled devices 210 in response to receiving device state information 215 associated with at least one of assistant-enabled devices 210. Additionally or alternatively, method 500 for displaying respective sets of controls 220 for performing respective sets of actions 120 may include displaying respective sets of controls 220 for one assistant-enabled device 210 differently from respective sets of controls 220 displayed in GUI 400 for other assistant-enabled devices 210. For example, an assistant-enabled device 210 having the highest proximity score 260 and ranked highest in ranked list 310 may be displayed more prominently in GUI 400 compared to other assistant-enabled devices 210 having lower proximity scores 260.

[0068] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some embodiments, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0069] Non-transitory memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transitory memory may be volatile addressable semiconductor memory and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0070] 6 is a schematic diagram of an exemplary computing device 600 that may be used to implement the systems and methods described herein. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are intended to be merely exemplary and are not intended to limit the implementations of the invention described and / or claimed herein.

[0071] Computing device 600 includes a processor 610, a memory 620, a storage device 630, a high-speed interface / controller 640 connected to memory 620 and a high-speed expansion port 650, and a low-speed interface / controller 660 connected to a low-speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 are connected to one another using various buses and may be implemented on a common motherboard or in other manners, as appropriate. Processor 610 is capable of processing instructions for execution within computing device 600, including instructions stored in memory 620 or in storage device 630, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 680 coupled to high-speed interface 640. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and multiple types of memory, as appropriate. Also, multiple computing devices 600 may be connected (eg, as a server bank, as a group of blade servers, or as a multiprocessor system), with each device providing a portion of the required operations.

[0072] The memory 620 stores information non-transiently within the computing device 600. The memory 620 may be a computer-readable medium, volatile memory unit(s), or non-volatile memory unit(s). The non-transient memory 620 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 600. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disk or tape.

[0073] The storage device 630 can provide mass storage for the computing device 600. In some implementations, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or devices in a storage area network configuration or other configuration. In further implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 620, the storage device 630, or memory on the processor 610.

[0074] The high-speed controller 640 manages bandwidth-intensive operations for the computing device 600, while the low-speed controller 660 manages less bandwidth-intensive operations. Such an allocation of duties is merely exemplary. In some implementations, the high-speed controller 640 is coupled to the memory 620, the display 680 (e.g., via a graphics processor or graphics accelerator), and also to a high-speed expansion port 650 capable of accepting various expansion cards (not shown). In some implementations, the low-speed controller 660 is coupled to the storage device 630 and the low-speed expansion port 690. The low-speed expansion port 690, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled, for example, via a network adapter, to one or more input / output devices such as a keyboard, pointing device, scanner, or a networking device such as a switch or router.

[0075] Computing device 600, as shown, may be implemented in several different forms, such as a standard server 600a, a group of multiple such servers 600a, a laptop computer 600b, or as part of a rack server system 600c.

[0076] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from the storage system on at least one input device and to transmit data and instructions to the storage system on at least one output device.

[0077] These computer programs (also known as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, computer-readable apparatus, and / or computer-readable device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0078] The processes and logic flows described herein may be implemented by one or more programmable processors, also referred to as data processing hardware, that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special purpose microprocessors, and any one or more processors of any type of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or is operatively coupled to receive data from or transfer data to such storage devices, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0079] To enable user interaction, one or more aspects of the present disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, e.g., a mouse or trackball, through which the user can provide input to the computer. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, voice input, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from a device used by the user, e.g., by sending a web page to a web browser on the user's client device in response to a request received from the web browser.

[0080] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0081] 32 Environment 80 Commands 106 Assistant Applications 108 User Request 120 Actions 200 user devices 202 Data Processing Hardware 210, 210a, 210b, 210c, 210d, 210e, 210f Assistant-enabled devices 220 Control 250 Proximity Information 255 Directional information 260 Proximity Score 310 Rated List 400 Graphical User Interface 402 Graphical Elements 600, 600a, 600b, 600c Computing Devices 610 processor 620 memory 630 Storage Devices 640, 660 Controller 650, 690 Expansion Port 670 Bus 680 display

Claims

1. 1. A computer-implemented method that, when executed on data processing hardware of a user device, causes the data processing hardware to perform an operation, the operation comprising: Initiating communication of each of a plurality of assistant-enabled devices over a network using an assistant application executing on the data processing hardware of the user device, wherein each assistant-enabled device of the plurality of assistant-enabled devices is located within a home of a user of the user device and is controllable by the assistant application to perform a respective set of available actions associated with the assistant-enabled device; receiving a current context of the user device; Automatically detecting a list of candidate assistant-enabled devices that the user wants to control using the assistant application from among the plurality of assistant-enabled devices based on the current context of the user device; Determining the current time; For each candidate assistant-enabled device in said list of candidate assistant-enabled devices, determining a respective confidence level for the respective set of available actions associated with the candidate assistant-enabled device based on the current time; and displaying graphical elements representing the candidate assistant-enabled devices based on the respective trust levels in a graphical user interface (GUI) displayed on a screen of the user device; When displaying the list of candidate assistant-enabled devices in the GUI, a graphical element of one of the candidate assistant-enabled devices having the highest confidence level is displayed in the GUI larger than the respective graphical elements of the other candidate assistant-enabled devices displayed in the GUI, Computer-implemented methods.

2. The operation further comprises: receiving a user request to launch the assistant application for execution on the user device; and executing the assistant application on the user device in response to receiving the user request. The method of claim 1.

3. The method of claim 2 , wherein receiving the user request includes receiving a user input indication indicating a selection of a graphical element displayed in the GUI representing the assistant application.

4. 3. The method of claim 2, wherein receiving the user request includes receiving voice input from the user, the voice input including an invocation command to launch the assistant application for execution on the user device.

5. The method of claim 1, wherein the graphical element of one of the candidate assistant-enabled devices displayed in the GUI larger than the respective graphical elements of the other candidate assistant-enabled devices is located in the center of the screen of the user device.

6. The method of claim 1, wherein the graphical element of one of the candidate assistant-enabled devices displayed in the GUI larger than the respective graphical elements of the other candidate assistant-enabled devices includes a font larger than the font of the respective graphical elements of the other candidate assistant-enabled devices displayed in the GUI.

7. The operation further comprises: receiving, from each assistant-enabled device of the plurality of assistant-enabled devices, a respective set of available actions associated with the corresponding assistant-enabled device; For each corresponding assistant-enabled device in the list of candidate assistant-enabled devices, determining a respective set of controls for performing the respective set of available actions associated with the corresponding assistant-enabled device. The method of claim 1.

8. The operation further comprises: receiving, in the data processing hardware, from at least one assistant-enabled device of the plurality of assistant-enabled devices, device state information associated with the corresponding assistant-enabled device; Determining the respective set of controls for performing the respective set of available actions associated with the corresponding assistant-enabled device is further based on the device state information associated with the corresponding assistant-enabled device. The method of claim 7.

9. The method of claim 1 , wherein the assistant application is installed on the user device prior to the user purchasing the user device.

10. The method of claim 1 , wherein the user device comprises a smartwatch.

11. data processing hardware; memory hardware in communication with the data processing hardware, the memory hardware, when executed on the data processing hardware, causing the data processing hardware to: Initiating communication of each of a plurality of assistant-enabled devices over a network using an assistant application executing on the data processing hardware of a user device, wherein each assistant-enabled device of the plurality of assistant-enabled devices is located within a home of a user of the user device and is controllable by the assistant application to perform a respective set of available actions associated with the assistant-enabled device; receiving a current context of the user device; Automatically detecting a list of candidate assistant-enabled devices that the user wants to control using the assistant application from among the plurality of assistant-enabled devices based on the current context of the user device; Determining the current time; For each candidate assistant-enabled device in said list of candidate assistant-enabled devices, determining a respective confidence level for the respective set of available actions associated with the candidate assistant-enabled device based on the current time; Displaying graphical elements representing the candidate assistant-enabled devices based on the respective trust levels in a graphical user interface (GUI) displayed on a screen of the user device; and memory hardware storing instructions for performing operations including: When displaying the list of candidate assistant-enabled devices in the GUI, a graphical element of one of the candidate assistant-enabled devices having the highest confidence level is displayed in the GUI larger than the respective graphical elements of the other candidate assistant-enabled devices displayed in the GUI, User device.

12. The operation further comprises: receiving a user request to launch the assistant application for execution on the user device; and executing the assistant application on the user device in response to receiving the user request. The user device of claim 11.

13. The user device of claim 12, wherein receiving the user request includes receiving a user input indication indicating a selection of a graphical element displayed in the GUI representing the assistant application.

14. 13. The user device of claim 12, wherein receiving the user request includes receiving voice input from the user, the voice input including a call command to launch the assistant application for execution on the user device.

15. The graphical element of one of the candidate assistant-enabled devices displayed in the GUI larger than the respective graphical elements of the other candidate assistant-enabled devices is located in the center of the screen of the user device. The user device of claim 11.

16. The graphical element of one of the candidate assistant-enabled devices displayed in the GUI larger than the respective graphical elements of the other candidate assistant-enabled devices includes a font larger than the font of the graphical element of the other candidate assistant-enabled devices displayed in the GUI. The user device of claim 11.

17. The operation further comprises: receiving, from each assistant-enabled device of the plurality of assistant-enabled devices, a respective set of available actions associated with the corresponding assistant-enabled device; For each corresponding assistant-enabled device in the list of candidate assistant-enabled devices, determining a respective set of controls for performing the respective set of available actions associated with the corresponding assistant-enabled device. The user device of claim 11.

18. The operation further comprises: receiving, from at least one assistant-enabled device of the plurality of assistant-enabled devices, device state information associated with the corresponding assistant-enabled device; Determining the respective set of controls for performing the respective set of available actions associated with the corresponding assistant-enabled device is further based on the device state information associated with the corresponding assistant-enabled device.

20. The user device of claim 17.

19. The user device of claim 11 , wherein the assistant application is installed on the user device prior to the user purchasing the user device.

20. The user device of claim 11 , wherein the user device comprises a smartwatch.

21. 1. A computer-implemented method that, when executed on data processing hardware of a user device, causes the data processing hardware to perform an operation, the operation comprising: Obtaining proximity information for each of a plurality of assistant-enabled devices in an environment of the user device, wherein each assistant-enabled device of the plurality of assistant-enabled devices is controllable by the user device to perform a respective set of available actions associated with the assistant-enabled device; determining a respective confidence level for the respective set of available actions associated with the assistant-enabled device based on a current time; and For each assistant-enabled device of the plurality of assistant-enabled devices, determine a proximity score based on the proximity information obtained for the corresponding assistant-enabled device and the trust level of the corresponding assistant-enabled device, wherein the proximity score indicates a proximity estimate of the corresponding assistant-enabled device relative to the user device in the environment; Selecting an assistant-enabled device from the plurality of assistant-enabled devices having a proximity score that meets a maximum distance threshold; Displaying a respective set of controls for performing a respective set of actions associated with the selected assistant-enabled device in a graphical user interface (GUI) displayed on a screen communicating with the user device; Computer-implemented methods.

22. 22. The method of claim 21 , wherein the operations further include receiving device state information for each assistant-enabled device of the plurality of assistant-enabled devices, and wherein selecting an assistant-enabled device from the plurality of assistant-enabled devices having a proximity score that meets a maximum distance threshold is further based on the device state information of the selected assistant-enabled device indicating that the assistant-enabled device is currently on.

23. The method of claim 21 , wherein the maximum distance threshold is configurable.

24. 22. The method of claim 21, wherein each assistant-enabled device of the plurality of assistant-enabled devices is located within a home of a user of the user device and is controllable by the user device to perform the respective set of available actions associated with the assistant-enabled device.

25. The operation further comprises: receiving a user request to launch an assistant application for execution on the user device; and executing the assistant application on the user device in response to receiving the user request.

22. The method of claim 21.

26. 26. The method of claim 25, wherein receiving the user request includes receiving a user input indication indicating a selection of a graphical element displayed in the GUI representing the assistant application.

27. 26. The method of claim 25, wherein receiving the user request includes receiving voice input from a user, the voice input including an invocation command to launch the assistant application for execution on the user device.

28. The operation further comprises: receiving, from each assistant-enabled device of the plurality of assistant-enabled devices, a respective set of available actions associated with the corresponding assistant-enabled device; receiving, from at least one assistant-enabled device of the plurality of assistant-enabled devices, device state information associated with the corresponding assistant-enabled device; For each assistant-enabled device of the plurality of assistant-enabled devices, determining a respective set of controls for performing the respective set of available actions associated with the corresponding assistant-enabled device based on the device state information associated with the corresponding assistant-enabled device.

22. The method of claim 21.

29. 22. The method of claim 21, wherein the operations further include receiving directional information for each assistant-enabled device of the plurality of assistant-enabled devices.

30. The method of claim 21 , wherein the user device comprises a smartwatch.

31. data processing hardware; memory hardware in communication with the data processing hardware, the memory hardware, when executed on the data processing hardware, causing the data processing hardware to: Obtaining proximity information for each of a plurality of assistant-enabled devices in an environment of a user device, wherein each assistant-enabled device of the plurality of assistant-enabled devices is controllable by the user device to perform a respective set of available actions associated with the assistant-enabled device; determining a respective confidence level for the respective set of available actions associated with the assistant-enabled device based on a current time; and For each assistant-enabled device of the plurality of assistant-enabled devices, determine a proximity score based on the proximity information obtained for the corresponding assistant-enabled device and the trust level of the corresponding assistant-enabled device, wherein the proximity score indicates a proximity estimate of the corresponding assistant-enabled device relative to the user device in the environment; Selecting an assistant-enabled device from the plurality of assistant-enabled devices having a proximity score that meets a maximum distance threshold; Displaying a graphical user interface (GUI) displayed on a screen communicating with the user device, a respective set of controls for performing a respective set of actions associated with the selected assistant-enabled device; and memory hardware storing instructions for performing operations including: system.

32. 32. The system of claim 31, wherein the operations further include receiving device state information for each assistant-enabled device of the plurality of assistant-enabled devices, and selecting an assistant-enabled device from the plurality of assistant-enabled devices having a proximity score that meets a maximum distance threshold is further based on the device state information of the selected assistant-enabled device indicating that the assistant-enabled device is currently on.

33. 32. The system of claim 31, wherein the maximum distance threshold is configurable.

34. 32. The system of claim 31, wherein each assistant-enabled device of the plurality of assistant-enabled devices is located within a home of a user of the user device and is controllable by an assistant application to perform the respective set of available actions associated with the assistant-enabled device.

35. The operation further comprises: receiving a user request to launch an assistant application for execution on the user device; and executing the assistant application on the user device in response to receiving the user request.

32. The system of claim 31.

36. 36. The system of claim 35, wherein receiving the user request includes receiving a user input indication indicating a selection of a graphical element displayed in the GUI representing the assistant application.

37. 36. The system of claim 35, wherein receiving the user request includes receiving voice input from a user, the voice input including an invocation command to launch the assistant application for execution on the user device.

38. The operation further comprises: receiving, from each assistant-enabled device of the plurality of assistant-enabled devices, a respective set of available actions associated with the corresponding assistant-enabled device; receiving, from at least one assistant-enabled device of the plurality of assistant-enabled devices, device state information associated with the corresponding assistant-enabled device; For each assistant-enabled device of the plurality of assistant-enabled devices, determining a respective set of controls for performing the respective set of available actions associated with the corresponding assistant-enabled device based on the device state information associated with the corresponding assistant-enabled device.

32. The system of claim 31.

39. The operations further include receiving directional information for each assistant-enabled device of the plurality of assistant-enabled devices; Determining the proximity score for the corresponding assistant-enabled device is further based on a proximity estimate of the corresponding assistant-enabled device relative to the user device in the environment.

32. The system of claim 31.

40. The system of claim 32 , wherein the user device comprises a smartwatch.

Images