Conditionally assign various auto-attendant functions to interactions with peripheral assistant control devices

Abstract

In response to a user interacting with a tangible peripheral assistant control device (e.g., pressing a button of the device), causing the automated assistant to perform one or more actions. The one or more actions performed may be based on input previously provided by the user when configuring the peripheral assistant control device. The one or more actions performed in response to interaction with the peripheral assistant control device may vary based on one or more conditions, such as which user is currently active, where the peripheral assistant control device is currently located (which may optionally be inferred based on which of a plurality of assistant computing devices the button is paired with), and / or the current state of one or more smart devices and / or other devices (e.g., as determined based on device topology). The utility of the peripheral assistant control device may be automatically extended beyond what was specifically requested by the user during configuration.

Description

Background Art

[0001] Humans may participate in human-computer conversations using interactive software applications referred to herein as "automated assistants" (also referred to as "digital agents," "chatbots," "interactive personal assistants," "intelligent personal assistants," "assistant applications," "conversational agents," and the like). For example, humans (who may be referred to as "users" when they interact with an automated assistant) may provide commands and / or requests to the automated assistant by using spoken natural language input (i.e., utterances) (which, in some cases, may be converted to text and then processed) and / or by providing textual (e.g., typed) natural language input. The automated assistant responds to the requests by providing responsive user interface output, which may include auditory and / or visual user interface output.

[0002] In many cases, users provide spoken utterances to cause an automated assistant to perform various actions. However, a large amount of computing and / or network resources may be used to process spoken utterances to perform automated assistant actions. For example, an automated assistant action of turning on a smart light may be implemented by transmitting a corresponding command directly to the smart light, or to a third-party server that processes the command and then transmits the corresponding command to the smart light. However, when performing such an action in response to the spoken utterance "Assistant, turn on smart light X", speech-to-text processing is performed on audio data that captures the spoken utterance, natural language processing is performed based on the resulting text, and then the corresponding command is determined. Further, where speech-to-text and / or natural language processing is performed remotely, the audio data and / or other data will need to be transmitted over one or more networks. Summary of the invention

[0003] The embodiments described herein relate to enabling an automated assistant to perform one or more actions in response to the interaction between a user and a peripheral assistant control device. The peripheral assistant control device may be paired with an assistant computing device (e.g., via Bluetooth or other communication protocols), the assistant computing device including an automated assistant interface, and enabling the execution of various automated assistant functions individually and / or by interacting with a remote automated assistant component. In response to the interaction between a user and a peripheral assistant control device, an output may be provided by the peripheral assistant control device to a paired assistant computing device. In response to receiving the output, the paired assistant computing device may initiate the execution of one or more automated assistant actions / functions. The embodiments described herein may be used to increase the utility of a peripheral assistant control device, so that fewer peripheral assistant control devices may be required to enable a given number of automated assistant functions. In addition, some embodiments may facilitate / reduce the complexity of the configuration of a peripheral assistant control device. In addition, since the peripheral assistant control device may reduce the amount of speech recognition processing (and associated bandwidth usage, etc.) required by the automated assistant system, increasing the utility / function of the peripheral assistant control device may be used to further reduce the processing, bandwidth, etc. used by the automated assistant system as a whole.

[0004] As described herein, the automatic assistant actions performed can be based on user interface input previously provided by a user when configuring the peripheral assistant control device. Moreover, the actions performed can vary based on one or more conditions, such as which of multiple users is determined to be currently active, where the peripheral assistant control device is currently located (which can optionally be inferred based on which of the multiple assistant computing devices the peripheral assistant control device is paired with), and / or the current state of one or more smart devices and / or other devices (e.g., as determined based on device topology). Also as described herein, the automatic assistant actions performed can include one or more actions that extend beyond the actions specifically requested by the user during configuration of the peripheral assistant control device.

[0005] The peripheral assistant control device itself may lack direct access to the automated assistant, and may lack various hardware resources present on the paired assistant computing device, or include less powerful versions of various hardware resources present on the paired assistant computing device. For example, the peripheral assistant control device may lack a microphone and a Wi-Fi module, while the paired assistant computing device includes both. Further, although the peripheral assistant control device may include a processor, it may be less powerful than the processor included at the paired assistant computing device. In some embodiments, the peripheral assistant control device may include a virtual button provided at a graphical user interface, and / or any other interface capable of presenting a virtual button. Alternatively or additionally, the peripheral assistant control device may include a physical button with one or more user interfaces, through which a user can provide one or more different inputs. The one or more different inputs may enable the peripheral assistant control device to provide one or more different outputs, respectively, over a wireless communication channel.

[0006] One or more users may provide user interface input to configure the peripheral assistant control device. For example, user interface input may be provided at the assistant computing device and / or the peripheral assistant control device to initially pair the assistant computing device and the peripheral assistant control device. Additional user interface input may be provided to explicitly assign a certain (or certain) automatic assistant function to the interaction with the peripheral assistant control device. For example, a user may say "Assistant, when I push this button, turn on the living room lights". In various embodiments, although the user only specifies a certain (or certain) automatic assistant function to be assigned to the interaction with the peripheral assistant control device, one or more additional automatic assistant functions may be further automatically suggested (and assigned if an affirmative user interface input is received), or further automatically assigned (without first suggesting).

[0007] As an example, for the spoken utterance “Assistant, when I push this button turn on the living room lights,” an automatic assistant action that causes the on / off attribute of the “living room” light to be commanded to the “on” state is explicitly indicated by the spoken utterance, and thus may be assigned to the peripheral assistant control device. Although not explicitly indicated by the spoken utterance, an additional automatic assistant function that causes the on / off attribute of the “living room” light to be commanded to the “off” state may be further assigned. In addition, causing the on / off attribute of the “living room” light to be commanded to the “on” state may be further assigned to the first condition that the on / off attribute of the “living room” light is in the “off” state, and causing the on / off attribute of the “living room” light to be commanded to the “off” state may be further assigned to the second condition that the on / off attribute of the “living room” light is in the “on” state. Thus, in response to receiving output in response to an interaction from a peripheral assistant control device, the current state of the on / off property of the "living room" light can be determined (e.g., by querying the light and / or by using the device topology (with the most recently updated device state)), and based on whether the current condition is the first condition or the second condition, the light is commanded to the on state or the off state.

[0008] As another example, for the spoken utterance “Assistant, when I push this button play some bluegrass music,” an automated assistant action that causes a paired assistant computing device (and / or other device, if specified) to play “bluegrass music” is explicitly indicated by the spoken utterance, and thus may be assigned to a peripheral assistant control device. Although not explicitly indicated by the spoken utterance, an additional automated assistant function may be additionally assigned that causes the currently playing “bluegrass song” to be replaced by the playing of another “bluegrass song” (e.g., a “next song” automated assistant function). Further, the play “bluegrass music” automated assistant function may be additionally assigned to a first condition that the assistant computing device is not currently presenting “bluegrass music” (or, alternatively, not presenting any music), and the “next song” automated assistant function may be additionally assigned to a second condition that “bluegrass music” (or, alternatively, any music) is currently being presented at the assistant computing device. Thus, in response to receiving output responsive to an interaction from a peripheral assistant control device, a play “bluegrass music” automatic assistant function may be initiated by the assistant computing device if a first condition is met, while a “next song” automatic assistant function may alternatively be initiated by the assistant computing device if a second condition is met.

[0009] Thus, even though a user may provide the same user interaction (e.g., a button press) at a peripheral assistant control device on two separate occasions, and in response transmit the same output from the peripheral assistant control device to a paired assistant computing device, different automated assistant functionality may ultimately appear at the two separate occasions due to different conditions at the two separate occasions.

[0010] Automatically suggesting and / or automatically assigning one or more additional automatic assistant functions to a peripheral assistant control device generates various technical benefits. For example, it enables the execution of additional automatic assistant functions in response to certain interactions between a user and a peripheral automatic assistant device. Additional automatic assistant functions can be executed in response to receiving an output provided by a peripheral automatic assistant device and determining that a certain (or certain) condition is satisfied-this can be significantly more effective (in terms of computing and / or networks) than processing corresponding spoken words. In addition, for example, it can reduce the amount of user input required for explicitly assigning additional automatic assistant functions by the user in other cases. Moreover, automatically changing the automatic assistant function that is activated in response to receiving an output from a peripheral assistant control device according to the current conditions also leads to various technical benefits. For example, starting the correct automatic assistant function for the current conditions can ensure that the processor and / or network resources used to execute the automatic assistant function will not be wasted. For example, the technology of not starting different automatic assistant functions for different conditions will wastefully generate and transmit repeated and ignored commands (for example, even if the light is already "on", the "turn on the light" command is generated and transmitted).

[0011] In some additional and / or alternative embodiments, different automated assistant functions may be assigned to the same interaction with a peripheral automated assistant control device, and which of the automated assistant functions is mimicked in response to a given interaction instance may depend on where the peripheral assistant control device is currently located. For example, when the peripheral assistant control device is located in or near the "living room", a first automated assistant function that turns on and / or turns off the "living room" lights may be initiated, while when the peripheral assistant control device is located in or near the "basement", a second automated assistant function that "plays some music on the basement device" may be initiated. Various techniques may be utilized to determine where the device is currently located. For example, where the device is currently located may be based on which of the multiple assistant computing devices the button is currently paired with. The pairing of the peripheral assistant control device may be switched between different assistant computing devices in response to user interface input (in the corresponding assistant computing device living room and / or at the peripheral assistant control device), or may be automatically switched so that the assistant computing device (e.g., Bluetooth) with the strongest (or only detectable) communication signal is paired with the peripheral assistant control device.

[0012] In some embodiments, an explicit user interface input may be provided that indicates which automated assistant function(s) should be performed for which locations of the peripheral assistant control device. For example, a user may provide a verbal utterance such as, "Assistant, when the interface device is in the living room, have it control the television, but when the interface device is in the kitchen, have it control playback of podcasts." In response, the automated assistant may establish corresponding configuration settings for the peripheral assistant control device. For example, by using data indicating the location of the peripheral assistant control device, the automated assistant may relate the output of the peripheral assistant control device to a specific function based on where the peripheral assistant control device is indicated to be located. For example, a peripheral assistant control device may be determined to be in the "living room" based on its pairing with an automated assistant computing device that is assigned to the "living room" location in the device topology. The location of the peripheral assistant control device can be determined using additional and / or alternative techniques, such as techniques for determining location based on network strength data, data from another device with which the peripheral assistant control device is paired or otherwise communicating, data identifying a specific device with which the peripheral assistant control device is paired or otherwise communicating, altitude data, motion data, audio data, image data, and / or any other data from which the location of the device can be determined.

[0013] In some embodiments, the user interface input may specify an automatic assistant function that controls a type of smart device and may specify that it is assigned to the peripheral assistant control device. However, the user interface input may not specify a specific smart device to be controlled. For example, a user may provide a verbal utterance such as "Assistant, have the interface device control turn the lights on and off for the room it is in". In some of these embodiments, specific automatic assistant functions that are each directed to control one or more specific smart devices of the type and are each associated with a corresponding condition (such as the location of the peripheral assistant control device) may be assigned. For example, a first automatic assistant function that turns on / off the "living room" light may be assigned to a first condition that the peripheral assistant control device is in the "living room", while a second automatic assistant function that turns on / off the "kitchen" light may be assigned to a second condition that the peripheral assistant control device is in the "kitchen". When the peripheral assistant control device is determined to be in the "kitchen" (e.g., based on pairing with a "kitchen" assistant computing device), the output received in response to the interaction with the peripheral assistant control device may result in control of the on / off state of the light that is also assigned to the "kitchen" (e.g., in the device topology). On the other hand, when the peripheral assistant control device is determined to be in the “Living Room” (e.g., based on pairing with a “Living Room” assistant computing device), output received in response to interaction with the peripheral assistant control device can result in control of the on / off state of a light that is also assigned to the “Living Room” (e.g., in the device topology).

[0014] Assigning different automatic assistant functions based on where the peripheral assistant control device is currently located leads to various technical benefits. For example, it enables the functions of the peripheral assistant control device to be dynamically adjusted based on the location without requiring the user to explicitly reconfigure the functions by providing resource-consuming additional interface inputs each time the peripheral assistant control device is moved. In addition, for example, it enables a single peripheral assistant control device and optionally a single interaction to have multiple functions based on the location - thus eliminating the need for multiple separate peripheral assistant control devices for multiple functions.

[0015] In some additional and / or alternative embodiments, different automated assistant functions may be assigned to the same interaction with a peripheral automated assistant control device, and which of the automated assistant functions is emulated in response to a given interaction instance may depend on which of the multiple users is currently the active user. For example, when the first user is the active user, a first automated assistant function that turns on and / or off the "living room" light may be initiated, and when the second user is the active user, a second automated assistant function that "plays some music" may be initiated. Various techniques may be utilized to determine which of the multiple users is currently the active user. For example, a user who has recently interacted with an assistant computing device paired with a peripheral assistant control device may be considered the currently active user (optionally depending on whether the most recent interaction was within the last 5 minutes or within another time threshold). Based on verifying the user based on, for example, matching the verbal input from the interaction with the stored speaker embedding of the user, it may be determined that the particular user has interacted with the assistant computing device. Additional and / or alternative authentication techniques may be utilized, such as a user-provided personal identification number (PIN), a user-provided password, and / or other biometric identification techniques (with permission from the corresponding user).

[0016] Assigning different automatic assistant functions based on which user is the currently active user of the peripheral assistant control device results in various technical benefits. For example, it enables the functions of the peripheral assistant control device to be dynamically adjusted based on the currently active user without requiring the user to explicitly reconfigure the functions by providing resource-consuming additional interface inputs each time the active user changes. In addition, for example, it enables a single peripheral assistant control device and optionally a single interaction to have multiple functions depending on the user - thus eliminating the need for multiple separate peripheral assistant control devices for multiple functions.

[0017] In some embodiments, in response to the automatic assistant detecting the presence of the peripheral assistant control device, the conditions for when the peripheral assistant control device is assigned the control of a specific function can be automatically established. In some embodiments, the peripheral assistant control device can communicate via Bluetooth, and when the user turns on the peripheral assistant control device, the device can be paired with an existing client device owned by the user or otherwise associated with the user. In response to the pairing of the peripheral assistant control device with the existing client device, the existing client device can call the automatic assistant, which can provide a natural language output to guide the additional configuration of the peripheral assistant control device. For example, the automatic assistant can cause the existing client device to present an audible natural language output, such as, "George, I've detected a new interface device in the living room. Would you like the new interface device to control the lights in the living room?". The natural language output can be generated based on the automatic assistant that identifies the location of the peripheral assistant control device, other existing devices in the living room that can be controlled via the automatic assistant, and / or any other device that communicates with the existing client device.

[0018] In response to the natural language output from the automated assistant, the user can confirm the configuration proposed by the automated assistant and / or request a different configuration of the peripheral assistant control device. For example, the user can provide natural language input, such as, "Assistant, when I am in the living room, have the interface device control the television, but when I am not in the living room, have the interface device control the lights in the living room" ("interface device" can refer to a peripheral assistant control device). In response to receiving natural language input from the user, the automated assistant can cause the peripheral assistant control device to be conditionally assigned control of the identified function. Specifically, in order to assign control of the lights to the peripheral assistant control device, the automated assistant should determine that the user is not present in the living room. Additionally, in order to assign control of the TV to the peripheral assistant control device, the automated assistant should determine that the user is actually present in the living room. This determination can be based on data accessible to existing client devices, automated assistants, and / or any other related devices, as discussed herein.

[0019] The above description is provided as an overview of some embodiments of the present disclosure. Further descriptions of these embodiments and other embodiments will be described in more detail below.

[0020] In some embodiments, a method implemented by one or more processors includes the following operations, such as determining whether a first user or a second user is a currently active user of a peripheral assistant control device based on data accessible via a computing device, wherein the peripheral assistant control device wirelessly communicates with the computing device and the automatic assistant is accessible via the computing device. The operation may also include receiving other data generated in response to receiving touch input from a touch interface of the peripheral assistant control device from the peripheral assistant control device, wherein the reception of other data generated in response to interaction with the touch interface causes the automatic assistant to perform a function selected from at least a first automatic assistant function and a second automatic assistant function according to whether the first user or the second user is determined to be the currently active user of the peripheral assistant control device. The operation may also include: when the first user is determined to be the currently active user, in response to receiving the other data and determining that the first user is the currently active user, causing the automatic assistant to perform the first automatic assistant function. The operation may also include: when the second user is determined to be the currently active user, in response to receiving the other data and determining that the second user is the currently active user, causing the automatic assistant to perform a second automatic assistant function different from the first automatic assistant function.

[0021] In some embodiments, the first automatic assistant function may also be executed in response to a first verbal request corresponding to the automatic assistant interface of the computing device, and the second automatic assistant function may also be executed in response to a second verbal request corresponding to the automatic assistant interface of the computing device. In some embodiments, the method may also include, before determining the current active user of the peripheral assistant control device: in response to the first user and / or the second user providing one or more verbal utterances for requesting the automatic assistant to assign the first automatic assistant function and the second automatic assistant function to the peripheral assistant control device, assigning the first automatic assistant function and the second automatic assistant function to the peripheral assistant control device. In some embodiments, the method may also include, before determining the current active user of the peripheral assistant control device: assigning the first automatic assistant function to the peripheral assistant control device based on determining that the first verbal utterance requests to assign the first automatic assistant function to the peripheral assistant control device, and based on determining that the first verbal utterance is provided by the first user, assigning the first automatic assistant function to the peripheral assistant control device and to the first user. In some embodiments, the method may also include assigning the first automatic assistant function to the peripheral assistant control device based on determining that the second verbal utterance requests to assign the first automatic assistant function to the peripheral assistant control device, and based on determining that the second verbal utterance is provided by the second user, assigning the second automatic assistant function to the peripheral assistant control device and to the second user. In some embodiments, the touch interface of the peripheral assistant control device includes a touch sensor that responds to tactile gestures. In some embodiments, the peripheral assistant controls the device to provide other data only in response to interactions with the touch interface, and the touch interface responds only to one or more tactile gestures.

[0022] In some embodiments, the data accessible via the computing device includes audio data detected via one or more microphones of the computing device before receiving other data from the peripheral assistant control device; and wherein determining whether the first user or the second user is the current active user of the peripheral assistant control device based on the data accessible via the computing device includes: determining that the audio data matches a stored voice embedding of the first user based on processing the audio data; and determining that the first user is the active user based on the audio data matching the stored voice embedding of the first user. In some embodiments, determining that the first user is the active user is also based on: determining that the audio data is detected within a threshold amount of time of receiving the other data from the peripheral assistant control device. In some embodiments, wherein the first automatic assistant function includes changing a property of the smart device, and wherein causing the automatic assistant to perform the first automatic assistant function includes: determining a current state of the property of the smart device, and generating a command to cause the property of the smart device to transition to an alternative state different from the current state based on the current state; and transmitting the command to the smart device or to a server associated with the smart device. In some embodiments, the method may further include receiving input requesting that only functions that cause properties of the smart device to transition to a current state be assigned to the peripheral assistant control device; and in response to receiving the input, automatically assigning a first automatic assistant function to the smart device, wherein the first automatic assistant function changes the properties of the smart device between at least a current state and an alternative state, rather than only causing the properties of the smart device to transition to the current state.

[0023] In yet another embodiment, a method implemented by one or more processors, the method comprising: determining whether a peripheral assistant control device is located in a first area or a second area based on data accessible via a computing device, wherein the peripheral assistant control device communicates wirelessly with the computing device, and the automatic assistant is accessible via the computing device. The method may also include receiving from the peripheral assistant control device other data generated in response to a touch interface of the peripheral assistant control device receiving a touch input, wherein the interaction with the touch interface causes the automatic assistant to perform a function selected from at least a first automatic assistant function and a second automatic assistant function, depending on whether the peripheral assistant control device is determined to be located in the first area or the second area. The method may also include, when the peripheral assistant control device is determined to be in the first area: in response to receiving other data and determining that the peripheral assistant control device is determined to be in the first area, causing the automatic assistant to perform the first automatic assistant function. The method may also include, when the peripheral assistant control device is determined to be in the second area: in response to receiving other data and determining that the peripheral assistant control device is determined to be in the second area, causing the automatic assistant to perform a second automatic assistant function different from the first automatic assistant function.

[0024] In some embodiments, the peripheral assistant control device is directly paired only with the computing device via a wireless communication protocol, via data indication accessible to the computing device, and wherein determining whether the peripheral assistant control device is located in the first area or the second area includes determining that the peripheral assistant control device is located in the first area based on the paired computing device being assigned to the first area in the device topology. In some embodiments, the method may also include, before determining whether the peripheral assistant control device is located in the first area or the second area: receiving an input for requesting that control of a type of smart device be assigned to the peripheral assistant control device. In some embodiments, the method may also include generating a first automatic assistant function based on the input to cause control of one or more first smart devices of the type based on the first smart device being of the type and being assigned to the first area in the device topology. In some embodiments, the method may also include generating a second automatic assistant function based on the input to cause control of one or more second smart devices of the type based on the second smart device being of the type and being assigned to the second area in the device topology.

[0025] In some embodiments, the method may further include, before determining whether the peripheral assistant control device is located in the first area or the second area: assigning the first automatic assistant function to the first area based on a user interface input for requesting that the first automatic assistant function be assigned to the peripheral assistant control device and to the first area. In some embodiments, the method may further include, before determining whether the peripheral assistant control device is located in the first area or the second area: assigning the second automatic assistant function to the second area based on a user interface input for requesting that the second automatic assistant function be assigned to the peripheral assistant control device and to the second area. In some embodiments, the computing device is one of a plurality of different assistant computing devices defined in a device topology accessible to the automatic assistant, and wherein determining whether the peripheral assistant control device is located in the first area or the second area includes determining which of the plurality of assistant computing devices the peripheral assistant control device is paired with and / or is closest to.

[0026] In other embodiments, a method implemented by one or more processors is described as including operations such as receiving audio data corresponding to a spoken utterance provided by a user to an automatic assistant interface of a computing device, wherein the computing device provides access to the automatic assistant and is connected to a peripheral assistant control device. The method may also include determining, based on the audio data, that the spoken utterance includes a request for controlling one or more functions in response to an interaction with a touch interface of a peripheral assistant control device, wherein the peripheral assistant control device provides output to a computing device or another computing device in response to an interaction with the touch interface. The method may also include determining, based on the audio data, that the spoken utterance at least partially identifies one or more functions associated with the output of the peripheral assistant control device. The method may also include identifying, based on data associated with one or more functions, one or more other functions executable via a computing device or an automatic assistant, wherein the spoken utterance does not specify the content of one or more other functions. The method may also include generating correlation data that associates the output of the peripheral assistant control device with one or more functions when at least the first condition is met, and associating the output of the peripheral assistant control device with one or more other functions when at least the second condition is met.

[0027] In some embodiments, the method may further include, after generating the correlation data, determining that the peripheral assistant control device has provided an output in response to the user interacting with the touch interface. In some embodiments, the method may further include determining whether the first condition or the second condition is met based on other data associated with the user. In some embodiments, the method may further include, when the first condition is met: in response to the user interacting with the touch interface of the peripheral assistant control device, causing one or more functions to be executed. In some embodiments, the method may further include, when the second condition is met: in response to the user interacting with the touch interface of the peripheral assistant control device, causing one or more functions to be executed.

[0028] In some embodiments, one or more functions include a start media action for causing presentation of media, and the first condition is satisfied when the media is not presented at the computing device and / or another device associated with the user, and wherein the second condition is satisfied when the media is presented at the computing device or other device associated with the user. In some embodiments, one or more other functions include a next media item action for causing a new media item to be presented to replace a current media item currently being presented. In some embodiments, one or more functions, when executed, cause the computing device or another device to operate according to a setting, and the second condition is satisfied when the computing device or another device operates according to the setting, and wherein one or more other functions, when executed, cause the computing device or another device to operate according to another setting, and the first condition is satisfied when the computing device or another device operates according to the other setting.

[0029] In some embodiments, one or more functions, when executed, cause the computing device to operate according to the setting, and one or more other functions, when executed, cause other devices to operate according to other settings. In some embodiments, one or more functions cause the computing device to switch the operating mode from a first operating mode to a second operating mode, and one or more other functions cause the computing device to switch the operating mode from the second operating mode to the first operating mode. In some embodiments, the first operating mode causes the computing device to provide a first output via a modality of the computing device, and the second operating mode causes the computing device to provide a second output different from the first output via a modality of the computing device. In some embodiments, the first condition includes that the attribute of the smart device is in a first state, and wherein the second condition includes that the attribute of the smart device is in a second state. In some embodiments, one or more functions include causing the attribute of the smart device to switch from a first state to a second state, and wherein one or more other functions include causing the attribute of the smart device to switch from a second state to a first state. In some embodiments, the method may further include determining that the peripheral assistant control device has provided an output in response to the user interacting with the touch interface; in some embodiments, the method may further include determining whether it is the first condition or the second condition in response to the output being provided and based on the device topology used to define the current state of the attribute of the smart device.

[0030] Other embodiments may include a non-transitory computer-readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform methods such as one or more of the methods described above and / or elsewhere herein. Other embodiments may include a system having one or more computers and / or one or more robots including one or more processors operable to execute the stored instructions to perform methods such as one or more of the methods described above and / or elsewhere herein.

[0031] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1A and Figure 1B A view is shown of a peripheral assistant control device having functions separately assigned according to whether a specific user is determined to be present in an area including the peripheral assistant control device.

[0033] Figure 2A and Figure 2B A diagram illustrating assigning control of one or more functions to a peripheral assistant control device based on the location of the peripheral assistant control device.

[0034] Figure 3 A system is shown for assigning control of one or more executable assistant functions to a peripheral assistant control device based on whether one or more different conditions are met.

[0035] Figure 4A and Figure 4B A method for assigning control of specific functions to a peripheral assistant control device based on whether one or more conditions are met is shown.

[0036] Figure 5 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0037] Figure 1A and Figure 1B The views 100 and 130 of the peripheral assistant control device 112 are shown to have functions separately assigned according to whether a specific user is determined to be present in the area including the peripheral assistant control device 112. Specifically, Figure 1AThe view 100 shows that the peripheral assistant control device 112 is assigned a first assistant function when the first user 106 is determined to be present at the location of the peripheral assistant control device 112 and / or is otherwise determined to be close to the peripheral assistant control device 112. For example, when the first user 106 is determined to be located within an area 128 of their residence 126 (e.g., the kitchen of the residence), the peripheral assistant control device 112 can reassign control of the first assistant function (e.g., turning on / off the kitchen lights). However, when the user is determined to have been relocated away from the area 128 and / or the residence 126 (e.g., Figure 1B ), and / or when the second user 108 is determined to be present within the area 128 of the residence 126 and / or otherwise close to the peripheral assistant control device 112, the peripheral assistant control device 112 can be assigned to control the second assistant function (e.g., playing music at the first client device 110).

[0038] For example, the first user 106 may provide a request to the application accessible via the computing device for requesting the peripheral assistant control device 112 to control the lights in their kitchen when the first user 106 specifically interacts with the peripheral assistant control device 112. In addition, the second user 108 may also use the application to request the peripheral assistant control device 112 to control the music presented by the first client device 110 when the second user 108 specifically interacts with the peripheral assistant control device 112. Thereafter, when the first user 106 is determined to be present and interacting with the peripheral assistant control device 112, the peripheral assistant control device 112 may be assigned control of the lights in the kitchen. Therefore, any interaction between the first user 106 and the peripheral assistant control device 112 at this time will result in the operation of the kitchen lights being modified. In addition, when the first user 106 no longer exists, but the second user 108 exists and interacts with the peripheral assistant control device 112, any interaction between the second user 108 and the peripheral assistant control device 112 will result in causing the first client device 110 to present music based on the interaction.

[0039] In some embodiments, the server device 120 and / or the client device may deem the first user 106 or the second user 108 to be "active" in terms of their assigned ability to affect the control of the peripheral assistant control device 112. Additionally, whether a particular user is "active" with respect to the peripheral assistant control device 112 may be based on the voice identification of the first user 106 and / or the second user 108. For example, and with prior permission from the first user 106, the first client device 110 and / or the server device 120 may perform voice processing to determine whether the first user 106 has provided a recent spoken utterance to the first client device 110. If the first user 106 has been identified as a user who has recently provided a spoken utterance based on the voice characteristics of the first user 106, the peripheral assistant control device 112 may be assigned control of one or more specific functions preferred by the first user 106 (e.g., controlling the kitchen lights). Additionally, the second user 108 may then interact with the first client device 110 by providing a spoken utterance. The first client device 110 and / or the server device 120 can identify the second user 108 as the user providing the subsequent spoken utterance and cause the peripheral assistant control device 112 to be assigned control of one or more other specific functions preferred by the second user 108 (e.g., controlling music presented by the first client device 110).

[0040] In some embodiments, when the first user 106 is considered to be "active" with respect to the peripheral assistant control device 112, the "active" time of the first user 106 can expire after a period of time and / or in response to the automated assistant confirming that another user is present or that the first user 106 has left the area 128. For example, after the first user 106 has interacted with the peripheral assistant control device 112 and the peripheral assistant control device 112 is assigned control of the functions preferred by the first user 106, the second user 108 can be determined to be present and / or interacting with the peripheral assistant control device 112. In response, the "active" status of the first user 106 with respect to the peripheral assistant control device 112 can expire, and the peripheral assistant control device 112 can be assigned control of one or more other functions preferred by the second user 108 (e.g., playing music). Additionally or alternatively, when the second user 108 has stopped interacting with the peripheral assistant control device 112 for a threshold period of time, the first client device 110 and / or the server device 120 can reallocate control of the peripheral assistant control device 112 to control one or more functions preferred by the first user 106. In some embodiments, the peripheral assistant control device 112 can transition back to having control of the functions preferred by the first user 106 in response to the first user 106 responding to a query from the first client device 110 (such as “First User, are you there?”) or the second user 108 not responding to a query from the first client device (such as “Second User, are you there?”) after a threshold period of time.

[0041] In some embodiments, the peripheral assistant control device 112 may include one or more lights that are illuminated to indicate the user who is "active" with respect to the peripheral assistant control device 112. For example, when the peripheral assistant control device 112 is assigned control of one or more functions preferred by the first user 106, the one or more lights of the peripheral assistant control device 112 may be caused to emit a first color (e.g., green). Additionally, when the peripheral assistant control device 112 is assigned control of one or more other functions preferred by the second user 108, the one or more lights of the peripheral assistant control device 112 may be caused to emit a second color (e.g., red) that is different from the first color.

[0042] The peripheral assistant control device 112 can be operated as a computing device having one or more touch sensors that respond to one or more tactile gestures. For example, the peripheral assistant control device 112 may include one or more touch-sensitive sensors that can provide signals in response to one or more different touch gestures provided by one or more users. The peripheral assistant control device 112 can be connected to one or more client devices and / or server devices to communicate and transmit signals generated in response to touch input. For example, the peripheral assistant control device 112 can communicate with the first client device 110 and / or the second client device 114 via a communication protocol (such as Bluetooth, Wi-Fi, LTE and / or any other wired or wireless communication protocol). In response to the user providing a touch gesture to the peripheral assistant control device 112, the peripheral assistant control device 112 can transmit a signal to the first client device 110 and / or the second client device 114. In some embodiments, the first client device 110 and / or the second client device 114 can communicate with a remote computing device 118 (e.g., a server device 120) via a network 116 (such as the Internet). The first client device 110 and / or the second client device 114 may communicate to the server device 120 data indicating that the user has provided touch input to the peripheral assistant control device 112. In this manner, various devices and / or applications (such as the automated assistant 122) may be controlled by the peripheral assistant control device 112.

[0043] In some embodiments, the peripheral assistant control device 112 can respond only to direct touch input from the user, and signals transmitted to and from the peripheral assistant control device 112 from one or more devices. In this way, the peripheral assistant control device 112 can include limited hardware (e.g., relative to a laptop computer and / or a cellular phone), thereby reducing power consumption and network latency that would otherwise be exhibited by a computing device that responds only to direct touch input from the user or does not have limited hardware. In some embodiments, the peripheral assistant control device 112 can respond to a variety of different touch gestures, such as single clicks, double clicks, other multiple clicks, long presses, and / or any variation of touch input. Depending on the environment or scenario in which the user provides a particular touch gesture, the client device and / or server device can respond differently to each of the touch gestures.

[0044] like Figure 1AAs shown, the first user 106 and the second user 108 may be located in an area of ​​a residence 126 where the users reside, and the area may include a peripheral assistant control device 112. Specifically, the first user 106 may exhibit a condition 102 of walking in a kitchen area, and the second user 108 may exhibit a condition 104 of also walking in the kitchen area. The peripheral assistant control device 112 may be connected to the first client device 110, the second client device 114, and / or the server device 120. By using the previous permission from the first user 106, the first client device 110, the second client device 114, and / or the server device 120 may determine the presence of the first user 106. Specifically, the presence of the first user 106 can be characterized by data indicating whether the first user 106 is within the residence 126, within the area 128, close to the peripheral assistant control device 112, and / or otherwise within a distance of the peripheral assistant control device 112 (the distance allows the first user 106 to interact with the peripheral assistant control device 112 without having to manipulate at least a threshold distance and / or manipulate for at least a threshold amount of time before reaching the peripheral assistant control device).

[0045] When the presence of the first user 106 is determined and / or otherwise confirmed, the first client device 110, the second client device 114, and / or the server device 118 may correlate one or more signals provided by the peripheral assistant control device 112 with specific functions. For example, the correlation between one or more specific functions and one or more specific peripheral assistant control device outputs may be characterized by the correlation data 124. Specifically, and by way of example, the correlation data 124 may characterize the relationship between the presence of the first user 106 and the control of one or more functions via the peripheral assistant control device 112. When it is determined that the first user 106 is present in the vicinity of the peripheral assistant control device 112, the correlation data 124 may identify a first assistant function of the one or more functions that may be controlled via the touch interface of the peripheral assistant control device 112.

[0046] Additionally, control of one or more other functions may be associated with a determination that the first user 106 is not present. Figure 1BAs shown, the peripheral assistant control device 112, the first client device 110, the second client device 114, and / or the server device 120 may determine that the first user is no longer present at or near the peripheral assistant control device 112. In contrast, the first user 106 may exhibit a condition 132 of being relocated from the area 128. In response to determining that the first user 106 is no longer present at or near the peripheral assistant control device 112, the data provided by the peripheral assistant control device 112 in response to the second user 108 interacting with the peripheral assistant control device 112 may be related to one or more other functions. For example, when the second user 108 interacts with the peripheral assistant control device 112 and the first user 106 has been relocated from the area 128, the interaction may cause one or more other functions to be executed or otherwise controlled. In some embodiments, one or more other functions may include a second automatic assistant function that is executed or controlled when the second user 108 provides an input gesture to the peripheral assistant control device 112 when the first user 106 is determined not to be present at or near the peripheral assistant control device 112.

[0047] In some embodiments, the allocation of control of one or more functions may be based on a specific area in which the first user 106 and the second user 108 are determined to be present. For example, when the first user 106 is determined by the second client device 114 to be present in the living room, the peripheral assistant control device 112 may be assigned control of the first function. In addition, when the first user 106 is determined by the first client device 110 to be present in the kitchen, the peripheral assistant control device 112 may be assigned control of a second function different from the first function. In some embodiments, when no specific user is detected, and / or an unidentified user is detected, the peripheral assistant control device 112 may be assigned control of one or more specific functions. In some embodiments, when a combination of different users is detected, the peripheral assistant control device 112 may be assigned control of one or more specific functions, which may be different from one or more other functions to which the peripheral assistant control device 112 will be assigned control if the presence of the first user 106 is detected and / or the presence of the second user 108 is detected.

[0048] In some implementations, one or more functions to which the peripheral assistant control device 112 may be assigned control may be automated assistant functions. In other words, the one or more functions may be controlled by or performed via an automated assistant accessible via the first client device 110, the second client device 114, and / or the server device 120. Thus, the one or more functions may be controlled and / or performed via spoken utterances provided to the automated assistant interface of the first client device 110, and also by providing input gestures (e.g., tap gestures) to the touch interface of the peripheral assistant control device 112.

[0049] Figure 2A and Figure 2B The views 200 and 230 respectively show that control of one or more functions is assigned to the peripheral assistant control device 212 based on the location of the peripheral assistant control device 212. For example, when the peripheral assistant control device 212 is in the first area 228 (such as the kitchen), the peripheral assistant control device 212 can be assigned control of one or more functions. In addition, when the peripheral assistant control device 212 is in the second area 208 (such as the living room), the peripheral assistant control device 212 can be assigned control of one or more other functions.

[0050] The determination of whether the peripheral assistant control device 212 is in the first area 228 or in the second area 208 can be based on data characterizing a device topology of devices associated with the user 206. For example, the device topology data can be accessed at a remote computing device 218 (such as a server device 220), which can be accessed via a network 216 (such as the Internet). The device topology data can include identifiers of the first client device 210 and the second client device 214. In addition, each device identifier can be associated with a region identifier corresponding to a geographic area accessible to the user 206. For example, the first identifier of the first client device 210 can be associated with the first region identifier of the first area 228. Additionally, the second identifier of the second client device 214 can be associated with the second region identifier of the second area 208.

[0051] The server device 220 may include correlation data 224 that may identify various correspondences between different areas accessible to the user 206 and different functions that may be controlled by the peripheral assistant control device 212. In addition, an assumption may be made that when the peripheral assistant control device 212 is connected to the first client device 210, the peripheral assistant control device 212 is located within the first area 228. For example, such an assumption may be based on the first client device 210 being assigned to the first area 228 in the device topology representation. Moreover, another assumption may be made that when the peripheral assistant control device 212 is connected to the second client device 214, the peripheral assistant control device 212 is located in the second area 208. For example, such an assumption may be based on the second client device 214 being assigned to the second area 208 in the device topology representation. Therefore, when the peripheral assistant control device 212 is paired or otherwise connected to the first client device 210, any data provided by the peripheral assistant control device 212 (such as data generated in response to touch input from the user 206) may be processed based on the assumption that the peripheral assistant control device 212 is located in the first area 228. Additionally, when the peripheral assistant control device 212 is paired with or otherwise connected to the second client device 214, any data provided by the peripheral assistant control device 212 can be processed based on the assumption that the peripheral assistant control device 212 is located in the second area 208.

[0052] In one non-limiting example, the user 206 can provide a spoken utterance to the automated assistant interface (such as a microphone of the first client device 210) to request that the peripheral assistant control device 212 be assigned to different functions based on certain conditions. The spoken utterance can be "Assistant, when my peripheral assistant control device is in the kitchen, please provide the news when I perform a touch input". In addition, in the same spoken utterance or a different spoken utterance, the user 206 can provide a request such as "Assistant, also, when the peripheral assistant control device is in the living room, please control the television when I perform a touch input". In response to receiving the spoken utterance, the first client device 210 can generate audio data that can be transmitted to the server device 220, which can process the audio data via the automated assistant 222 application at the server device 220. The audio data can be processed to determine the configuration of certain settings for the peripheral assistant control device 212. For example, the server device 220 and / or any other device discussed herein may determine that the user 206 is requesting that the peripheral assistant control device 212 control a first function (e.g., Figure 2A ), and controls a second function (such as Figure 2B 230).

[0053] Additionally or alternatively, the user 206 may provide a spoken utterance, such as "Assistant, please control the lights in the room that I am in," and / or otherwise provide input (e.g., text input) with the same meaning. In response to receiving the spoken utterance, the first client device 210 may generate audio data that may be transmitted to the server device 220, which may process the audio data via the automated assistant 222 application at the server device 220. The audio data may be processed to determine a configuration for certain settings of the peripheral assistant control device 212. For example, the server device 220 and / or any other device discussed herein may identify another device located in the current vicinity or current area of ​​the user 206. Based on data indicating the location of the other device, settings may be generated such that when the peripheral assistant control device 212 is paired or otherwise communicates with the other device, the peripheral assistant control device 212 will be assigned control of the lights in that location. Additionally, when the peripheral assistant control device 212 is paired with or otherwise communicating with another device, the device topology representation relating the location of the other device to the light can be used as a basis for assigning control of the light to the peripheral assistant control device 212.

[0054] In some embodiments, the determination of where the peripheral assistant control device 212 is located can be based on device topology data. Alternatively or additionally, the determination of where the peripheral assistant control device 212 is located can be based on the output of one or more sensors and / or one or more transmitters of the peripheral assistant control device 212, the first client device 210, and / or the second client device 214. For example, the signal strength of a local area network (e.g., a Wi-Fi network) available to the peripheral assistant control device 212 can be used as an indication of where the peripheral assistant control device 212 is located within the residence 226. In addition, when the device topology data provides an indication of the location of a local area network signal source, the signal strength of the local area network can be used in conjunction with the device topology data to determine where the peripheral assistant control device 212 is located. Optionally, the location of the peripheral assistant control device 212 can be based on a comparison between the signal strengths at the first client device 210, the second client device 214, and the peripheral assistant control device 212, respectively, in combination with the known location of the local area network signal source.

[0055] In some embodiments, the determination of where the peripheral assistant control device 212 is located can be based on the output of one or more sensors and / or one or more transmitters, such as one or more such devices included in a global positioning system (GPS) device. For example, the peripheral assistant control device 212 can include a GPS device that can provide an output indicating the location of the peripheral assistant control device 212. Therefore, when the user provides touch input to the peripheral assistant control device 212, the peripheral assistant control device 212 can communicate output data that identifies the location of the peripheral assistant control device according to the GPS device and identifies the type of touch input received from the user 206. The device receiving the output data can compare the indicated location of the peripheral assistant control device 212 with the correlation data 224 to identify one or more functions that the user 206 intends to control via the touch input at the peripheral assistant control device 212 when the peripheral assistant control device 212 is in the indicated location. In other words, in some embodiments, data used to characterize touch input from user 206 can be the same regardless of where peripheral assistant control device 212 is located, however, the data can be responded to differently by another device (such as server device 220) depending on where peripheral assistant control device 212 is located.

[0056] Figure 3 A system 300 for assigning control of one or more executable assistant functions to a peripheral assistant control device based on whether one or more different conditions are met is shown. An automated assistant 306 can operate as part of an automated assistant application provided at one or more computing devices (e.g., a client device 334 and / or a server device 302). A user can interact with an automated assistant 308 via one or more assistant interfaces 336, which can include one or more of a microphone, a camera, a touch screen display, a user interface, and / or any other device capable of providing an interface between a user and an application. For example, a user can initialize the automated assistant 308 by providing verbal, textual, or graphical input to the assistant interface to cause the automated assistant 308 to perform a function (e.g., provide data, control a peripheral device, access an agent, etc.). The client device 334 and / or the peripheral assistant control device 342 can include a display device, which can be a display panel including a touch interface for receiving touch input and / or gestures to allow a user to control the client device 334 and / or the application of the server device 302 via the touch interface.

[0057] In some embodiments, the client device 334 and / or the peripheral assistant control device 342 may not have a display device, but include an audio interface (e.g., a speaker and / or a microphone) to provide audible user interface output without providing a graphical user interface output, and provide a user interface (such as a microphone) for receiving spoken natural language input from a user. For example, in some embodiments, the peripheral assistant control device 342 may include one or more tactile input interfaces (such as one or more buttons) and omit a display panel that provides graphics data from a graphics processing unit (GPU). In this way, a large amount of energy and processing resources can be saved compared to a computing device that includes a display panel and a GPU.

[0058] The client device 334 can communicate with the server device 302 via a network 346, such as the Internet. The client device 334 can offload computing tasks to the server device 302 in order to preserve computing resources at the client device 334 and / or the peripheral assistant control device 342. For example, the server device 302 can host the automated assistant 308, and the client device 334 can transmit input received at the user interface 344 of one or more assistant interfaces and / or the peripheral assistant control device 342 to the server device 302. However, in some embodiments, the automated assistant 308 can be hosted at the client device 334. In various embodiments, all or less than all aspects of the automated assistant 308 can be implemented on the server computing device 304 and / or the client device 334. In some of these embodiments, various aspects of the automated assistant 308 are implemented via a local automated assistant 338 of the client device 334 and interfaced with the server device 302, which can implement other aspects of the automated assistant 308. The server device 302 can optionally serve multiple users and their associated assistant applications via multithreading. In embodiments where all or less than all aspects of automated assistant 308 are implemented via a local automated assistant 338 of client device 334, local automated assistant 338 may be an application separate from the operating system of client device 334 (e.g., installed “on top of” the operating system)—or may alternatively be implemented directly by the operating system of client device 334 (e.g., considered an application within the operating system, but integral to the operating system).

[0059] In some implementations, automated assistant 308 and / or automated assistant 338 may include an input processing engine 312 that may employ a number of different modules to process input and / or output for client device 334. For example, input processing engine 312 may include a speech processing module 314 that may process audio data received at assistant interface 336 to identify text contained in the audio data. The audio data may be transmitted from, for example, client device 334 to server device 302 in order to preserve computing resources at client device 334.

[0060] The process for converting audio data into text can include a speech recognition algorithm, which can use a neural network, a word2vec algorithm and / or a statistical model to identify an audio data group corresponding to a word or phrase. The text converted from the audio data can be parsed by a data parsing module 316, and made as text data that can be used by an automatic assistant 308, and the text data can be used to generate and / or identify a command phrase from a user. In some embodiments, the output data provided by the data parsing module 316 can be provided to an action engine 318 to determine whether the user has provided an input corresponding to a specific action and / or routine performed by an application or agent that can be accessed by an automatic assistant 308 and / or an automatic assistant 308. For example, assistant data 322 can be stored as client data 340 at a server device 302 and / or a client device 334, and can include data for defining one or more actions that can be performed by an automatic assistant 308, and parameters involved in performing these actions.

[0061] When input processing engine 312 has determined that the user has requested that a particular action or routine be performed, action engine 318 can determine one or more parameters for the particular action or routine, and output generation engine 320 can then provide output to the user based on the particular action, routine, and / or one or more parameters. For example, in some embodiments, in response to user input (such as a gesture at user interface 344 of peripheral assistant control device 342), automated assistant 338 can cause data representing the gesture to be transmitted to server device 302 for use in determining the action that the user wants automated assistant 308 and / or automated assistant 338 to perform.

[0062] The system 300 may include one or more peripheral assistant control devices 342, and each peripheral assistant control device 342 may include one or more user interfaces 344. The user interface 344 may include one or more sensors capable of providing output in response to user input from a user. For example, one or more sensors may include a tactile sensor (i.e., a touch sensor) that provides an output signal in response to a tactile gesture from a user. The output signal may be transmitted to the client device 334 and / or the server device 302 via a communication protocol (such as Bluetooth, LTE, Wi-Fi, and / or any other communication protocol). In some embodiments, the output signal from one or more sensors of the user interface 344 may be converted into user input data by one or more processors of the peripheral assistant control device 342. The user input data may be transmitted to the client device 334 via a communication protocol, and either processed at the client device 334 and / or transmitted to the server device 302 via a network 346. For example, the client device 334 and / or the server device 302 may determine the function to be controlled by the peripheral assistant control device 342 based on the user input data.

[0063] In some limitations, determining the function to be controlled by the peripheral assistant control device 342 can be based on the identification of the client device 334 with which the peripheral assistant control device 342 is paired or otherwise communicated. In other words, the client device 334 and / or the server device 302 will determine the different functions to be controlled by the peripheral assistant control device 342 according to the specific client device to which the peripheral assistant control device 342 is connected. In some embodiments, assigning control of a specific function to the peripheral assistant control device 342 can be based on the physical characteristics of the client device 334. For example, when the peripheral assistant control device 342 is paired with a device including a touch display panel, the peripheral assistant control device 342 can be assigned control of a first function. Additionally, when the peripheral assistant control device 342 is paired with a device that does not include a touch display panel, the peripheral assistant control device 342 can be assigned control of a second function different from the first function. Alternatively or additionally, when the peripheral assistant control device 342 is paired with a device identified by the account of the first user, the peripheral assistant control device 342 can be assigned control of the first function. In addition, when the peripheral assistant control device 342 is paired with a different device identified by a different account of a second user different from the first user, the peripheral assistant control device 342 can be assigned control of the second function.

[0064] In some embodiments, a request from a user to assign control of a function to the peripheral assistant control device 342 can be extended so that the peripheral assistant control device 342 can have control over other functions that are not explicitly identified in the request from the user. For example, a user can provide a spoken utterance such as "Assistant, go to the next song in my playlist when I tap the peripheral assistant control device." The spoken utterance can be received at the assistant interface 336, converted into audio data by the client device 334, and transmitted to the server device 302 for additional processing. By using the audio data, the automated assistant 308 can determine that the user is requesting that the peripheral assistant control device 342 have control over the function of skipping a song that is being played from a playlist.

[0065] Thereafter, when the user is listening to music from a playlist and presses the user interface 344 of the peripheral assistant control device 342, the automated assistant 308 can cause the playback of the playlist to switch from the current song to the next song in the playlist. In some embodiments, the peripheral assistant control device 342 can provide data in response to receiving a touch input at the user interface 344. The data can be transmitted to the client device 334 and then to the server device 302 for additional processing. The server device 302 may include a control device module 324 that can determine how the automated assistant 308 and / or any other application associated with the server device 302 and / or the client device 334 will respond to the user providing input to the peripheral assistant control device 342.

[0066] The control device module 324 may include a gesture processing engine 326 that receives data from the peripheral assistant control device 342 in response to input from the user and determines the type of gesture provided by the user. For example, the data may be processed by the gesture processing engine 326 to determine whether the input gesture is a single click, double click, multiple clicks, long clicks, hard presses, soft presses, and / or any other distinguishable gesture that may be provided to an interface of the computing device. In some embodiments, the control device module 324 may access and / or receive a certain amount of client data 340 that indicates that one or more users are present in an area associated with the peripheral assistant control device 342 and / or the client device 334. In this manner, the control device module 324 may determine the function that the output at the peripheral assistant control device 342 will affect based on the received touch input and the one or more users determined to be present. For example, the control device module 324 may include a function selection engine 330 that may access client data 340 for indicating the presence of one or more users, and may also access correlation data 332 that may provide a correspondence between functions to be controlled by the peripheral assistant control device 342 and users indicated as being present when the user interface 344 receives the input. In other words, the correlation data 332 will characterize settings for controlling functions requested by a user via the user interface 344 based on whether the user is present. Thus, if such settings are characterized by the correlation data 332, and the client data 240 indicates that the user is present when the user interface 344 receives the input, the corresponding function identified by the correlation data 332 may be controlled in response to the input.

[0067] Control of a function may be defined as executing a function, modifying one or more instances of a function executed by one or more devices, modifying one or more instances of a function executed by one or more applications, and / or any other control of a function and / or the execution of a function that may be exhibited. In some embodiments, control of a function may be defined as providing values ​​for parameters used for a function, modifying values ​​for parameters used for a function, and / or otherwise modifying data used when executing a function. For example, a function to skip songs in a playlist may include a parameter for providing a device name. Thus, when control of a function is assigned to the peripheral assistant control device 342, the control device module 324 may cause execution of the function in response to a touch input at the user interface 344, and also provide a device name for the parameters used for the function so that the function may be executed without error.

[0068] In some embodiments, when a user requests the peripheral assistant control device 342 to control a function to skip to the next song in a playlist, one or more other functions associated with the request from the user may be assigned to the peripheral assistant control device 342. For example, in response to processing audio data corresponding to a spoken utterance, the server device 302 may identify the function to which the user requests the peripheral assistant control device 342 to be assigned control. The function may be associated with one or more applications and / or one or more devices, and each application and / or device may exhibit one or more different operating states (i.e., operating modes) at a given time. For example, when an application that performs playback of songs in a playlist causes a particular computing device to cast music from the playlist, the application may be in a "playback" state. The control device module 324 may identify the state and associate the state with a "skip" function that the user is requesting the peripheral assistant control device 342 to have control of via the correlation data 332. In other words, the control device module 324 will cause the peripheral assistant control device 342 to operate so that when the application is in the “playback” state, the peripheral assistant control device 342 will execute the “skip” function in response to the user providing a single-click gesture at the user interface 344 .

[0069] Alternatively or additionally, the function selection engine 330 of the control device module 324 can identify other functions associated with the application and / or other states of the application. For example, the application may have a "play" function and / or an "open" function, and the application may also have a "closed" state and an "open" and / or "play" state. Based on this information, the control device module 324 can generate correlation data 332, which correlates the open function of the application with the closed state and the skip function of the application with the play state. As a result, and in response to a request from the user, providing a touch input to the user interface 344 of the peripheral assistant control device 242 when the application is in the closed state will cause the application to start playing music. In addition, providing a touch input when the application is in the play state will cause the application to skip the song currently being played. In this way, although the user only requests the peripheral assistant control device 242 to control a single function, the peripheral assistant control device 342 will be assigned multiple functions to be controlled. This can eliminate the need for the user to provide additional verbal utterances to request the peripheral assistant control device 342 to control multiple other functions. This can save computing resources and / or network resources because audio data corresponding to the additional spoken utterances need not be generated and / or transmitted from the client device 334 to the server device 302. Instead, the peripheral assistant control device 342 can be assigned control of a plurality of different functions that are not identifiable by spoken utterances from the user or that were not specifically requested by the user.

[0070] In some embodiments, the control device module 324 can generate correlation data 332, which correlates one or more functions to be controlled by the peripheral assistant control device 342 with one or more conditions. For example, the function selection engine 330 can assign control of a specific function based on whether a specific condition is met. The control device module 324 can determine whether the condition is met based on the assistant data 322, the client data 340, and / or any other data that can be transmitted to the client device 334 and / or the server device 302. For example, when the first user is determined to be located in a definable area including a peripheral assistant control device, a specific condition can be met. Therefore, when the first user is determined to be within a definable area, the function selection engine 330 can assign control of the function to the peripheral assistant control device 342 based on meeting a specific condition. Alternatively or additionally, when the peripheral assistant control device 342 is determined to be located in another definable area, another condition can be met. Therefore, when the peripheral assistant control device 342 is determined to be within another definable area, the function selection engine 330 can assign control of another function to the peripheral assistant control device 342 based on meeting another condition.

[0071] Figure 4A and Figure 4B Methods 400 and 412 are shown for assigning control of a particular function to a peripheral assistant control device based on whether one or more conditions are met. In some embodiments, method 400 can be based on Figure 4A The operations identified in are performed, which may optionally include Figure 4B In some embodiments, method 412 may be performed according to Figure 4B The operations identified in are performed, which may optionally include Figure 4A Each of method 400 and method 412 may be performed by one or more devices, one or more applications, and / or any other means or modules capable of operating according to a computer process.

[0072] Method 400 may include an operation 402 of receiving audio data corresponding to a spoken utterance provided by a user (or other data provided as input to an application via a modality such as a keyboard, a mouse, a touch display, and / or any other means for providing input to a computing device). The spoken utterance (or other data) may be a request for an automated assistant to assign control of a first automated assistant function and a second automated assistant function to a peripheral assistant control device. For example, the spoken utterance may be “Assistant, give me the news when I press the button and play music when my husband presses the button”. Alternatively, the spoken utterance may be multiple spoken utterances provided by different users. For example, a first user in a household may provide a spoken utterance such as “Assistant, play music when I press the button”, while a second user in the household may provide another spoken utterance such as “Assistant, when I press the button, give me the news”.

[0073] Method 400 may also include operation 404: determining that the request is a request from the user to set the peripheral assistant control device to perform one or more functions. For example, audio data generated based on the spoken utterance can be processed to identify an identifier and one or more functions of the peripheral assistant control device. Based at least on the content of the audio data, it can be determined that the user is requesting that the peripheral assistant control device be assigned control of one or more functions. Method 400 may also include operation 406: identifying one or more functions requested by the user based on the audio data. For example, one or more functions may include a play message function and / or a play music function. In some embodiments, the automated assistant may be associated with multiple different functions that can be performed by the automated assistant and / or via a separate application or a separate assistant agent.

[0074] Method 400 may further include operation 408: identifying one or more other functions associated with one or more functions requested by the user. One or more other functions may be identified based on any data characterizing an association or correspondence between one or more functions and one or more other functions. For example, when one or more functions correspond to a specific application, other functions associated with the application may be identified via operation 408. Alternatively or additionally, when one or more functions correspond to a specific device, other functions associated with the device may be identified via operation 408. Alternatively or additionally, when a specific user has previously used one or more functions and has also used one or more other functions at least as frequently as the one or more functions, one or more other functions may be identified via operation 408. In this way, although the user does not explicitly identify one or more other functions in their spoken utterances, the other functions may be identified to assign control of those other functions to the peripheral assistant control device.

[0075] Method 400 also includes operation 410: generating correlation data that correlates the output of the peripheral assistant control device with one or more functions when the first condition is met, and correlates the output of the peripheral assistant control device with one or more other functions when the second condition is met. In some embodiments, the user can characterize the first condition and / or the second condition via verbal speech so that the user can have exclusive control over when the peripheral assistant control device will control a specific function in the one or more functions. Alternatively or additionally, one or more conditions can be identified without being explicitly requested by the user so as to adapt the peripheral assistant control device to various scenarios in which the automatic assistant predicts or expects that the peripheral assistant control device will be used. For example, when the verbal speech includes a request from the user to request the peripheral assistant control device to control in their kitchen, receiving the verbal speech can characterize various conditions based on other areas within the user's residence. Specifically, correlation data characterizing a first condition can be generated, in which the peripheral assistant control device controls the kitchen light based on the peripheral assistant control device being in the kitchen. Additionally, the correlation data can also characterize a second condition, in which the peripheral assistant control device controls the living room light based on the peripheral assistant control device being in the living room.

[0076] Method 400 may proceed from operation 410 to operation 414, as in Figure 4A and Figure 4BOperation 414 includes determining whether touch input has been received at the peripheral assistant control device. Operation 414 can be determined periodically until the user provides touch input to the peripheral assistant control device. The touch input can be one or more physical gestures, such as a click, a double-click, multiple clicks, a long press, a pull, a rhythm of presses, and / or any other physical gesture that can be received at the user interface. In response to determining that touch input is received at the peripheral assistant control device, method 412 can proceed to operation 416.

[0077] Operation 416 may include determining whether the first condition or the second condition has been satisfied. Each of the first condition and the second condition may be any condition that can be characterized by data stored at the computing device. For example, the first condition may be satisfied when the user is determined to be located within an area including the peripheral assistant control device, and the second condition may be satisfied when the second user is determined to be located within an area including the peripheral assistant control device. Alternatively or additionally, the first condition may be satisfied when the peripheral assistant control device is determined to be located within the first area, and the second condition may be satisfied when the assistant control device is determined to be located within the second area.

[0078] When the first condition is met, method 412 can proceed to operation 418, which can include performing one or more functions in response to the user interacting with the touch interface. When the second condition is met, method 412 can proceed to operation 420, which can include performing one or more functions in response to the user interacting with the touch interface. In this way, despite receiving the same touch input-whether the first condition or the second condition is met, this same touch input can lead to different results, depending on whether the first condition is met or the second condition is met. This can reduce delays in setting up the device, which otherwise may occur if the user has to explicitly request each specific condition and each specific function through verbal speech or other interactions with the computing device. By eliminating the need for the user to provide such additional verbal speech or interaction, computing resources and network resources can be reserved for any computing device responsible for establishing settings for the peripheral assistant control device.

[0079] Figure 55 is a block diagram of an example computer system 510. The computer system 510 generally includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computer system 510. The network interface subsystem 516 provides an interface to an external network and couples to corresponding interface devices in other computer systems.

[0080] The user interface input devices 522 may include a keyboard, a pointing device (such as a mouse, trackball, touch pad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system), a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computer system 510 or onto a communication network.

[0081] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for forming a visual image. The display subsystem may also provide a non-visual display, such as through an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and methods for outputting information from the computer system 510 to a user or to another machine or computing device.

[0082] Storage subsystem 524 stores programming and data constructs for providing the functionality of some or all of the modules described herein. For example, storage subsystem 524 may include logic for performing selected aspects of method 400, method 412, and / or first client device 110, second client device 114, peripheral assistant control device 112, server device 120, automated assistant 122, first client device 210, second client device 214, peripheral assistant control device 212, server device 220, automated assistant 222, system 300, automated assistant 308, server device 302, control device module 324, client device 334, and / or peripheral assistant control device 342.

[0083] These software modules are typically executed by the processor 514 alone or in conjunction with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 may provide permanent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules for implementing the functionality of certain embodiments may be stored in the storage subsystem 524 by the file storage subsystem 526, or in other machines accessible by the processor 514.

[0084] The bus subsystem 512 provides a mechanism for the various components and subsystems of the computer system 510 to communicate with each other as intended. Although the bus subsystem 512 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0085] Computer system 510 can be of various types, including a workstation, server, computing cluster, blade server, server group, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 The description of the computer system 510 depicted in FIG. 5 is intended only as a specific example for the purpose of illustrating some embodiments. Many other configurations of the computer system 510 may have more Figure 5 The computer system depicted in FIG.

[0086] In situations where the systems described herein collect personal information about users (or generally referred to herein as "participants") or may utilize personal information, users may be provided with the opportunity to control whether a program or feature collects user information (e.g., information about the user's social network, social behavior or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from a content server. Moreover, certain data may be processed in one or more ways before it is stored or used to remove personally identifiable information. For example, the user's identity may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be generalized (such as to a city level, ZIP code level, or state level) when geographic location information is available so that the user's specific geographic location cannot be determined. Thus, a user may have control over how information about the user is collected and / or used.

[0087] Although several embodiments have been described and shown herein, various other devices and / or structures for performing functions and / or obtaining results and / or one or more of the advantages described herein may be utilized, and each of these variations and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and actual parameters, dimensions, materials, and / or configurations will depend on the specific application in which the teachings are used. Those skilled in the art will recognize, or will be able to assert many equivalents of the specific embodiments described herein using only routine experiments. Therefore, it should be understood that the aforementioned embodiments are presented only as examples, and within the scope of the appended claims and their equivalents, the embodiments may be practiced in a manner different from that specifically described and claimed. The embodiments of the present disclosure relate to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods (if such features, systems, articles, materials, kits, and / or methods are not mutually contradictory) is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors, the method comprising: determining, based on data accessible via the computing device, whether the first user or the second user is a currently active user of the peripheral assistant control device, wherein the peripheral assistant control device communicates wirelessly with the computing device, and wherein the automated assistant is accessible via the computing device; receiving other data from the peripheral assistant control device, the other data being generated in response to a touch interface of the peripheral assistant control device receiving a touch input, wherein the receipt of the other data generated in response to the interaction with the touch interface causes the automated assistant to perform a function selected from at least a first automated assistant function and a second automated assistant function based on whether the first user or the second user is determined to be the currently active user of the peripheral assistant control device; When the first user is determined as the currently active user, In response to receiving the other data and determining that the first user is the currently active user, causing the automated assistant to perform the first automated assistant function; and When the second user is determined as the current active user: In response to receiving the other data and determining that the second user is the currently active user, causing the automated assistant to perform a second automated assistant function that is different from the first automated assistant function.

2. A method according to claim 1, wherein the first automatic assistant function is also capable of being executed in response to a corresponding first verbal request to the automatic assistant interface of the computing device, and the second automatic assistant function is also capable of being executed in response to a corresponding second verbal request to the automatic assistant interface of the computing device.

3. The method according to claim 1, further comprising: Prior to determining the currently active user of the peripheral assistant controlling device: In response to the first user and / or the second user providing one or more spoken utterances for requesting the automatic assistant to assign the first automatic assistant function and the second automatic assistant function to the peripheral assistant control device, the first automatic assistant function and the second automatic assistant function are assigned to the peripheral assistant control device.

4. The method according to claim 1, further comprising: Prior to determining the currently active user of the peripheral assistant controlling device: assigning the first automated assistant functionality to the peripheral assistant control device based on determining that a first spoken utterance requests assignment of the first automated assistant functionality to the peripheral assistant control device, and assigning the first automated assistant functionality to the peripheral assistant control device and to the first user based on determining that the first spoken utterance was provided by the first user; as well as Based on determining that a second spoken utterance requests assignment of the first automatic assistant functionality to the peripheral assistant control device, and based on determining that the second spoken utterance is provided by the second user, assigning the second automatic assistant functionality to the peripheral assistant control device and to the second user.

5. The method of claim 1, wherein the touch interface of the peripheral assistant control device comprises a touch sensor that responds to tactile gestures.

6. The method of claim 5, wherein the peripheral assistant control device provides the other data only in response to interaction with the touch interface, and the touch interface is only responsive to one or more tactile gestures.

7. The method of claim 1 , wherein the data accessible via the computing device comprises audio data detected via one or more microphones of the computing device prior to receiving the other data from the peripheral assistant control device; and Wherein determining whether the first user or the second user is a currently active user of the peripheral assistant control device based on data accessible via the computing device comprises: determining, based on processing the audio data, that the audio data matches a stored speech embedding of the first user; as well as Based on matching the audio data with the stored speech embedding of the first user, determining that the first user is the active user.

8. The method of claim 7, wherein determining that the first user is the active user is further based on: Determining that the audio data is detected within a threshold amount of time of receiving the other data from the peripheral assistant control device.

9. The method according to claim 1, wherein the first automated assistant function includes changing a property of a smart device, and Wherein causing the automatic assistant to perform the first automatic assistant function comprises: determining a current state of a property of the smart device, and Based on the current state, generating a command for causing a property of the smart device to transition to an alternative state different from the current state; as well as The command is transmitted to the smart device or to a server associated with the smart device.

10. The method according to claim 9, further comprising: receiving an input requesting that only a function for changing the property of the smart device to the current state be assigned to the peripheral assistant control device; as well as In response to receiving the input, the first automated assistant function is automatically assigned to the smart device, wherein the first automated assistant function changes a property of the smart device between at least the current state and an alternate state, rather than merely transitioning the property of the smart device to the current state.

11. The method according to any one of claims 1 to 10, further comprising: determining, based on additional data accessible via the computing device, whether the peripheral assistant control device is located in a first area or a second area; When the first user is determined to be the currently active user and the peripheral assistant control device is determined to be located in the first area: In response to determining that the peripheral assistant control device is located in the first area, causing the automated assistant to perform the first automated assistant function; and When the second user is determined to be the currently active user and the peripheral assistant control device is determined to be located in the second area: In response to determining that the peripheral assistant control device is located in the second area, causing the automated assistant to perform the second automated assistant function.

12. The method of claim 11, wherein the additional data accessible via the computing device indicates that the peripheral assistant control device is only directly paired with the computing device via a wireless communication protocol; and wherein determining whether the peripheral assistant control device is located in the first area or the second area comprises: The peripheral assistant control device is determined to be located in the first area based on the paired computing device being assigned to the first area in the device topology.

13. The method of claim 11, wherein the computing device is one of a plurality of different assistant computing devices defined in a device topology accessible to the automated assistant, and wherein determining whether the peripheral assistant control device is located in the first region or the second region comprises: A determination is made as to which of the plurality of assistant computing devices the peripheral assistant control device is closest to.

14. A method implemented by one or more processors, the method comprising: determining whether the peripheral assistant control device is located in the first area or the second area based on data accessible via the computing device, wherein the data accessible via the computing device indicates that the peripheral assistant control device is only directly paired with the computing device via a wireless communication protocol; and wherein determining whether the peripheral assistant control device is located in the first area or the second area comprises: determining that the peripheral assistant control device is located in the first area based on the paired computing device being assigned to the first area in the device topology; wherein the peripheral assistant control device is in wireless communication with the computing device and the automated assistant is accessible via the computing device; receiving other data from the peripheral assistant control device, the other data being generated in response to a touch interface of the peripheral assistant control device receiving a touch input, wherein interaction with the touch interface causes the automated assistant to perform a function selected from at least a first automated assistant function and a second automated assistant function based on whether the peripheral assistant control device is determined to be located in the first area or the second area; When the peripheral assistant control device is determined to be in the first area: In response to receiving the other data and determining that the peripheral assistant control device is determined to be in the first area, causing the automated assistant to perform the first automated assistant function; and When the peripheral assistant control device is determined to be in the second area: In response to receiving the other data and determining that the peripheral assistant control device is determined to be in the second area, causing the automated assistant to perform a second automated assistant function different from the first automated assistant function.

15. The method according to claim 14, further comprising: Before determining whether the peripheral assistant control device is located in the first area or the second area: receiving input for requesting assignment of control of a type of smart device to the peripheral assistant control device; generating the first automated assistant function based on the input to cause control of one or more first smart devices of the type based on the first smart devices being of the type and being assigned to the first zone in a device topology; as well as The second auto assistant function is generated based on the input to cause control of one or more second smart devices of the type based on the second smart devices being of the type and assigned to the second zone in the device topology.

16. The method according to claim 14, further comprising: Before determining whether the peripheral assistant control device is located in the first area or the second area: The first automatic assistant function is assigned to the first zone based on a user interface input requesting that the first automatic assistant function be assigned to the peripheral assistant control device and to the first zone.

17. The method according to claim 16, further comprising: Before determining whether the peripheral assistant control device is located in the first area or the second area: The second automatic assistant function is assigned to the second zone based on a user interface input requesting that the second automatic assistant function be assigned to the peripheral assistant control device and to the second zone.

18. The method of any one of claims 14 to 17, wherein the computing device is one of a plurality of different assistant computing devices defined in a device topology accessible to the automated assistant, and wherein determining whether the peripheral assistant control device is located in the first region or the second region comprises: A determination is made as to which of the plurality of assistant computing devices the peripheral assistant control device is paired with and / or is in closest proximity to.

19. A method implemented by one or more processors, the method comprising: receiving audio data corresponding to spoken utterances provided by a user to an automated assistant interface of a computing device, wherein the computing device provides access to the automated assistant and is connected to a peripheral assistant control device; determining, based on the audio data, that the spoken utterance includes a request to control one or more functions in response to an interaction with a touch interface of the peripheral assistant control device, wherein the peripheral assistant control device provides output to the computing device or another computing device in response to the interaction with the touch interface; determining, based on the amount of audio data, that the spoken utterance at least partially identifies the one or more functions associated with the output of the peripheral assistant control device; identifying one or more other functions executable via the computing device or the automated assistant based on the data associated with the one or more functions, wherein the spoken utterance does not specify content of the one or more other functions; as well as When at least a first condition is met, correlation data is generated that relates the output of the peripheral assistant control device to the one or more functions, and when at least a second condition is met, the output of the peripheral assistant control device is related to the one or more other functions.

20. The method according to claim 19, further comprising: After generating the correlation data, determining that the peripheral assistant control device has provided the output in response to the user interacting with the touch interface; determining whether the first condition or the second condition is satisfied based on other data associated with the user; When the first condition is met: causing the one or more functions to be performed in response to the user interacting with the touch interface of the peripheral assistant control device; as well as When the second condition is met: In response to the user interacting with the touch interface of the peripheral assistant control device, the one or more functions are executed.

21. The method according to claim 19, wherein the one or more functions include initiating a media action for causing presentation of media, and the first condition is satisfied when the media is not presented at the computing device and / or another device associated with the user, and Wherein the second condition is satisfied when media is presented at the computing device or the other device associated with the user.

22. The method of claim 21, wherein the one or more other functions comprises a next media item action for causing a new media item to be presented in place of a current media item currently being presented.

23. The method according to claim 19, wherein the one or more functions, when executed, cause the computing device or the other device to operate in accordance with a setting, and the second condition is satisfied when the computing device or the other device operates in accordance with the setting, and The one or more other functions, when executed, cause the computing device or another device to operate according to another setting, and the first condition is satisfied when the computing device or the another device operates according to the another setting.

24. The method according to claim 23, wherein the one or more functions, when executed, cause the computing device to operate according to the settings, and The one or more other functions, when executed, cause the other device to operate according to the other setting.

25. The method according to claim 24, wherein the one or more functions cause the computing device to switch an operating mode from a first operating mode to a second operating mode, and The one or more other functions cause the computing device to switch the operating mode from the second operating mode to the first operating mode.

26. The method of claim 25, wherein the first operating mode causes the computing device to provide a first output via a modality of the computing device, and the second operating mode causes the computing device to provide a second output different from the first output via the modality of the computing device.

27. The method of claim 19, wherein the first condition comprises a property of a smart device being in a first state, and wherein the second condition comprises the property of the smart device being in a second state.

28. The method of claim 27, wherein the one or more functions include causing a property of the smart device to transition from the first state to the second state, and wherein the one or more other functions include causing a property of the smart device to transition from the second state to the first state.

29. The method according to claim 27 or 28, further comprising: determining that the peripheral assistant control device has provided the output in response to the user interacting with the touch interface; The first condition or the second condition is determined in response to the output being provided and based on a device topology defining a current state of a property of the smart device.

30. A non-transitory computer-readable storage medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 29.

31. A system for performing automated assistant functions in response to user interactions, the system comprising one or more processors for carrying out the method of any one of claims 1 to 29.

Images