Conditionally assigning various automatic assistant functions to interactions with peripheral assistant control devices

Abstract

The invention relates to conditionally assigning various automatic assistant functions to interactions with peripheral assistant control devices. In response to the user interacting with the tangible peripheral assistant control device, the automated assistant is caused to perform one or more actions. The one or more actions performed may be based on input previously provided by a user when configuring the peripheral assistant control device. One or more actions performed in response to interaction with the peripheral assistant control device may vary based on one or more conditions, in some embodiments, the method may include, for example, which user is currently active, where a peripheral assistant control device is currently located (which may optionally be inferred based on which pairing of the button with a plurality of assistant computing devices), and / or a current state of one or more smart devices and / or other devices (e.g., as determined based on a device topology). The utility of the peripheral assistant control device may be automatically extended beyond the range specially requested by the user during configuration.

Description

[0001] Division Explanation This application is a divisional application of the patent application with the application number 201980043378.0, titled "Conditionally Assigning Various Automatic Assistant Functions to Interactions with a Peripheral Assistant Control Device", which was filed with the Chinese Patent Office on December 25, 2020. Background Art

[0002] Humans can engage in human-machine conversations using interactive software applications referred to herein as "automatic assistants" (also known as "digital agents", "chatbots", "interactive personal assistants", "intelligent personal assistants", "assistant applications", "conversational agents", etc.). For example, humans (who may be referred to as "users" when they interact with an automatic assistant) can provide commands and / or requests to an automatic assistant by using oral natural language inputs (i.e., utterances) (in some cases, the oral natural language input can be converted into text and then processed) and / or by providing text (e.g., typed) natural language inputs. The automatic assistant responds to requests by providing response user interface outputs, which can include auditory and / or visual user interface outputs.

[0003] In many cases, users provide oral utterances to cause an automatic assistant to perform various actions. However, a large amount of computing and / or network resources can be used to process oral utterances to perform automatic assistant actions. For example, an automatic assistant action of turning on a smart light can be achieved by directly transmitting a corresponding command to the smart light or transmitting it to a third-party server that processes the command and then transmits the corresponding command to the smart light. However, when performing such an action in response to an oral utterance such as "Assistant, turn on smart light X", speech-to-text processing is performed on the audio data of the collected oral utterance, natural language processing is performed based on the resulting text, and then the corresponding command is determined. Further, in the case of remotely performing speech-to-text and / or natural language processing, the audio data and / or other data will need to be transmitted over one or more networks. Summary of the Invention

[0004] The embodiments described herein relate to causing an automatic assistant to perform one or more actions in response to user interaction with a peripheral assistant control device. The peripheral assistant control device may be paired (e.g., via Bluetooth or other communication protocol) with an assistant computing device that includes an automatic assistant interface and enables execution of various automatic assistant functions, either alone and / or by interacting with remote automatic assistant components. In response to user interaction with the peripheral assistant control device, output may be provided by the peripheral assistant control device to the paired assistant computing device. In response to receiving the output, the paired assistant computing device may initiate execution of one or more automatic assistant actions / functions. The embodiments described herein may be used to increase the utility of the peripheral assistant control device such that fewer peripheral assistant control devices may be needed to enable a given number of automatic assistant functions. Additionally, some embodiments may facilitate / reduce the complexity of the configuration of the peripheral assistant control device. Further, since the peripheral assistant control device may reduce the amount of speech recognition processing (and related bandwidth usage, etc.) required by the automatic assistant system, increasing the utility / functionality of the peripheral assistant control device may be used to further reduce the processing, bandwidth, etc. used by the automatic assistant system as a whole.

[0005] As described herein, the automatic assistant actions performed may be based on user interface inputs previously provided by the user when configuring the peripheral assistant control device. Also, the actions performed may vary according to one or more conditions such as which of a plurality of users is determined to be currently active, where the peripheral assistant control device is currently located (which may optionally be inferred based on which of a plurality of assistant computing devices the peripheral assistant control device is paired with), and / or the current state of one or more smart devices and / or other devices (e.g., as determined based on a device topology). Also as described herein, the automatic assistant actions performed may include one or more actions that extend beyond the actions specifically requested by the user during the configuration of the peripheral assistant control device.

[0006] The peripheral assistant control device itself may lack direct access to the automatic assistant and may lack various hardware resources present on the paired assistant computing device or a less capable version of the various hardware resources present on the paired assistant computing device. For example, the peripheral assistant control device may lack a microphone and a Wi-Fi module, while the paired assistant computing device includes both. Further, although the peripheral assistant control device may include a processor, it may be less capable compared to the processor included at the paired assistant computing device. In some embodiments, the peripheral assistant control device may include virtual buttons provided at a graphical user interface and / or any other interface capable of presenting virtual buttons. Alternatively or additionally, the peripheral assistant control device may include physical buttons having one or more user interfaces through which a user may provide one or more different inputs. The one or more different inputs may cause the peripheral assistant control device to provide one or more different outputs via a wireless communication channel, respectively.

[0007] One or more users may provide user interface inputs to configure the peripheral assistant control device. For example, user interface inputs may be provided at the assistant computing device and / or the peripheral assistant control device to initially pair the assistant computing device and the peripheral assistant control device. Additional user interface inputs may be provided to explicitly assign a certain (or certain) automatic assistant function to the interaction with the peripheral assistant control device. For example, a user may say "Assistant, when I push this button turn on the living room lights". In various embodiments, although a user only specifies a certain (or certain) automatic assistant function to be assigned to the interaction with the peripheral assistant control device, one or more additional automatic assistant functions may be further automatically suggested (and assigned if a positive user interface input is received) or may be further automatically assigned (without first being suggested).

[0008] As an example, for the spoken utterance "Assistant, when I push this button turn on the living room lights", the automated assistant action that causes the on / off attribute of the "living room" lights to be commanded to the "on" state is explicitly indicated by the spoken utterance and can thus be assigned to a peripheral assistant control device. Although not explicitly indicated by the spoken utterance, an additional automated assistant function that causes the on / off attribute of the "living room" lights to be commanded to the "off" state can be further assigned. Additionally, causing the on / off attribute of the "living room" lights to be commanded to the "on" state can be further assigned to a first condition where the on / off attribute of the "living room" lights is in the "off" state, and causing the on / off attribute of the "living room" lights to be commanded to the "off" state can be further assigned to a second condition where the on / off attribute of the "living room" lights is in the "on" state. Thus, in response to receiving an output in response to an interaction from a peripheral assistant control device, the current state of the on / off attribute of the "living room" lights can be determined (e.g., by querying the lights and / or by using a device topology (with the most recently updated device state)), and depending on whether the current condition is the first condition or the second condition, the lights are commanded to the on state or the off state.

[0009] As another example, for the spoken utterance "Assistant, when I push this button play some bluegrass music", the automated assistant action that causes the paired assistant computing device (and / or other devices, if specified) to play "bluegrass music" is explicitly indicated by the spoken utterance and can thus be assigned to a peripheral assistant control device. Although not explicitly indicated by the spoken utterance, an additional automated assistant function can be additionally assigned, the additional automated assistant function causing the currently playing "bluegrass song" to be replaced by the playing of another "bluegrass song" (e.g., a "next song" automated assistant function). Further, the play "bluegrass music" automated assistant function can be additionally assigned to a first condition where the assistant computing device is not currently presenting "bluegrass music" (or optionally, no music), and the "next song" automated assistant function can be additionally assigned to a second condition where "bluegrass music" (or optionally, any music) is currently being presented at the assistant computing device. Thus, in response to receiving an output in response to an interaction from a peripheral assistant control device, if the first condition is met, the play "bluegrass music" automated assistant function can be initiated by the assistant computing device, while if the second condition is met, the "next song" automated assistant function can alternatively be initiated by the assistant computing device.

[0010] Thus, even if a user can provide the same user interaction (e.g., button press) at a peripheral assistant control device on two separate occasions and, in response, transmit the same output from the peripheral assistant control device to a paired assistant computing device, different automatic assistant functions may ultimately occur on the two separate occasions due to different conditions at the two separate occasions.

[0011] Automatically suggesting and / or automatically allocating one or more additional automatic assistant functions to a peripheral assistant control device yields various technical benefits. For example, it enables additional automatic assistant functions to be performed in response to certain interactions of a user with the peripheral automatic assistant device. The additional automatic assistant functions can be performed in response to receiving an output provided by the peripheral automatic assistant device and determining that a certain (or certain) condition is met - which can be significantly more efficient (in terms of computing and / or networking) than processing the corresponding verbal utterance. Additionally, for example, it can reduce the amount of user input required for the user to explicitly allocate additional automatic assistant functions in other cases. Moreover, automatically changing the automatic assistant function initiated in response to receiving an output from a peripheral assistant control device based on current conditions also results in various technical benefits. For example, initiating the correct automatic assistant function for current conditions can ensure that the processor and / or network resources used to perform that automatic assistant function are not wasted. For example, a technique that does not initiate different automatic assistant functions for different conditions would wastefully generate and transmit duplicate commands that will be ignored (e.g., generating and transmitting an "turn on the light" command even if the light is already "on").

[0012] In some additional and / or alternative embodiments, different automatic assistant functions can be assigned to the same interaction with a peripheral automatic assistant control device, and which automatic assistant function to mimic in response to a given instance of the interaction can depend on where the peripheral assistant control device is currently located. For example, when the peripheral assistant control device is located inside or near a "living room", a first automatic assistant function of turning on and / or off the "living room" lights can be initiated, while when the peripheral assistant control device is located inside or near a "basement", a second automatic assistant function of "playing some music on basement devices" can be initiated. Various techniques can be utilized to determine where the device is currently located. For example, where the device is currently located can be based on which of multiple assistant computing devices the button is currently paired with. The pairing of the peripheral assistant control device can be switched between different assistant computing devices in response to user interface input (in the respective assistant computing device living room and / or at the peripheral assistant control device), or can be automatically switched so that the assistant computing device with the strongest (or only detectable) communication signal (e.g., Bluetooth) is paired with the peripheral assistant control device.

[0013] In some embodiments, explicit user interface input may be provided that indicates which of the automatic assistant functions should be performed at which locations of the peripheral assistant control device. For example, the user may provide an oral utterance such as, "Assistant, when the interface device is in the living room, have it control the television, but when the interface device is in the kitchen, have it control playback of podcasts". In response, the automatic assistant may establish corresponding configuration settings for the peripheral assistant control device. For example, by using data indicative of the location of the peripheral assistant control device, the automatic assistant may associate the output of the peripheral assistant control device with a specific function based on where the peripheral assistant control device is indicated to be located. For example, the peripheral assistant control device may be determined to be in the "living room" based on its pairing with an automatic assistant computing device assigned to the "living room" location in the device topology. The location of the peripheral assistant control device may be determined using additional and / or alternative techniques, such as techniques for determining location based on network strength data, data from another device with which the peripheral assistant control device is paired or otherwise communicates, data identifying a specific device with which the peripheral assistant control device is paired or otherwise communicates, altitude data, motion data, audio data, image data, and / or any other data based on which the location of the device may be determined.

[0014] In some embodiments, a user interface input can specify an auto-assistant function for controlling a type of smart device and can specify its assignment to a peripheral assistant control device. However, the user interface input may not specify a particular smart device to be controlled. For example, a user can provide an utterance such as "Assistant, have the interface device control turn the lights on and off for the room it is in". In some of these embodiments, particular auto-assistant functions can be assigned that are each directed to control one or more particular smart devices of that type and are each associated with a corresponding condition, such as the location of the peripheral assistant control device. For example, a first auto-assistant function for turning on / off the "living room" lights can be assigned to a first condition where the peripheral assistant control device is in the "living room", while a second auto-assistant function for turning on / off the "kitchen" lights can be assigned to a second condition where the peripheral assistant control device is in the "kitchen". When the peripheral assistant control device is determined to be in the "kitchen" (e.g., based on pairing with a "kitchen" assistant computing device), the output received in response to an interaction with the peripheral assistant control device can result in control of the on / off state of the lights that are also assigned to the "kitchen" (e.g., in the device topology). On the other hand, when the peripheral assistant control device is determined to be in the "living room" (e.g., based on pairing with a "living room" assistant computing device), the output received in response to an interaction with the peripheral assistant control device can result in control of the on / off state of the lights that are also assigned to the "living room" (e.g., in the device topology).

[0015] Assigning different auto-assistant functions based on where the peripheral assistant control device currently is can result in various technical benefits. For example, it enables the functionality of the peripheral assistant control device to be dynamically adjusted based on location without the user having to provide resource-consuming additional interface inputs to explicitly reconfigure the functionality each time the peripheral assistant control device is moved. Additionally, for example, it enables a single peripheral assistant control device and an optional single interaction to have multiple functions based on location - thus eliminating the need for multiple separate peripheral assistant control devices for multiple functions.

[0016] In some additional and / or alternative embodiments, different assistant functions can be assigned to the same interaction with a peripheral assistant control device, and which assistant function to mimic in response to a given interaction instance can depend on which of multiple users is the currently active user. For example, when a first user is the active user, a first assistant function of turning on and / or off the "living room" lights can be initiated, while when a second user is the active user, a second assistant function of "playing some music" can be initiated. Various techniques can be utilized to determine which of the multiple users is the currently active user. For example, the user who most recently interacted with the assistant computing device paired with the peripheral assistant control device can be considered the currently active user (optionally depending on whether the most recent interaction was within the last 5 minutes or within another time threshold). The specific user can be determined to have interacted with the assistant computing device by verifying the user based on, for example, matching the verbal input from the interaction with the stored speaker embedding of that user. Additional and / or alternative verification techniques can be utilized, such as a personal identification number (PIN) provided by the user, a password provided by the user, and / or other biometric identification techniques (with the permission of the corresponding user).

[0017] Assigning different assistant functions based on which user is the currently active user of the peripheral assistant control device results in various technical benefits. For example, it enables the functions of the peripheral assistant control device to be dynamically adjusted based on the currently active user without the user having to provide resource-consuming additional interface inputs to explicitly reconfigure the functions each time the active user changes. Additionally, for example, it enables a single peripheral assistant control device and an optional single interaction to have multiple functions depending on the user - thus eliminating the need for multiple separate peripheral assistant control devices for multiple functions.

[0018] In some embodiments, in response to the automatic assistant detecting the presence of a peripheral assistant control device, conditions for when the peripheral assistant control device is assigned control of a specific function can be automatically established. In some embodiments, the peripheral assistant control device can communicate via Bluetooth, and when the user turns on the peripheral assistant control device, the device can be paired with an existing client device that the user owns or is otherwise associated with. In response to the pairing of the peripheral assistant control device with the existing client device, the existing client device can invoke an automatic assistant that can provide natural language output to guide further configuration of the peripheral assistant control device. For example, the automatic assistant can cause the existing client device to present audible natural language output such as, "George, I've detected a new interface device in the living room. Would you like the new interface device to control the lights in the living room?" The natural language output can be generated based on the automatic assistant identifying the location of the peripheral assistant control device, other existing devices in the living room that can be controlled via the automatic assistant, and / or any other devices communicating with the existing client device.

[0019] In response to the natural language output from the automatic assistant, the user can confirm the configuration proposed by the automatic assistant and / or request a different configuration for the peripheral assistant control device. For example, the user can provide a natural language input such as, "Assistant, when I am in the living room, have the interface device control the television, but when I am not in the living room, have the interface device control the lights in the living room" ("interface device" can refer to the peripheral assistant control device). In response to receiving the natural language input from the user, the automatic assistant can cause the peripheral assistant control device to be conditionally assigned control of the identified function. Specifically, in order to assign control of the lights to the peripheral assistant control device, the automatic assistant should determine that the user is not present in the living room. Additionally, in order to assign control of the television to the peripheral assistant control device, the automatic assistant should determine that the user is actually present in the living room. Such determination can be based on data accessible by the existing client device, the automatic assistant, and / or any other relevant device, as discussed herein.

[0020] The above description is provided as an overview of some embodiments of the present disclosure. These embodiments and further descriptions of other embodiments will be described in more detail below.

[0021] In some embodiments, a method implemented by one or more processors includes operations such as determining whether a first user or a second user is the current active user of a peripheral assistant control device based on data accessible via a computing device, where the peripheral assistant control device communicates wirelessly with the computing device and an automatic assistant is accessible via the computing device. The operations may further include receiving, from the peripheral assistant control device, additional data generated in response to a touch input being received at a touch interface of the peripheral assistant control device, where the receipt of the additional data generated in response to an interaction with the touch interface causes the automatic assistant to perform a function selected from at least a first automatic assistant function and a second automatic assistant function based on whether the first user or the second user is determined to be the current active user of the peripheral assistant control device. The operations may further include: when the first user is determined to be the current active user, causing the automatic assistant to perform the first automatic assistant function in response to receiving the additional data and determining that the first user is the current active user. The operations may further include: when the second user is determined to be the current active user, causing the automatic assistant to perform a second automatic assistant function different from the first automatic assistant function in response to receiving the additional data and determining that the second user is the current active user.

[0022] In some embodiments, the first automatic assistant function may also be performed in response to a first verbal request corresponding to an automatic assistant interface of the computing device, and the second automatic assistant function may also be performed in response to a second verbal request corresponding to the automatic assistant interface of the computing device. In some embodiments, the method may further include, before determining the current active user of the peripheral assistant control device: in response to one or more verbal utterances provided by the first user and / or the second user for requesting the automatic assistant to assign the first automatic assistant function and the second automatic assistant function to the peripheral assistant control device, assigning the first automatic assistant function and the second automatic assistant function to the peripheral assistant control device. In some embodiments, the method may further include, before determining the current active user of the peripheral assistant control device: based on determining that the first verbal utterance requests the assignment of the first automatic assistant function to the peripheral assistant control device and based on determining that the first verbal utterance is provided by the first user, assigning the first automatic assistant function to the peripheral assistant control device and to the first user. In some embodiments, the method may further include based on determining that the second verbal utterance requests the assignment of the first automatic assistant function to the peripheral assistant control device and based on determining that the second verbal utterance is provided by the second user, assigning the second automatic assistant function to the peripheral assistant control device and to the second user. In some embodiments, the touch interface of the peripheral assistant control device includes touch sensors responsive to tactile gestures. In some embodiments, the peripheral assistant control device provides additional data only in response to an interaction with the touch interface, and the touch interface responds only to one or more tactile gestures.

[0023] In some embodiments, the data accessible via the computing device includes audio data detected via one or more microphones of the computing device before receiving other data from the peripheral assistant control device; and determining whether the first user or the second user is the current active user of the peripheral assistant control device based on the data accessible via the computing device includes: determining that the audio data matches the stored voice embedding of the first user based on processing the audio data; and determining that the first user is the active user based on the audio data matching the stored voice embedding of the first user. In some embodiments, determining that the first user is the active user is further based on: determining that the audio data is detected within a threshold amount of time from receiving the other data from the peripheral assistant control device. In some embodiments, wherein the first automatic assistant function includes changing an attribute of the smart device, and causing the automatic assistant to perform the first automatic assistant function includes: determining the current state of the attribute of the smart device, and generating a command based on the current state to cause the attribute of the smart device to transition to an alternative state different from the current state; and transmitting the command to the smart device or to a server associated with the smart device. In some embodiments, the method may further include receiving an input that requests assigning only the function of causing the attribute of the smart device to transition to the current state to the peripheral assistant control device; and in response to receiving the input, automatically assigning the first automatic assistant function to the smart device, wherein the first automatic assistant function changes the attribute of the smart device at least between the current state and the alternative state, rather than only causing the attribute of the smart device to transition to the current state.

[0024] In yet another embodiment, a method implemented by one or more processors includes: determining, based on data accessible via a computing device, whether a peripheral assistant control device is located in a first region or a second region, wherein the peripheral assistant control device wirelessly communicates with the computing device and the automatic assistant is accessible via the computing device. The method may further include receiving, from the peripheral assistant control device, other data generated in response to a touch input received by a touch interface of the peripheral assistant control device, wherein interacting with the touch interface causes the automatic assistant to perform a function selected from at least a first automatic assistant function and a second automatic assistant function depending on whether the peripheral assistant control device is determined to be located in the first region or the second region. The method may further include, when the peripheral assistant control device is determined to be in the first region: causing the automatic assistant to perform the first automatic assistant function in response to receiving the other data and determining that the peripheral assistant control device is determined to be in the first region. The method may further include, when the peripheral assistant control device is determined to be in the second region: causing the automatic assistant to perform a second automatic assistant function different from the first automatic assistant function in response to receiving the other data and determining that the peripheral assistant control device is determined to be in the second region.

[0025] In some embodiments, data accessible via a computing device indicates that a peripheral assistant control device is directly paired only with the computing device via a wireless communication protocol, and wherein determining whether the peripheral assistant control device is located in a first region or a second region includes determining that the peripheral assistant control device is located in the first region based on the paired computing device being assigned to the first region in a device topology. In some embodiments, the method may further include, before determining whether the peripheral assistant control device is located in the first region or the second region: receiving an input for requesting an assignment of control of a type of smart device to the peripheral assistant control device. In some embodiments, the method may further include generating a first automatic assistant function based on the input to effect control of one or more first smart devices of the type based on the first smart device being of the type and being assigned to the first region in the device topology. In some embodiments, the method may further include generating a second automatic assistant function based on the input to effect control of one or more second smart devices of the type based on the second smart device being of the type and being assigned to the second region in the device topology.

[0026] In some embodiments, the method may further include, before determining whether the peripheral assistant control device is located in the first region or the second region: assigning the first automatic assistant function to the first region based on a user interface input for requesting an assignment of the first automatic assistant function to the peripheral assistant control device and to the first region. In some embodiments, the method may further include, before determining whether the peripheral assistant control device is located in the first region or the second region: assigning the second automatic assistant function to the second region based on a user interface input for requesting an assignment of the second automatic assistant function to the peripheral assistant control device and to the second region. In some embodiments, the computing device is one of a plurality of different assistant computing devices defined in a device topology accessible by an automatic assistant, and wherein determining whether the peripheral assistant control device is located in the first region or the second region includes determining which of the plurality of assistant computing devices the peripheral assistant control device is paired with and / or is closest to.

[0027] In other embodiments, a method implemented by one or more processors is described as including operations such as receiving audio data corresponding to spoken utterances provided by a user to an automated assistant interface of a computing device, where the computing device provides access to an automated assistant and is connected to a peripheral assistant control device. The method may further include determining, based on the audio data, that the spoken utterance includes a request to control one or more functions in response to an interaction with a touch interface of the peripheral assistant control device, where the peripheral assistant control device provides output to the computing device or another computing device in response to an interaction with the touch interface. The method may further include determining, based on the audio data, that the spoken utterance at least partially identifies one or more functions related to the output of the peripheral assistant control device. The method may further include identifying, based on data associated with the one or more functions, one or more other functions executable via the computing device or the automated assistant, where the spoken utterance does not specify the content of the one or more other functions. The method may further include generating, when at least a first condition is satisfied, correlation data relating the output of the peripheral assistant control device to the one or more functions, and relating, when at least a second condition is satisfied, the output of the peripheral assistant control device to the one or more other functions.

[0028] In some embodiments, the method may further include, after generating the correlation data, determining that the peripheral assistant control device has provided output in response to a user interacting with the touch interface. In some embodiments, the method may further include determining whether the first condition or the second condition is satisfied based on other data associated with the user. In some embodiments, the method may further include, when the first condition is satisfied: in response to a user interacting with the touch interface of the peripheral assistant control device, causing one or more functions to be performed. In some embodiments, the method may further include, when the second condition is satisfied: in response to a user interacting with the touch interface of the peripheral assistant control device, causing one or more functions to be performed.

[0029] In some embodiments, one or more functions include a start media action for causing the presentation of media, and a first condition is satisfied when the media is not presented at the computing device and / or another device associated with the user, and a second condition is satisfied when the media is presented at the computing device or another device associated with the user. In some embodiments, one or more other functions include a next media item action for causing a new media item to be presented in place of the current media item currently being presented. In some embodiments, one or more functions, when executed, cause the computing device or another device to operate according to a setting, and a second condition is satisfied when the computing device or another device operates according to the setting, and one or more other functions, when executed, cause the computing device or another device to operate according to another setting, and a first condition is satisfied when the computing device or another device operates according to the other setting.

[0030] In some embodiments, one or more functions, when executed, cause the computing device to operate according to the setting, and one or more other functions, when executed, cause another device to operate according to another setting. In some embodiments, one or more functions cause the computing device to switch the operating mode from a first operating mode to a second operating mode, and one or more other functions cause the computing device to switch the operating mode from the second operating mode to the first operating mode. In some embodiments, the first operating mode causes the computing device to provide a first output via a modality of the computing device, and the second operating mode causes the computing device to provide a second output different from the first output via the modality of the computing device. In some embodiments, the first condition includes that an attribute of the smart device is in a first state, and the second condition includes that an attribute of the smart device is in a second state. In some embodiments, one or more functions include causing the attribute of the smart device to transition from the first state to the second state, and one or more other functions include causing the attribute of the smart device to transition from the second state to the first state. In some embodiments, the method may further include determining that a peripheral assistant control device has provided an output in response to a user interacting with a touch interface; in some embodiments, the method may further include determining whether the first condition or the second condition based on the output being provided and the device topology defining the current state of the attributes of the smart device.

[0031] Other embodiments may include a non - transitory computer - readable storage medium storing instructions executable by one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) to perform one or more of the methods such as those described above and / or elsewhere herein. Other embodiments may include a system having one or more computers and / or one or more robots, the one or more computers and / or one or more robots including one or more processors operable to run the stored instructions to perform one or more of the methods such as those described above and / or elsewhere herein.

[0032] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1A and Figure 1B Views of a peripheral assistant control device having functions separately assigned based on whether a particular user is determined to be present in an area including the peripheral assistant control device are shown.

[0034] Figure 2A and Figure 2B Views of a peripheral assistant control device having control of one or more functions assigned based on the position of the peripheral assistant control device are shown.

[0035] Figure 3 A system for assigning control of one or more executable assistant functions to a peripheral assistant control device based on whether one or more different conditions are met is shown.

[0036] Figure 4A and Figure 4B A method for assigning control of a particular function to a peripheral assistant control device based on whether one or more conditions are met is shown.

[0037] Figure 5 is a block diagram of an example computer system. DETAILED DESCRIPTION

[0038] Figure 1A and Figure 1B Views 100 and 130 of a peripheral assistant control device 112 having functions separately assigned based on whether a particular user is determined to be present in an area including the peripheral assistant control device 112 are shown. Specifically, Figure 1AFIG. 100 shows a view where when the first user 106 is determined to be present at the location of the peripheral assistant control device 112 and / or otherwise determined to be close to the peripheral assistant control device 112, the peripheral assistant control device 112 is assigned a first assistant function. For example, when the first user 106 is determined to be located within area 128 of their residence 126 (e.g., the kitchen of the residence), the peripheral assistant control device 112 can reassign control of the first assistant function (e.g., turning on / off the kitchen lights). However, when the user is determined to have relocated away from area 128 and / or residence 126 (as Figure 1B shown), and / or when the second user 108 is determined to be present within area 128 of the residence 126 and / or otherwise close to the peripheral assistant control device 112, the peripheral assistant control device 112 can be assigned to control a second assistant function (e.g., playing music at the first client device 110).

[0039] For example, the first user 106 can provide a request to an application accessible via a computing device to request that the peripheral assistant control device 112 control the lights in their kitchen when the first user 106 specifically interacts with the peripheral assistant control device 112. Additionally, the second user 108 can also use the application to request that the peripheral assistant control device 112 control the music presented by the first client device 110 when the second user 108 specifically interacts with the peripheral assistant control device 112. Thereafter, when the first user 106 is determined to be present and interacting with the peripheral assistant control device 112, the peripheral assistant control device 112 can be assigned control of the lights in the kitchen. Thus, any interaction between the first user 106 and the peripheral assistant control device 112 at this time will result in the operation of the kitchen lights being modified. Additionally, when the first user 106 is no longer present, but the second user 108 is present and interacting with the peripheral assistant control device 112, any interaction between the second user 108 and the peripheral assistant control device 112 will result in the first client device 110 presenting music based on the interaction.

[0040] In some embodiments, the server device 120 and / or the client device may consider the first user 106 or the second user 108 to be "active" in terms of their ability to influence the allocation of control of the peripheral assistant control device 112. Additionally, whether a particular user is "active" with respect to the peripheral assistant control device 112 may be based on the voice identification of the first user 106 and / or the second user 108. For example, and in the case of having prior permission from the first user 106, the first client device 110 and / or the server device 120 may perform voice processing to determine whether the first user 106 has provided a most recent verbal utterance to the first client device 110. If the first user 106 has been identified as the user who most recently provided a verbal utterance based on the voice characteristics of the first user 106, the peripheral assistant control device 112 may be allocated control of one or more particular functions preferred by the first user 106 (e.g., controlling kitchen lights). Additionally, the second user 108 may subsequently interact with the first client device 110 by providing a verbal utterance. The first client device 110 and / or the server device 120 may identify the second user 108 as the user who provided the subsequent verbal utterance, and cause the peripheral assistant control device 112 to be allocated control of one or more other particular functions preferred by the second user 108 (e.g., controlling music presented by the first client device 110).

[0041] In some embodiments, when the first user 106 is considered "active" relative to the peripheral assistant control device 112, the "active" time of the first user 106 can expire after a period of time and / or in response to the auto-assistant confirming the presence of another user or the first user 106 has left the area 128. For example, after the first user 106 has interacted with the peripheral assistant control device 112 and the peripheral assistant control device 112 has been assigned control of the functions preferred by the first user 106, the second user 108 can be determined to be present and / or interact with the peripheral assistant control device 112. In response, the "active" status of the first user 106 relative to the peripheral assistant control device 112 can expire, and the peripheral assistant control device 112 can be assigned control of one or more other functions (e.g., playing music) preferred by the second user 108. Additionally or alternatively, when the second user 108 has stopped interacting with the peripheral assistant control device 112 for a threshold period of time, the first client device 110 and / or the server device 120 can reassign control of the peripheral assistant control device 112 to control one or more functions preferred by the first user 106. In some embodiments, the peripheral assistant control device 112 can switch back to having control of the functions preferred by the first user 106 in response to the first user 106 responding to a query from the first client device 110 (such as "First User, are you there?") or the second user 108 not responding to a query from the first client device (such as "Second User, are you there?") after a threshold period of time.

[0042] In some embodiments, the peripheral assistant control device 112 can include one or more lights that are lit to indicate the user who is "active" relative to the peripheral assistant control device 112. For example, when the peripheral assistant control device 112 is assigned control of one or more functions preferred by the first user 106, one or more lights of the peripheral assistant control device 112 can emit a first color (e.g., green). Additionally, when the peripheral assistant control device 112 is assigned control of one or more other functions preferred by the second user 108, one or more lights of the peripheral assistant control device 112 can emit a second color (e.g., red) different from the first color.

[0043] The peripheral assistant control device 112 can operate as a computing device having one or more touch sensors responsive to one or more tactile gestures. For example, the peripheral assistant control device 112 can include one or more touch-sensitive sensors capable of providing signals in response to one or more different touch gestures provided by one or more users. The peripheral assistant control device 112 can be connected to one or more client devices and / or server devices to communicatively transmit signals generated in response to touch inputs. For example, the peripheral assistant control device 112 can communicate with the first client device 110 and / or the second client device 114 via a communication protocol such as Bluetooth, Wi-Fi, LTE, and / or any other wired or wireless communication protocol. In response to a user providing a touch gesture to the peripheral assistant control device 112, the peripheral assistant control device 112 can transmit signals to the first client device 110 and / or the second client device 114. In some embodiments, the first client device 110 and / or the second client device 114 can communicate with a remote computing device 118 (such as a server device 120) via a network 116 (such as the Internet). The first client device 110 and / or the second client device 114 can communicatively transmit data to the server device 120 for indicating that the user has provided a touch input to the peripheral assistant control device 112. In this way, various devices and / or applications (such as the automated assistant 122) can be controlled via the peripheral assistant control device 112.

[0044] In some embodiments, the peripheral assistant control device 112 can respond only to direct touch inputs from a user, as well as signals transmitted to and from the peripheral assistant control device 112 from one or more devices. In this way, the peripheral assistant control device 112 can include limited hardware (e.g., relative to a laptop computer and / or a cellular phone), thereby reducing power consumption and network latency that would otherwise be exhibited by computing devices that do not respond only to direct touch inputs from a user or that do not have limited hardware. In some embodiments, the peripheral assistant control device 112 can respond to various different touch gestures, such as single taps, double taps, other multi-taps, long presses, and / or any change in a touch input. Depending on the environment or scenario in which a user provides a particular touch gesture, the client device and / or the server device can respond differently to each of the touch gestures.

[0045] As Figure 1AAs shown, the first user 106 and the second user 108 may be located within the area of the residence 126 where the users live, and this area may include the peripheral assistant control device 112. Specifically, the first user 106 may exhibit the condition 102 of walking in the kitchen area, and the second user 108 may exhibit the condition 104 of also walking in the kitchen area. The peripheral assistant control device 112 may be connected to the first client device 110, the second client device 114, and / or the server device 120. By using the previous permission from the first user 106, the first client device 110, the second client device 114, and / or the server device 120 may determine the presence of the first user 106. Specifically, the presence of the first user 106 may be characterized by data indicating whether the first user 106 is within the residence 126, within the area 128, close to the peripheral assistant control device 112, and / or otherwise within a distance of the peripheral assistant control device 112 that allows the first user 106 to interact with the peripheral assistant control device 112 without having to maneuver at least a threshold distance and / or maneuver for at least a threshold amount of time before reaching the peripheral assistant control device).

[0046] When the presence of the first user 106 is determined and / or otherwise confirmed, the first client device 110, the second client device 114, and / or the server device 118 may associate one or more signals provided by the peripheral assistant control device 112 with specific functions. For example, the correlation between one or more specific functions and one or more specific peripheral assistant control device outputs may be characterized by correlation data 124. Specifically, and by way of example, the correlation data 124 may characterize the relationship between the presence of the first user 106 and the control of one or more functions via the peripheral assistant control device 112. When it is determined that the first user 106 is present near the peripheral assistant control device 112, the correlation data 124 may identify a first assistant function among one or more functions that can be controlled via the touch interface of the peripheral assistant control device 112.

[0047] In addition, the control of one or more other functions may be related to the determination of the absence of the first user 106. For example, as Figure 1BAs shown, the peripheral assistant control device 112, the first client device 110, the second client device 114, and / or the server device 120 may determine that the first user is no longer present at or near the peripheral assistant control device 112. Instead, the first user 106 may exhibit a condition 132 of being repositioned from the area 128. In response to determining that the first user 106 is no longer present at or near the peripheral assistant control device 112, the data provided by the peripheral assistant control device 112 in response to the second user 108 interacting with the peripheral assistant control device 112 may be related to one or more other functions. For example, when the second user 108 interacts with the peripheral assistant control device 112 and the first user 106 has been repositioned from the area 128, the interaction may cause one or more other functions to be executed or otherwise controlled. In some embodiments, one or more other functions may include a second automatic assistant function that is executed or controlled when the second user 108 provides an input gesture to the peripheral assistant control device 112 when the first user 106 is determined not to be present at or near the peripheral assistant control device 112.

[0048] In some embodiments, the assignment of control of one or more functions may be based on the specific areas in which the first user 106 and the second user 108 are determined to be present. For example, when the first user 106 is determined by the second client device 114 to be present in the living room, the peripheral assistant control device 112 may be assigned control of a first function. Additionally, when the first user 106 is determined by the first client device 110 to be present in the kitchen, the peripheral assistant control device 112 may be assigned control of a second function different from the first function. In some embodiments, when no specific user is detected and / or an unidentifiable user is detected, the peripheral assistant control device 112 may be assigned control of one or more specific functions. In some embodiments, when a combination of different users is detected, then the peripheral assistant control device 112 may be assigned control of one or more specific functions that may be different from the one or more other functions that the peripheral assistant control device 112 would be assigned control of if the presence of the first user 106 and / or the presence of the second user 108 were detected.

[0049] In some embodiments, one or more functions for which control can be assigned to the peripheral assistant control device 112 can be automated assistant functions. In other words, one or more functions can be controlled or executed by an automated assistant accessible via the first client device 110, the second client device 114, and / or the server device 120 or via the automated assistant. Thus, the one or more functions can be controlled and / or executed via verbal utterances provided to the automated assistant interface of the first client device 110 and also by providing input gestures (such as tap gestures) to the touch interface of the peripheral assistant control device 112.

[0050] Figure 2A and Figure 2B Views 200 and 230 respectively illustrate the assignment of control of one or more functions to the peripheral assistant control device 212 based on the location of the peripheral assistant control device 212. For example, when the peripheral assistant control device 212 is in a first area 228 (such as a kitchen), the peripheral assistant control device 212 can be assigned control of one or more functions. Additionally, when the peripheral assistant control device 212 is in a second area 208 (such as a living room), the peripheral assistant control device 212 can be assigned control of one or more other functions.

[0051] The determination of whether the peripheral assistant control device 212 is in the first area 228 or the second area 208 can be based on device topology data used to characterize the devices associated with the user 206. For example, the device topology data can be accessed at a remote computing device 218 (such as the server device 220), which can be accessed via a network 216 (such as the Internet). The device topology data can include identifiers of the first client device 210 and the second client device 214. Additionally, each device identifier can be associated with a region identifier corresponding to a geographical region accessible to the user 206. For example, a first identifier of the first client device 210 can be associated with a first region identifier of the first area 228. Additionally, a second identifier of the second client device 214 can be associated with a second region identifier of the second area 208.

[0052] The server device 220 may include correlation data 224 that can identify various correspondences between different areas accessible to the user 206 and different functions that can be controlled by the peripheral assistant control device 212. Additionally, it may be assumed that when the peripheral assistant control device 212 is connected to the first client device 210, the peripheral assistant control device 212 is located within the first area 228. For example, such an assumption may be based on the first client device 210 being assigned to the first area 228 in the device topology representation. Moreover, it may be assumed that when the peripheral assistant control device 212 is connected to the second client device 214, the peripheral assistant control device 212 is located within the second area 208. For example, such an assumption may be based on the second client device 214 being assigned to the second area 208 in the device topology representation. Thus, when the peripheral assistant control device 212 is paired or otherwise connected to the first client device 210, any data provided by the peripheral assistant control device 212 (such as data generated in response to a touch input from the user 206) may be processed based on the assumption that the peripheral assistant control device 212 is located within the first area 228. Additionally, when the peripheral assistant control device 212 is paired or otherwise connected to the second client device 214, any data provided by the peripheral assistant control device 212 may be processed based on the assumption that the peripheral assistant control device 212 is located within the second area 208.

[0053] In a non - limiting example, user 206 can provide oral utterances to an auto - assistant interface (such as the microphone of the first client device 210) to request the assignment of the peripheral assistant control device 212 to different functions according to certain conditions. The oral utterance can be "Assistant, when my peripheral assistant control device is in the kitchen, please provide the news when I perform a touch input". Additionally, in the same oral utterance or a different oral utterance, user 206 can provide a request such as "Assistant, also, when the peripheral assistant control device is in the living room, please control the television when I perform a touch input". In response to receiving the oral utterance, the first client device 210 can generate audio data that can be transmitted to the server device 220, and the server device can process the audio data via the auto - assistant 222 application at the server device 220. The audio data can be processed to determine the configuration of certain settings for the peripheral assistant control device 212. For example, the server device 220 and / or any other device discussed herein can determine that user 206 is requesting that the peripheral assistant control device 212 control a first function when located in the first area 228 (as shown in the view 200 of Figure 2A ), and control a second function when located in the second area 208 (as shown in the view 230 of Figure 2B ).

[0054] Additionally or alternatively, user 206 may provide an oral utterance, such as "Assistant, please control the lights in the room that I am in", and / or otherwise provide an input having the same meaning (e.g., a text input). In response to receiving the oral utterance, the first client device 210 may generate audio data that can be transmitted to the server device 220, which may process the audio data via the automatic assistant 222 application at the server device 220. The audio data may be processed to determine a configuration for certain settings of the peripheral assistant control device 212. For example, the server device 220 and / or any other device discussed herein may identify another device located in the current vicinity or current area of the user 206. Based on data indicating the location of the other device, settings may be generated such that when the peripheral assistant control device 212 is paired with or otherwise communicates with the other device, the peripheral assistant control device 212 will be assigned control of the lights in that location. Additionally, when the peripheral assistant control device 212 is paired with or otherwise communicates with the other device, a device topology representation correlating the location of the other device with the lights may be used as a basis for assigning control of the lights to the peripheral assistant control device 212.

[0055] In some embodiments, the determination of where the peripheral assistant control device 212 is located may be based on device topology data. Alternatively or additionally, the determination of where the peripheral assistant control device 212 is located may be based on the output of one or more sensors and / or one or more transmitters of the peripheral assistant control device 212, the first client device 210, and / or the second client device 214. For example, the signal strength of a local area network (e.g., a Wi-Fi network) available to the peripheral assistant control device 212 may be used as an indication of where the peripheral assistant control device 212 is located within the residence 226. Additionally, when the device topology data provides an indication of the location of the local area network signal source, the signal strength of the local area network may be used in combination with the device topology data to determine where the peripheral assistant control device 212 is located. Optionally, the location of the peripheral assistant control device 212 may be based on a comparison between the signal strengths at the first client device 210, the second client device 214, and the peripheral assistant control device 212, respectively, in combination with the known location of the local area network signal source.

[0056] In some embodiments, the determination of where the peripheral assistant control device 212 is located can be based on the output of one or more sensors and / or one or more transmitters (such as one or more such devices included in a Global Positioning System (GPS) device). For example, the peripheral assistant control device 212 can include a GPS device that can provide an output indicating the location of the peripheral assistant control device 212. Thus, when a user provides a touch input to the peripheral assistant control device 212, the peripheral assistant control device 212 can communicatively transmit output data that identifies the location of the peripheral assistant control device based on the GPS device and the type of touch input received from the user 206. A device that receives the output data can compare the indicated location of the peripheral assistant control device 212 with the correlation data 224 to identify one or more functions that the user 206 intends to control via the touch input at the peripheral assistant control device 212 when the peripheral assistant control device 212 is at the indicated location. In other words, in some embodiments, the data used to characterize a touch input from the user 206 can be the same regardless of where the peripheral assistant control device 212 is located; however, depending on where the peripheral assistant control device 212 is located, the data can be responded to differently by another device (such as the server device 220).

[0057] Figure 3 A system 300 is shown for allocating control of one or more executable assistant functions to a peripheral assistant control device based on whether one or more different conditions are met. The automated assistant 306 can operate as part of an automated assistant application provided at one or more computing devices (such as the client device 334 and / or the server device 302). A user can interact with the automated assistant 308 via one or more assistant interfaces 336, which can include one or more of a microphone, a camera, a touchscreen display, a user interface, and / or any other device capable of providing an interface between the user and the application. For example, a user can initialize the automated assistant 308 by providing verbal, text, or graphical input to the assistant interface to cause the automated assistant 308 to perform functions (such as providing data, controlling peripheral devices, accessing agents, etc.). The client device 334 and / or the peripheral assistant control device 342 can include a display device, which can be a display panel including a touch interface for receiving touch inputs and / or gestures to allow the user to control applications of the client device 334 and / or the server device 302 via the touch interface.

[0058] In some embodiments, the client device 334 and / or the peripheral assistant control device 342 may not have a display device, but include an audio interface (e.g., speakers and / or microphones) to provide audible user interface output without providing graphical user interface output, and provide a user interface (such as a microphone) for receiving spoken natural language input from the user. For example, in some embodiments, the peripheral assistant control device 342 may include one or more tactile input interfaces (such as one or more buttons) and omit a display panel that provides graphical data from a graphics processing unit (GPU). In this way, compared with a computing device including a display panel and a GPU, a significant amount of energy and processing resources can be saved.

[0059] The client device 334 may communicate with the server device 302 via a network 346 (such as the Internet). The client device 334 may offload computing tasks to the server device 302 in order to conserve computing resources at the client device 334 and / or the peripheral assistant control device 342. For example, the server device 302 may host the automatic assistant 308, and the client device 334 may transmit inputs received at one or more assistant interfaces and / or the user interface 344 of the peripheral assistant control device 342 to the server device 302. However, in some embodiments, the automatic assistant 308 may be hosted at the client device 334. In various embodiments, all or less than all aspects of the automatic assistant 308 may be implemented on the server computing device 304 and / or the client device 334. In some of these embodiments, aspects of the automatic assistant 308 are implemented via the local automatic assistant 338 of the client device 334 and interface with the server device 302, which may implement other aspects of the automatic assistant 308. The server device 302 may optionally serve multiple users and their associated assistant applications via multithreading. In embodiments where all or less than all aspects of the automatic assistant 308 are implemented via the local automatic assistant 338 of the client device 334, the local automatic assistant 338 may be an application separate from the operating system of the client device 334 (e.g., installed "on top" of the operating system) - or alternatively may be directly implemented by the operating system of the client device 334 (e.g., considered an application within the operating system but integrated with the operating system).

[0060] In some embodiments, the Auto-Assistant 308 and / or the Auto-Assistant 338 may include an Input Processing Engine 312, which may employ a plurality of different modules to process the input and / or output of the client device 334. For example, the Input Processing Engine 312 may include a Speech Processing Module 314, which may process the audio data received at the Assistant Interface 336 to identify the text contained in the audio data. The audio data may be transmitted from, for example, the client device 334 to the server device 302 in order to conserve the computing resources at the client device 334.

[0061] The process for converting the audio data into text may include a speech recognition algorithm, which may employ a neural network, a word2vec algorithm, and / or a statistical model to identify the audio data sets corresponding to words or phrases. The text converted from the audio data may be parsed by a Data Parsing Module 316 and made available as text data that can be used by the Auto-Assistant 308, and the text data may be used to generate and / or identify command phrases from the user. In some embodiments, the output data provided by the Data Parsing Module 316 may be provided to an Action Engine 318 to determine whether the user has provided an input corresponding to a particular action and / or routine that can be performed by the Auto-Assistant 308 and / or by an application or agent accessible by the Auto-Assistant 308. For example, Assistant Data 322 may be stored as Client Data 340 at the server device 302 and / or the client device 334, and may include data for defining one or more actions that can be performed by the Auto-Assistant 308, as well as the parameters involved in performing these actions.

[0062] When the Input Processing Engine 312 has determined that the user has requested the execution of a particular action or routine, the Action Engine 318 may determine one or more parameters for the particular action or routine, and then an Output Generation Engine 320 may provide an output to the user based on the particular action, routine, and / or one or more parameters. For example, in some embodiments, in response to a user input (such as a gesture at the user interface 344 of the Peripheral Assistant Control Device 342), the Auto-Assistant 338 may cause the data characterizing the gesture to be transmitted to the server device 302 for determining the action that the user wants the Auto-Assistant 308 and / or the Auto-Assistant 338 to perform.

[0063] System 300 may include one or more peripheral assistant control devices 342, and each peripheral assistant control device 342 may include one or more user interfaces 344. The user interface 344 may include one or more sensors capable of providing output in response to user input from a user. For example, the one or more sensors may include a haptic sensor (i.e., a touch sensor) that provides an output signal in response to a haptic gesture from a user. The output signal may be transmitted to the client device 334 and / or the server device 302 via a communication protocol such as Bluetooth, LTE, Wi-Fi, and / or any other communication protocol. In some embodiments, the output signal from the one or more sensors of the user interface 344 may be converted into user input data by one or more processors of the peripheral assistant control device 342. The user input data may be transmitted to the client device 334 via a communication protocol and / or processed at the client device 334 and / or transmitted to the server device 302 via the network 346. For example, the client device 334 and / or the server device 302 may determine a function to be controlled by the peripheral assistant control device 342 based on the user input data.

[0064] In some limitations, determining the function to be controlled by the peripheral assistant control device 342 may be based on the identity of the client device 334 with which the peripheral assistant control device 342 is paired or otherwise communicates. In other words, the client device 334 and / or the server device 302 will determine different functions for the peripheral assistant control device 342 to control based on the particular client device to which the peripheral assistant control device 342 is connected. In some embodiments, assigning control of a particular function to the peripheral assistant control device 342 may be based on the physical characteristics of the client device 334. For example, when the peripheral assistant control device 342 is paired with a device including a touch display panel, the peripheral assistant control device 342 may be assigned control of a first function. Additionally, when the peripheral assistant control device 342 is paired with a device that does not include a touch display panel, the peripheral assistant control device 342 may be assigned control of a second function different from the first function. Alternatively or additionally, when the peripheral assistant control device 342 is paired with a device identified by an account of a first user, the peripheral assistant control device 342 may be assigned control of a first function. Additionally, when the peripheral assistant control device 342 is paired with a different device identified by a different account of a second user different from the first user, the peripheral assistant control device 342 may be assigned control of a second function.

[0065] In some embodiments, a request from a user to assign control of a function to the peripheral assistant control device 342 can be extended so that the peripheral assistant control device 342 can have control over additional functions not explicitly identified in the request from the user. For example, a user can provide an oral utterance such as "Assistant, go to the next song in my playlist when I tap the peripheral assistant control device". The oral utterance can be received at the assistant interface 336, converted into audio data by the client device 334, and transmitted to the server device 302 for further processing. By using the audio data, the automated assistant 308 can determine that the user is requesting that the peripheral assistant control device 342 have control over the function of skipping the currently playing song from the playlist.

[0066] Thereafter, when the user is listening to music from a playlist and presses the user interface 344 of the peripheral assistant control device 342, the automated assistant 308 can cause the playback of the playlist to transition from the current song to the next song in the playlist. In some embodiments, the peripheral assistant control device 342 can provide data in response to receiving a touch input at the user interface 344. The data can be transmitted to the client device 334 and then to the server device 302 for further processing. The server device 302 can include a control device module 324 that can determine how the automated assistant 308 and / or any other application associated with the server device 302 and / or the client device 334 will respond to the user providing input to the peripheral assistant control device 342.

[0067] The control device module 324 may include a gesture processing engine 326 that receives data from a peripheral assistant control device 342 in response to input from a user and determines the type of gesture provided by the user. For example, the data may be processed by the gesture processing engine 326 to determine whether the input gesture is a single tap, double tap, multiple taps, long tap, hard press, soft press, and / or any other distinguishable gesture that may be provided to the interface of the computing device. In some embodiments, the control device module 324 may access and / or receive a quantity of client data 340 that is used to indicate the presence of one or more users within an area associated with the peripheral assistant control device 342 and / or the client device 334. In this way, the control device module 324 may determine the function that the output at the peripheral assistant control device 342 will affect based on the received touch input and the one or more users determined to be present. For example, the control device module 324 may include a function selection engine 330 that may access the client data 340 for indicating the presence of one or more users and may also access correlation data 332 that may provide a correspondence between the functions to be controlled by the peripheral assistant control device 342 and the users indicated as present when the user interface 344 receives input. In other words, the correlation data 332 will characterize the settings for controlling functions requested by the user via the user interface 344 based on whether the user is present. Thus, if such a setting is characterized by the correlation data 332 and the client data 240 indicates that the user is present when the user interface 344 receives input, the corresponding function identified by the correlation data 332 may be controlled in response to the input.

[0068] The control of a function may be defined as performing the function, modifying one or more instances of the function performed by one or more devices, modifying one or more instances of the function performed by one or more applications, and / or any other control that may be exhibited over the function and / or the performance of the function. In some embodiments, the control of a function may be defined as providing a value for a parameter of the function, modifying the value of a parameter of the function, and / or otherwise modifying the data used in performing the function. For example, the function of skipping a song in a playlist may include a parameter for providing the device name. Thus, when the control of a function is assigned to the peripheral assistant control device 342, the control device module 324 may cause the performance of the function in response to a touch input at the user interface 344 and may also provide the device name for the parameter of the function so that the function may be performed without error.

[0069] In some embodiments, when a user requests that the peripheral assistant control device 342 control the function of skipping to the next song in a playlist, one or more other functions associated with the request from the user may be assigned to the peripheral assistant control device 342. For example, in response to processing audio data corresponding to an oral utterance, the server device 302 may identify the function for which the user requests that the peripheral assistant control device 342 be assigned control. The function may be associated with one or more applications and / or one or more devices, and each application and / or device may exhibit one or more different operating states (i.e., operating modes) at a given time. For example, when an application that performs playback of a song in a playlist causes a particular computing device to stream music from the playlist, the application may be in a "playback" state. The control device module 324 may identify this state and correlate this state with the "skip" function for which the user is requesting that the peripheral assistant control device 342 have control via the correlation data 332. In other words, the control device module 324 will operate the peripheral assistant control device 342 such that when the application is in the "playback" state, the peripheral assistant control device 342 will perform the "skip" function in response to a single click gesture provided by the user at the user interface 344.

[0070] Alternatively or additionally, the function selection engine 330 of the control device module 324 may identify other functions associated with the application and / or other states of the application. For example, the application may have a "play" function and / or an "open" function, and the application may also have a "closed" state as well as an "open" and / or "play" state. Based on this information, the control device module 324 may generate correlation data 332 that correlates the open function of the application with the closed state and correlates the skip function of the application with the play state. As a result, and in response to a request from the user, providing a touch input to the user interface 344 of the peripheral assistant control device 242 when the application is in the closed state will cause the application to start playing music. Additionally, providing a touch input when the application is in the play state will cause the application to skip the currently playing song. In this way, although the user only requests that the peripheral assistant control device 242 control a single function, the peripheral assistant control device 342 will be assigned multiple functions to control. This can eliminate the need for the user to provide additional oral utterances to request that the peripheral assistant control device 342 control multiple other functions. This can save computing resources and / or network resources because the audio data corresponding to the additional oral utterances does not need to be generated and / or transmitted from the client device 334 to the server device 302. Instead, the peripheral assistant control device 342 may be assigned control of multiple different functions that are not identifiable from the user's oral utterance or not specifically requested by the user.

[0071] In some embodiments, the control device module 324 may generate correlation data 332 that correlates one or more functions to be controlled by the peripheral assistant control device 342 with one or more conditions. For example, the function selection engine 330 may allocate control of a particular function based on whether a particular condition is met. The control device module 324 may determine whether the condition is met based on the assistant data 322, the client data 340, and / or any other data that may be transmitted to the client device 334 and / or the server device 302. For example, a particular condition may be met when a first user is determined to be located within a definable area that includes the peripheral assistant control device. Thus, when the first user is determined to be within the definable area, the function selection engine 330 may allocate control of the function to the peripheral assistant control device 342 based on the particular condition being met. Alternatively or additionally, another condition may be met when the peripheral assistant control device 342 is determined to be located within another definable area. Thus, when the peripheral assistant control device 342 is determined to be within the other definable area, the function selection engine 330 may allocate control of another function to the peripheral assistant control device 342 based on the other condition being met.

[0072] Figure 4A and Figure 4B illustrates methods 400 and 412 for allocating control of a particular function to a peripheral assistant control device based on whether one or more conditions are met. In some embodiments, method 400 may be performed according to the operations identified in Figure 4A optionally including the operations identified in Figure 4B In some embodiments, method 412 may be performed according to the operations identified in Figure 4B optionally including the operations identified in Figure 4A Each of methods 400 and 412 may be performed by one or more devices, one or more applications, and / or any other means or module capable of operating according to a computer process.

[0073] Method 400 may include an operation 402 of receiving audio data corresponding to an oral utterance provided by a user (or other data provided as input to an application via a modality such as a keyboard, a mouse, a touch display, and / or any other device for providing input to a computing device). The oral utterance (or other data) may be a request for an automatic assistant to assign control of a first automatic assistant function and a second automatic assistant function to a peripheral assistant control device. For example, the oral utterance may be "Assistant, give me the news when I press the button and play music when my husband presses the button". Alternatively, the oral utterance may be multiple oral utterances provided by different users. For example, a first user in a household may provide an oral utterance such as "Assistant, play music when I press the button", while a second user in the household may provide another oral utterance such as "Assistant, give me the news when I press the button".

[0074] Method 400 may further include an operation 404 of determining that the request is a request from the user to set a peripheral assistant control device to perform one or more functions. For example, the audio data generated based on the oral utterance may be processed to identify an identifier of the peripheral assistant control device and one or more functions. At least based on the content of the audio data, it may be determined that the user is requesting that the peripheral assistant control device be assigned control of one or more functions. Method 400 may further include an operation 406 of identifying, based on the audio data, one or more functions requested by the user. For example, the one or more functions may include a message playback function and / or a music playback function. In some embodiments, the automatic assistant may be associated with multiple different functions that can be performed by the automatic assistant and / or via a separate application or a separate assistant agent.

[0075] Method 400 may further include operation 408: identifying one or more other functions associated with one or more functions requested by a user. The one or more other functions may be identified based on any data used to characterize the association or correspondence between the one or more functions and the one or more other functions. For example, when the one or more functions correspond to a specific application, the one or more other functions associated with the application may be identified via operation 408. Alternatively or additionally, when the one or more functions correspond to a specific device, the one or more other functions associated with the device may be identified via operation 408. Alternatively or additionally, when a particular user has previously used the one or more functions and has also used the one or more other functions at least as frequently as the one or more functions, the one or more other functions may be identified via operation 408. In this way, although the user does not explicitly identify the one or more other functions in their verbal discourse, the other functions may be identified to assign control of those other functions to the peripheral assistant control device.

[0076] Method 400 also includes operation 410: generating correlation data that causes the output of the peripheral assistant control device to be related to the one or more functions when a first condition is met and to the one or more other functions when a second condition is met. In some embodiments, the user may characterize the first condition and / or the second condition via verbal discourse so that the user may have exclusive control over when the peripheral assistant control device will control a particular function among the one or more functions. Alternatively or additionally, one or more conditions may be identified without being explicitly requested by the user so that the peripheral assistant control device can adapt to various scenarios in which the automatic assistant predicts or anticipates that the peripheral assistant control device will be used. For example, when the verbal discourse includes a request from the user to request the peripheral assistant control device to control in their kitchen, receiving the verbal discourse may characterize various conditions based on other areas within the user's residence. Specifically, correlation data may be generated that characterizes a first condition under which the peripheral assistant control device's control of the kitchen lights is based on the peripheral assistant control device being in the kitchen. Additionally, the correlation data may also characterize a second condition under which the peripheral assistant control device's control of the living room lights is based on the peripheral assistant control device being in the living room.

[0077] Method 400 may proceed from operation 410 to operation 414, as in Figure 4A and Figure 4BAs indicated by the continuation element "A" shown in [description]. Operation 414 includes determining whether a touch input has been received at the peripheral assistant control device. Operation 414 can be determined periodically until the user provides a touch input to the peripheral assistant control device. The touch input can be one or more physical gestures, such as a click, double-click, multiple clicks, long press, pull, press rhythm, and / or any other physical gesture that can be received at the user interface. In response to determining that a touch input has been received at the peripheral assistant control device, method 412 can proceed to operation 416.

[0078] Operation 416 can include determining whether a first condition or a second condition has been met. Each of the first condition and the second condition can be any condition that can be characterized by data stored at the computing device. For example, the first condition can be met when the user is determined to be located within the area including the peripheral assistant control device, and the second condition can be met when a second user is determined to be located within the area including the peripheral assistant control device. Alternatively or additionally, the first condition can be met when the peripheral assistant control device is determined to be located within a first area, and the second condition can be met when the assistant control device is determined to be located within a second area.

[0079] When the first condition is met, method 412 can proceed to operation 418, which can include performing one or more functions in response to the user interacting with the touch interface. When the second condition is met, method 412 can proceed to operation 420, which can include performing one or more functions in response to the user interacting with the touch interface. In this way, although the same touch input is received - regardless of whether the first condition or the second condition is met, this same touch input can result in different outcomes, depending on whether the first condition or the second condition is met. This can reduce latency in the setup device, which might otherwise occur if the user had to explicitly request each specific condition and each specific function through verbal utterances or other interactions with the computing device. By eliminating the need for the user to provide such additional verbal utterances or interactions, computing resources and network resources can be conserved for any computing device responsible for establishing the setup for the peripheral assistant control device.

[0080] Figure 5is a block diagram of an example computer system 510. The computer system 510 generally includes at least one processor 514 communicating with a plurality of peripheral devices via a bus subsystem 512. These peripheral devices can include a storage subsystem 524 (including, for example, a memory 525 and a file storage subsystem 526), a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computer system 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0081] The user interface input device 522 can include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system), a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computer system 510 or onto a communication network.

[0082] The user interface output device 520 can include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem can include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for forming a visual image. The display subsystem can also provide a non-visual display, for example, via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways of outputting information from the computer system 510 to a user or to another machine or computing device.

[0083] The storage subsystem 524 stores programming and data constructs for providing the functionality of some or all of the modules described herein. For example, the storage subsystem 524 can include logic for performing selected aspects of method 400, method 412, and / or the first client device 110, the second client device 114, the peripheral assistant control device 112, the server device 120, the automated assistant 122, the first client device 210, the second client device 214, the peripheral assistant control device 212, the server device 220, the automated assistant 222, the system 300, the automated assistant 308, the server device 302, the control device module 324, the client device 334, and / or the peripheral assistant control device 342.

[0084] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 for storing fixed instructions therein. The file storage subsystem 526 may provide permanent storage for program and data files and may include a hard disk drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical disk drive or a removable media cartridge. Modules for implementing the functions of certain embodiments may be stored in the storage subsystem 524 by the file storage subsystem 526 or in other machines accessible by the processor 514.

[0085] The bus subsystem 512 provides a mechanism for enabling the various components and subsystems of the computer system 510 to communicate with each other as expected. Although the bus subsystem 512 is schematically shown as a single bus, alternative embodiments of the bus subsystem may use multiple buses.

[0086] The computer system 510 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 5 the description of the computer system 510 depicted herein is only intended as a specific example for the purpose of illustrating some embodiments. Many other configurations of the computer system 510 may have more or fewer components than the Figure 5 computer system depicted herein.

[0087] In cases where the systems described herein collect personal information about a user (or what is commonly referred to herein as a "participant") or may make use of personal information, users may be provided with an opportunity to control whether programs or features collect user information (e.g., information about the user's social network, social behavior or activities, profession, user preferences or the user's current geographical location), or to control whether and / or how content that may be more relevant to the user is received from a content server. Also, certain data may be processed in one or more ways before being stored or used so as to remove personally identifiable information. For example, a user's identity may be processed so that personally identifiable information cannot be determined for the user, or the user's geographical location may be generalized (such as to the city level, ZIP code level or state level) in cases where geographical location information is obtained so that the user's specific geographical location cannot be determined. Thus, the user may control how information about that user is collected and / or used.

[0088] Although several embodiments have been described and illustrated herein, various other devices and / or structures may be utilized for performing the functions and / or obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is considered to be within the scope of the embodiments described herein. More generally, all parameters, dimensions, materials, and configurations described herein are intended to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend upon the particular application in which the teachings are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific embodiments described herein. Accordingly, it should be understood that the foregoing embodiments are presented by way of example only, and that within the scope of the appended claims and their equivalents, embodiments may be practiced otherwise than as specifically described and claimed. Embodiments of the present disclosure relate to each and every individual feature, system, article, material, kit, and / or method described herein. Furthermore, any combination of two or more such features, systems, articles, materials, kits, and / or methods (if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent) is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors, the method comprising: In response to a user providing one or more verbal utterances that identify only a first automatic assistant function, assign the first automatic assistant function and a second automatic assistant function to a peripheral assistant control device; Receive, at a computing device and via a wireless communication channel, interaction data from the peripheral assistant control device, the interaction data being generated by the peripheral assistant control device in response to a touch input provided by the user at a touch interface of the peripheral assistant control device; Determine, based on data accessible via the computing device, a current operating state of an application controlled by the computing device; In response to receiving the interaction data: Select a specific function based on the current operating state and from at least the first automatic assistant function and the second automatic assistant function, wherein the selection includes: When the current operating state is a first state, select the first automatic assistant function as the specific function, and When the current operating state is a second state different from the first state, select the second automatic assistant function as the specific function; and Cause an automatic assistant accessible via the computing device to perform the specific function.

2. The method according to claim 1, wherein the first automatic assistant function is also capable of being executed in response to a corresponding first verbal request for the automatic assistant interface of the computing device, and the second automatic assistant function is also capable of being executed in response to a corresponding second verbal request for the automatic assistant interface of the computing device.

3. The method according to claim 1, wherein the touch interface of the peripheral assistant control device includes touch sensors that respond to one or more haptic gestures.

4. The method according to claim 3, wherein the peripheral assistant control device provides the interaction data only in response to an interaction with the touch interface, and the touch interface responds only to the one or more haptic gestures.

5. The method according to claim 1, wherein the application executes the playing of a song, and wherein the first state is the playing state.

6. The method according to claim 5, wherein, The first automatic assistant function, when executed, causes the application to skip to the next song.

7. The method according to claim 6, wherein, The second function, when executed, causes the application to start playing music.

8. The method according to claim 1, wherein, The peripheral assistant control device lacks a microphone.

9. The method according to claim 8, wherein, The peripheral assistant control device lacks a Wi-Fi module.

10. A method implemented by one or more processors, the method comprising: Determine, based on data accessible via the computing device, whether the peripheral assistant control device is connected to a first client device or a second client device, wherein the peripheral assistant control device communicates wirelessly with the computing device, and the automatic assistant is accessible via the computing device; Receive interaction data from the peripheral assistant control device, the interaction data being generated in response to a touch input received at a touch interface of the peripheral assistant control device; In response to receiving the interaction data: Select a specific function from at least a first automatic assistant function and a second automatic assistant function based on whether the peripheral assistant control device is connected to the first client device or the second client device, wherein the selection includes: When the peripheral assistant control device is connected to the first client device, select the first automatic assistant function as the specific function, and When the peripheral assistant control device is connected to the second client device, select the second automatic assistant function as the specific function; and Cause the automatic assistant to perform the specific function.

11. The method according to claim 10, wherein the first automatic assistant function is also capable of being executed in response to a corresponding first verbal request for the automatic assistant interface of the automatic assistant, and the second automatic assistant function is also capable of being executed in response to a corresponding second verbal request for the automatic assistant interface of the automatic assistant.

12. The method according to claim 10, further comprising: In response to the user providing one or more verbal utterances that identify the first automatic assistant function and the second automatic assistant function, assign the first automatic assistant function and the second automatic assistant function to the peripheral assistant control device.

13. The method according to claim 10, wherein the touch interface of the peripheral assistant control device includes touch sensors that respond to one or more haptic gestures.

14. The method according to claim 13, wherein the peripheral assistant control device provides the interaction data only in response to an interaction with the touch interface, and the touch interface responds only to the one or more haptic gestures.

15. A system, comprising: A computing device; A peripheral assistant control device that communicates with the computing device via a wireless communication channel, wherein the peripheral assistant control device includes a touch interface, and wherein the peripheral assistant control device transmits, via the wireless communication channel, interaction data in response to a touch input at the touch interface; wherein the computing device includes one or more processors configured to execute the following instructions: in response to one or more verbal utterances that identify only a first auto-assistant function, assign the first auto-assistant function and a second auto-assistant function to the peripheral assistant control device; receive the interaction data via the wireless communication channel; determine a current operating state of an application controlled by the computing device; in response to receiving the interaction data: select a specific function based on the current operating state and from at least the first auto-assistant function and the second auto-assistant function, wherein when selecting the current operating state the one or more processors: when the current operating state is a first state, select the first auto-assistant function as the specific function, and when the current operating state is a second state different from the first state, select the second auto-assistant function as the specific function; and cause an auto-assistant accessible via the computing device to execute the specific function.

16. A system, comprising: a memory storing the instructions; one or more processors operable to execute the instructions to: receive natural language input provided by a user to an auto-assistant interface of a computing device, wherein the computing device provides access to an auto-assistant and is connected to a peripheral assistant control device; determine, based on the natural language input, that the natural language input includes a request to control one or more functions in response to an interaction with a touch interface of the peripheral assistant control device, wherein the peripheral assistant control device provides an output to the computing device or another computing device in response to the interaction with the touch interface; determine, based on the natural language input, that the natural language input at least partially identifies the one or more functions related to the output of the peripheral assistant control device; identify, based on data associated with the one or more functions, one or more other functions executable via the computing device or the auto-assistant, wherein the natural language input does not specify the content of the one or more other functions; and generate relevance data that: relates the output of the peripheral assistant control device to the one or more functions when at least a first condition is met, and relates the output of the peripheral assistant control device to the one or more other functions when at least a second condition is met.

17. The system according to claim 16, wherein the one or more processors are further operable to execute the instructions to: After generating the correlation data, it is determined that the peripheral assistant control device has provided the output in response to the user interacting with the touch interface; determine whether the first condition or the second condition is met based on other data associated with the user; when the first condition is met: in response to the user interacting with the touch interface of the peripheral assistant control device, cause the one or more functions to be executed; and when the second condition is met: in response to the user interacting with the touch interface of the peripheral assistant control device, cause the one or more functions to be executed.

18. The system according to claim 16, wherein the one or more functions include a start media action for causing the presentation of media, wherein the first condition is satisfied when the media is not presented at the computing device and / or another device associated with the user, and wherein the second condition is satisfied when the media is presented at the computing device or the other device associated with the user.

19. The system according to claim 18, wherein the one or more other functions include a next media item action for causing the presentation of a new media item.

20. The system according to claim 16, wherein the one or more processors are further operable to execute the instructions to: generate suggestions characterizing the one or more other functions; cause the suggestions to be presented in response to receiving the natural language input; and receive an affirmative user interface input in response to the suggestion; The generating of the correlation data is in response to receiving the affirmative user interface input, and the correlation data correlates the output of the peripheral assistant control device with the one or more other functions when at least a second condition is met.

21. The system according to claim 16, wherein generating association data to associate the output of the peripheral assistant control device with one or more other functions when at least a second condition is satisfied is performed automatically without first suggesting the one or more other functions to the user.

22. The system according to claim 21, wherein, The one or more functions include a start media action that causes the presentation of media. Wherein, the first condition is met when media is not presented on the computing device and / or another device associated with the user, and wherein, the second condition is met when media is presented on the computing device or the other device associated with the user.

23. The system according to claim 22, wherein, The one or more other functions include a next media item action that causes the presentation of a new media item.

24. The system according to claim 16, wherein, The natural language input is an oral utterance.

25. The system according to claim 16, further comprising the peripheral assistant control device.

26. A method implemented by one or more processors, the method comprising: Receiving natural language input provided by a user to an auto assistant interface of a computing device, wherein the computing device provides access to an auto assistant and is connected to a peripheral assistant control device; Based on the natural language input, determining that the natural language input includes a request to control one or more functions in response to an interaction with the peripheral assistant control device, wherein the peripheral assistant control device provides an output to the computing device or another computing device in response to the interaction with the peripheral assistant control device; Based on the natural language input, determining that the natural language input at least partially identifies the one or more functions related to the output of the peripheral assistant control device; Based on data associated with the one or more functions, identifying one or more other functions executable via the computing device or the auto assistant, wherein the natural language input does not specify the content of the one or more other functions; And Generating correlation data, the correlation data: When at least a first condition is met, correlates the output of the peripheral assistant control device with the one or more functions, and When at least a second condition is met, correlates the output of the peripheral assistant control device with the one or more other functions.

27. The method according to claim 26, further comprising: After generating the correlation data, determining that the peripheral assistant control device has provided the output in response to the user interacting with the peripheral assistant control device; Based on other data associated with the user, determining whether the first condition or the second condition is met; When the first condition is met: In response to the user interacting with the peripheral assistant control device, causing the one or more functions to be executed; And When the second condition is met: In response to the user interacting with the peripheral assistant control device, causing the one or more functions to be executed.

28. The method according to claim 26, wherein the one or more functions include a start media action for causing the presentation of media, wherein the first condition is satisfied when the media is not presented at the computing device and / or another device associated with the user, and wherein the second condition is satisfied when the media is presented at the computing device or the other device associated with the user.

29. The method according to claim 28, wherein the one or more other functions include a next media item action for causing the presentation of a new media item.

30. The method according to claim 26, further comprising: Generating a suggestion characterizing the one or more other functions; In response to receiving the natural language input, causing the suggestion to be presented; And Receiving an affirmative user interface input in response to the suggestion; Wherein the generating of the correlation data is in response to receiving the affirmative user interface input, and the correlation data correlates the output of the peripheral assistant control device with the one or more other functions when at least a second condition is met.

31. The method according to claim 26, wherein generating association data to associate the output of a peripheral assistant control device with one or more other functions when at least a second condition is satisfied is performed automatically without first suggesting the one or more other functions to the user.

32. The method according to claim 31, wherein,The one or more functions include a start media action that causes the presentation of media. Wherein, when the media is not presented on the computing device and / or another device associated with the user, the first condition is satisfied, and Wherein, when the media is presented on the computing device or the other device associated with the user, the second condition is satisfied.

33. According to the method of claim 32, wherein, The one or more other functions include a next media item action that causes the presentation of a new media item.

34. According to the method of claim 26, wherein, The natural language input is an oral utterance.

Images