Group Hotwords

Abstract

The method includes receiving, by a first assistant-enabled device (AED), an assignment instruction that assigns a group hotword to a selected group of AEDs, including the first AED and one or more other AEDs. Each AED is configured to wake up from a low power state when the group hotword is detected in streaming audio by at least one of the AEDs. The method also includes receiving audio data that corresponds to an utterance spoken by a user and includes a query that specifies an action to be performed. In response to detecting the group hotword in the audio data, the method also includes triggering the first AED to wake up from a low power state and executing a collaboration routine that causes the first AED and each of the other AEDs in the selected group to cooperate with each other to accomplish the execution of the action specified by the query.

Description

[Technical field]

[0001] The present disclosure relates to group hot words. [Background technology]

[0002] A voice-enabled environment (e.g., home, work, school, automobile, etc.) allows a user to speak queries or commands aloud to a computer-based system that responds and answers the query and / or performs a function based on the command. A voice-enabled environment may be implemented using a network of connected microphone devices located throughout various rooms or areas of the environment. These devices may use hot words to help distinguish when a given utterance is directed to the system as opposed to an utterance directed to another individual present in the environment. Thus, the device may operate in a sleep or hibernation state and wake up only if a detected utterance contains the hot word. Once woken up, the device can go about performing more costly processing such as fully on-device automated speech recognition (ASR) or server-based ASR. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the disclosure provides a method for enabling and assigning a group hotword to two or more selected groups of assistant-enabled devices. The method includes receiving, in data processing hardware of a first assistant-enabled device (AED), an assignment instruction that assigns the group hotword to the selected group of AEDs associated with a user. The selected group of AEDs includes the first AED and one or more other AEDs. Each AED in the selected group of AEDs is configured to wake up from a low power state when the group hotword is detected in streaming audio by at least one of the AEDs in the selected group of AEDs. The method also includes receiving, in the data processing hardware, audio data corresponding to an utterance spoken by the user and including a query that specifies an action to be performed. In response to detecting the group hotword in the audio data, the method includes triggering, by the data processing hardware, the first AED to wake up from a low power state. The method also includes executing, by the data processing hardware, a collaboration routine that causes the first AED and each other AED in the selected group of AEDs to collaborate with each other to accomplish the execution of the action specified by the query.

[0004] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the method further includes performing, by the data processing hardware, a leader selection process to select, from the AEDs of the selected group, one or more AEDs to listen for the presence of the group hotword in the streaming audio on behalf of the AEDs of the selected group based on respective device characteristics associated with each AED in the selected group, wherein at least a first AED in the AEDs of the selected group is selected by the leader selection process to listen for the presence of the group hotword in the streaming audio. In these implementations, the method may include re-performing, by the data processing hardware, the leader selection process in response to at least one of: updating the AEDs of the selected group that have been assigned the group hotword, adding one or more additional AEDs to the AEDs of the selected group and / or removing one or more of the AEDs from the AEDs of the selected group, or a device state change occurring in one of the AEDs of the AEDs of the selected group. Optionally, each device characteristic associated with each AED in the selected group of AEDs may include at least one of processing capability, device type, user configurable device settings, power usage, battery level, physical location of the AED, or network capabilities.

[0005] In response to detecting the group hotword in the audio data, the method may further include invoking, by the data processing hardware, each other AED in the selected group of AEDs that did not detect the group hotword in the audio data to wake up from a low power state and cooperate with all AEDs in the selected group of AEDs to perform the operation specified by the query. In some implementations, in response to detecting the group hotword in the audio data, the method further includes identifying, by the data processing hardware, each of the one or more other AEDs in the selected group of AEDs to which the group hotword has been assigned by accessing a hotword registry. The hotword registry includes a list of one or more hotwords each assigned to one or more AEDs associated with a user (or a group of users), the list of one or more hotwords including the group hotword. In these implementations, invoking each other AED in the selected group of AEDs is based on each of the one or more other AEDs to which the group hotword has been assigned identified by accessing the hotword registry. In some embodiments, the hotword registry is stored in at least one of the first AED, at least one other AED associated with the user, or a server in communication with the first AED.

[0006] Optionally, receiving an assignment instruction to assign a group hotword to the selected group of AEDs may include receiving a voice input from a user requesting the digital assistant to activate the group hotword and assign the group hotword to the selected group of AEDs, instructing a voice recognizer to perform voice recognition on the audio data to generate an automated speech recognition (ASR) result for the voice input, and performing query interpretation on the ASR result for the voice input to identify a group hotword that specifies the name of the group hotword to enable each AED in the selected group of AEDs to be assigned to the group hotword. In some implementations, receiving an assignment instruction to assign a group hotword to the selected group of AEDs includes receiving a user input instruction indicating a user interaction with one or more objects displayed in a graphical user interface to instruct the digital assistant to activate the group hotword and enable each AED in the selected group of AEDs to be assigned the group hotword.

[0007] The method may further include receiving, in the data processing hardware, previous audio data spoken by the user corresponding to a command for the digital assistant to perform a long-lived action at two or more AEDs associated with the user. The digital assistant is configured to automatically create a group hotword for use in follow-up queries related to the long-lived action, where receiving an assignment instruction assigning the group hotword to the selected group of AEDs includes receiving a group hotword automatically created by the digital assistant. The selected group of AEDs to which the group hotword is assigned includes two or more AEDs performing the long-lived action. In some implementations, the method further includes outputting, by the data processing hardware, for audible playback from the first AED, a response from the digital assistant indicating that execution of the long-lived action is in progress, and a synthesized voice corresponding to the automatically created group hotword for use in follow-up queries related to the long-lived action. In these implementations, the digital assistant cancels use of the automatically created group hotword when the long-lived action is terminated. In some embodiments, each AED in the selected group of AEDs is associated with the same device type or shares a common attribute, the group hotwords assigned to the AEDs in the selected group include implicit hotwords that uniquely identify the device types or common attributes associated with the AEDs in the selected group, and the hotword detection model is pre-trained to detect the presence of the group hotwords in streaming audio without performing speech recognition on the audio data.

[0008] In some implementations, the group hotwords assigned to the AEDs of the selected group include proximity-based hotwords that are directed to only a subset of one or more AEDs from the selected group of AEDs that are currently closest in proximity to the user without explicitly identifying any particular AED among the subset of one or more AEDs. The hotword detection model is pre-trained to detect the presence of the group hotword in the streaming audio without performing voice recognition on the audio data, and executing the collaboration routine causes the first AED and each other AED among the selected group of AEDs to each determine a respective proximity value to the user and perform arbitration using the respective proximity values ​​among the AEDs of the selected group to select a subset of one or more AEDs that should perform the execution of the action specified by the query. Optionally, the group hotwords may include one of a custom hotword created by the user or a predefined hotword selected from a list of one or more available group hotwords. During execution of the collaboration routine, the first AED and each other AED in the selected group of AEDs collaborate with one another by designating one of the AEDs in the selected group of AEDs to generate a voice recognition result on the audio data, perform a query interpretation on the voice recognition result to determine that the voice recognition result identifies a query that specifies an action to be performed, and share the query interpretation performed on the voice recognition result with the other AEDs in the selected group of AEDs.

[0009] In some implementations, the query specifying the operation to be performed includes a query for the AEDs of the selected group to perform the long-lived operation, and during execution of the collaboration routine, each AED in the selected group of AEDs cooperates with each other by pairing with each other for the duration of the long-lived operation and coordinating the execution of sub-actions related to the long-lived operation. The operation specified by the query may include a device-level operation to be performed at each AED in the selected group of AEDs, and during execution of the collaboration routine, each AED in the selected group of AEDs cooperates with each other by independently pursuing the execution of the device-level operation.

[0010] Another aspect of the disclosure provides a first assistant-enabled device (AED) for activating and assigning a group hotword to two or more selected groups of assistant-enabled devices. The first AED includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed in the data processing hardware, cause the data processing hardware to perform an operation. The operation includes receiving an assignment instruction that assigns the group hotword to the selected group of AEDs associated with a user. The selected group of AEDs includes the first AED and one or more other AEDs. Each AED in the selected group of AEDs is configured to wake up from a low power state when the group hotword is detected in streaming audio by at least one of the AEDs in the selected group of AEDs. The operation also includes receiving audio data that corresponds to an utterance spoken by the user and includes a query that specifies an operation to be performed. In response to detecting the group hotword in the audio data, the operation includes triggering the first AED to wake up from a low power state. The operations also include executing a collaboration routine that causes the first AED and each other AED in the selected group of AEDs to collaborate with each other to accomplish performance of the operation specified by the query.

[0011] Implementations of the present disclosure may include one or more of the following optional features: In some implementations, the operations further include performing a leader selection process to select one or more AEDs from the selected group of AEDs to listen for the presence of the group hotword in the streaming audio on behalf of the AEDs in the selected group based on respective device characteristics associated with each AED in the selected group of AEDs, where at least a first AED in the selected group of AEDs is selected by the leader selection process to listen for the presence of the group hotword in the streaming audio. In these implementations, the operations may include re-performing the leader selection process in response to at least one of: updating the AEDs in the selected group that have been assigned the group hotword to add one or more additional AEDs to the AEDs in the selected group and / or remove one or more of the AEDs from the AEDs in the selected group, or a device state change occurring in one of the AEDs in the selected group of AEDs. Optionally, each device characteristic associated with each AED in the selected group of AEDs may include at least one of processing capability, device type, user configurable device settings, power usage, battery level, physical location of the AED, or network capabilities.

[0012] In response to detecting the group hotword in the audio data, the operations may further include invoking each other AED in the selected group of AEDs that did not detect the group hotword in the audio data to wake up from a low power state and cooperate with all AEDs in the selected group of AEDs to perform the operation specified by the query. In some implementations, in response to detecting the group hotword in the audio data, the operations further include identifying each of the one or more other AEDs in the selected group of AEDs to which the group hotword has been assigned by accessing a hotword registry. The hotword registry includes a list of one or more hotwords each assigned to one or more AEDs associated with the user, the list of one or more hotwords including the group hotword. In these implementations, invoking each other AED in the selected group of AEDs is based on each of the one or more other AEDs to which the group hotword has been assigned identified by accessing the hotword registry. In some implementations, the hotword registry is stored in at least one of the first AED, at least one other AED associated with the user, or a server in communication with the first AED.

[0013] Optionally, receiving an assignment instruction to assign a group hotword to the selected group of AEDs may include receiving a voice input from a user requesting the digital assistant to activate the group hotword and assign the group hotword to the selected group of AEDs, instructing a voice recognizer to perform voice recognition on the audio data to generate an automated speech recognition (ASR) result for the voice input, and performing query interpretation on the ASR result for the voice input to identify a group hotword that specifies the name of the group hotword to enable each AED in the selected group of AEDs to be assigned to the group hotword. In some implementations, receiving an assignment instruction to assign a group hotword to the selected group of AEDs includes receiving a user input instruction indicating a user interaction with one or more objects displayed in a graphical user interface to instruct the digital assistant to activate the group hotword and enable each AED in the selected group of AEDs to be assigned the group hotword.

[0014] In some examples, the operations further include receiving previous audio data spoken by the user corresponding to a command for the digital assistant to perform a long-lived action at two or more AEDs associated with the user. The digital assistant is configured to automatically create a group hotword for use in follow-up queries related to the long-lived action, where receiving an assignment instruction that assigns the group hotword to the selected group of AEDs includes receiving a group hotword automatically created by the digital assistant. The selected group of AEDs to which the group hotword is assigned includes two or more AEDs performing the long-lived action. In some implementations, the operations further include outputting for audible playback from the first AED a response from the digital assistant indicating that execution of the long-lived action is in progress and a synthesized voice corresponding to the automatically created group hotword for use in follow-up queries related to the long-lived action. In these implementations, the digital assistant cancels use of the automatically created group hotword when the long-lived action ends. In some embodiments, each AED in the selected group of AEDs is associated with the same device type or shares a common attribute, the group hotwords assigned to the AEDs in the selected group include implicit hotwords that uniquely identify the device types or common attributes associated with the AEDs in the selected group, and the hotword detection model is pre-trained to detect the presence of the group hotwords in streaming audio without performing speech recognition on the audio data.

[0015] In some implementations, the group hotwords assigned to the AEDs of the selected group include proximity-based hotwords that are directed to only a subset of one or more AEDs from the selected group of AEDs that are currently closest in proximity to the user without explicitly identifying any particular AED among the subset of one or more AEDs. The hotword detection model is pre-trained to detect the presence of the group hotword in the streaming audio without performing voice recognition on the audio data, and executing the collaboration routine causes the first AED and each other AED among the selected group of AEDs to each determine a respective proximity value to the user and perform arbitration using the respective proximity values ​​among the AEDs of the selected group to select a subset of one or more AEDs that should perform the execution of the action specified by the query. Optionally, the group hotwords may include one of a custom hotword created by the user or a predefined hotword selected from a list of one or more available group hotwords. During execution of the collaboration routine, the first AED and each other AED in the selected group of AEDs collaborate with one another by designating one of the AEDs in the selected group of AEDs to generate a voice recognition result on the audio data, perform a query interpretation on the voice recognition result to determine that the voice recognition result identifies a query that specifies an action to be performed, and share the query interpretation performed on the voice recognition result with the other AEDs in the selected group of AEDs.

[0016] In some implementations, the query specifying the operation to be performed includes a query for the AEDs of the selected group to perform the long-lived operation, and during execution of the collaboration routine, each AED in the selected group of AEDs cooperates with each other by pairing with each other for the duration of the long-lived operation and coordinating the execution of sub-actions related to the long-lived operation. The operation specified by the query may include a device-level operation to be performed at each AED in the selected group of AEDs, and during execution of the collaboration routine, each AED in the selected group of AEDs cooperates with each other by independently pursuing the execution of the device-level operation.

[0017] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description that follows. Other aspects, features, and advantages will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0018] [Figure 1A] FIG. 1 is a schematic diagram illustrating a system for enabling and assigning group hotwords to selected groups of assistant-enabled devices. [Figure 1B] FIG. 1 is a schematic diagram illustrating a system for enabling and assigning group hotwords to selected groups of assistant-enabled devices. [Figure 1C] FIG. 1 is a schematic diagram illustrating a system for enabling and assigning group hotwords to selected groups of assistant-enabled devices. [Figure 2A] FIG. 13 is a schematic diagram illustrating an example graphical user interface for enabling and assigning group hotwords to assistant-enabled devices in different selected groups. [Figure 2B] FIG. 13 is a schematic diagram illustrating an example graphical user interface for enabling and assigning group hotwords to assistant-enabled devices in different selected groups. [Diagram 3] FIG. 10 is a schematic diagram illustrating an example leader selection process for selecting one or more assistant-enabled devices to listen for the presence of a group hotword in streaming audio. [Figure 4] FIG. 1 is a schematic diagram illustrating an example voice-enabled environment including multiple assistant-enabled devices located in various areas of the environment. [Diagram 5] FIG. 5 illustrates an example hotword registry that includes a list of hotwords each assigned to a different selected group of assistant-enabled devices disposed in the voice-enabled environment of FIG. [Figure 6] 10 is a flowchart illustrating an example configuration of operations for a method of enabling and assigning group hotwords to a selected group of assistant-enabled devices. [Figure 7] FIG. 1 illustrates an exemplary computing device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Like numbers in the various drawings indicate like elements.

[0020] Ideally, when conversing with a digital assistant interface, a user should be able to communicate as if the user were speaking to another individual via verbal requests directed to the user's assistant-enabled device running the digital assistant interface. The digital assistant interface provides these verbal requests to an automated speech recognizer that processes and recognizes the verbal requests so that actions can be performed. However, in practice, it is prohibitively expensive to continuously perform speech recognition on a resource-constrained voice-enabled device such as a smartphone or smartwatch, making it difficult for the device to respond to these verbal requests all the time.

[0021] To provide a user experience that supports always-on voice, an assistant-enabled device typically executes a compact hot word detection model configured to recognize audio features that characterize a narrow set of phrases that, when spoken by a user, initiates full automated speech recognition (ASR) for any subsequent speech spoken by the user. Advantageously, the hot word detection model can be executed on low-power hardware, such as a digital signal processor (DSP) chip, and may respond to a variety of fixed phrase commands, such as "Hey Google" or "Hey living room speaker."

[0022] As the number of Assistant-enabled devices in a user's environment (e.g., home or office) increases, the user may want to trigger multiple Assistant-enabled devices simultaneously, for example, to adjust volume levels across a group of Assistant-enabled smart speakers or adjust illumination levels across a group of Assistant-enabled smart lights. If a user wants to trigger multiple different Assistant-enabled devices, the user is currently required to issue separate queries to each device independently. For example, to turn off the kitchen light and dining room light in a user's home, the user needs to speak separate queries such as "Hey kitchen lightbulb, turn off" and "Hey dining room lightbulb, turn off."

[0023] Implementations herein are directed to allowing a user to issue a single query to a group of assistant-enabled devices (AEDs) to enable faster, more natural interactions with multiple different AEDs that the user may want to control simultaneously. Specifically, implementations are directed to creating and assigning a group hotword to two or more AEDs in a group selected by a user, such that each device responds to a verbal query that includes the group hotword by triggering from a low power state when the group hotword is detected in streaming audio. That is, each AED in the selected group of AEDs to which the hotword is assigned may execute a hotword detection model trained to detect the presence of the group hotword in streaming audio without performing voice recognition. In some implementations, the group hotwords assigned to the AEDs in the selected group are predefined such that the corresponding hotword detection model is pre-trained to detect the presence of the predefined group hotword. On the other hand, a user may create a custom group hotword that includes any word or phrase that the user wants to use to target a specific group of AEDs in a single query. Here, the user may be requested to provide one or more training utterances of the user speaking the custom hotword to train a corresponding hotword detection model to detect the custom hotword.

[0024] In some examples, a user uses the digital assistant interface to select a group of AEDs and manually enable a group hotword (e.g., predefined or custom) to be assigned to the AEDs in the selected group that the user wants to target simultaneously in a single query. The AEDs in the selected group of AEDs may be configured to receive an assignment command from the digital assistant interface that assigns the group hotword to the AEDs in the group, such that each AED in the selected group wakes up from a low power state when the group hotword is detected in streaming audio by at least one of the AEDs in the selected group. For example, a user may assign a group hotword "family room speakers" to a group of four smart speakers located in the living room of the user's home so that the user can target all four of these smart speakers by speaking an utterance that includes the group hotword "Family room speakers" followed by a single query that specifies an action to be performed, e.g., "play that 12-6-97 Phish show." In this example, at least one of the smart speakers in the group of four smart speakers detecting the group hotword "Family room speakers" in the user's speech triggers the corresponding smart speaker to wake up from a low power state and execute a collaboration routine that causes each smart speaker in the group of four smart speakers to collaborate with each other to accomplish the execution of the action specified by the query. For example, the four living room speakers may collaborate to play a song corresponding to a concert performed by the band Phish on the date December 6, 1997.In this embodiment, one of the speakers may be tasked with streaming the song from a local storage device, a network storage device, or a remote streaming service, and then broadcasting the song to the other speakers to play the song audibly from those speakers. Optionally, when working together to perform an action, some of the smart speakers may perform different playback responsibilities related to the action, such as two of the smart speakers may play audio corresponding to the left channel and two other of the smart speakers may play audio corresponding to the right channel, resulting in a stereo configuration.

[0025] Continuing with this same example, other AEDs in the selected group that are not assigned to the group hotword, such as AEDs corresponding to device types other than smart speakers and smart speakers located in rooms other than the living room of the user's home, do not respond to the group hotword and remain asleep when the user speaks "Family room speakers". Furthermore, each AED may be assigned a unique device-specific hotword that only the corresponding AED is configured to detect in the streaming audio if the user desires to address only the corresponding AED. For example, the unique device-specific hotword assigned to the AED may include an identifier of the AED, such as "Hey Device 1", or may also include a device type, such as "Hey Smart Speaker 1", and / or other attributes associated with the AED. Furthermore, the four smart speakers in the selected group located in the living room, as well as any other AEDs associated with the user but not assigned to a group hotword, may also be configured to respond to a generic default hotword, such as "Hey Assistant". In some embodiments, two or more group hot words may be simultaneously assigned to at least one AED associated with a user such that the at least one AED is a member of different selected groups of AEDs each assigned a corresponding one of the two or more group hot words, In these embodiments, the AEDs of each selected group may include a combination of AEDs assigned corresponding group hot words that is different from the combination of AEDs assigned different corresponding hot words.

[0026] In some implementations, the user manually enables the group hotword to be assigned to the AEDs of the selected group. For example, the user may access a digital assistant application that displays a graphical user interface to allow the user to configure and adjust the settings of all AEDs associated with the user. Here, the graphical user interface may present a group hotword screen that renders various graphical objects (text fields, buttons, pull-down menus) for creating and enabling group hotwords and for selecting which AEDs the user wants to assign the group hotword to. Thus, the AEDs of the selected group may receive an assignment instruction to assign the group hotword in response to receiving a user input instruction indicating a user interaction with one or more objects displayed in the graphical user interface to enable the group hotword and instruct the digital assistant to assign the group hotword to each AED in the selected group of AEDs. The user may update the AEDs of the selected group via the GUI by selecting one or more additional AEDs to add to the group and / or selecting one or more AEDs to remove from the group.

[0027] The user may also select the AEDs of the group through a voice input corresponding to the group hotword query and enable the group hotword to be assigned to the AEDs of the selected group. Here, the user may speak a voice input that requests the digital assistant to enable the group hotword and assign the group hotword to the AEDs of the selected group. For example, the voice input spoken by the user to enable the group hotword to be assigned to the first and second AEDs located in the downstairs area of ​​the user's home may include "Device 1 and device 2, respond to downstairs devices." Here, the term "Device 1" spoken by the user includes a respective device-specific hotword assigned to the first AED and the term "Device 2" spoken by the user includes a different respective device-specific hotword assigned to the second AED, such that each of the first AED and the second AED detects its respective device-specific hotword and initiates to process subsequent audio data corresponding to the group hotword query "respond to downstairs devices". Thus, at least one of the first AED or the second AED may generate an ASR result for the voice input and then instruct a speech recognizer (e.g., an on-device ASR or a server-side ASR) to perform speech recognition on the audio data to perform query interpretation on the ASR result to identify the group hotword query. The group hotword query identified by the query interpretation performed on the ASR result specifies the name of the group hotword to be activated (e.g., "downstairs devices") and each AED in the selected set of AEDs to which the group hotword should be assigned. Users could also provide voice input corresponding to a group hotword query by directly invoking the digital assistant through the generic hotword.For example, a user could speak the group hotword query, "Hey Assistant, have device 1 and device 2 respond to downstairs devices." In this example, any AED associated with the user may detect the predefined default hotword, "Hey Assistant," and may be triggered to initiate speech recognition on the audio data to generate an ASR result and perform speech recognition to identify the group hotword and each AED in the selected group of AEDs to which the group hotword should be assigned.

[0028] As with the GUI embodiment described above, the user may similarly update the AEDs of the selected group via subsequent voice inputs specifying one or more additional AEDs to be added to the group and / or selecting one or more AEDs to be removed from the group. For example, the user may say "Hey downstairs devices, add device 3" to add a third AED 104c (device 3) to the group of AEDs that has been assigned the group hotword "downstairs devices". Similarly, the user may say "Hey device 1, leave the downstairs devices group" to remove the first AED 104a (device 1) from the group such that the first AED is no longer assigned the group hotword and is not triggered when the user says "Hey downstairs devices". The user may provide a verbal confirmation to confirm (or undo) the updates made to the AEDs of the group. Additionally, once all devices have left a selected group, the hotword may cease to exist, requiring the user to recreate or re-enable the group hotword.

[0029] In a further embodiment, the group hotword is available implicitly. For example, the user may utter the group hotword "Hey nearby devices" or "Hey nearby device" to target only AEDs in close proximity to the user. The hotword detector may detect the singular group hotword, the plural group hotword, or detect only the singular and rely on voice recognition to recognize the "s" suffix. This type of implicit group hotword includes a proximity-based group hotword. The user may access the digital assistant application and interact with the GUI to specify which AEDs should be assigned the implicit proximity-based group hotword. Thus, the group hotword in this example provides a context that specifies that the user wishes to call only the AED or AEDs currently closest in proximity to the user, without requiring the user to explicitly identify those AEDs, either by their respective unique hotwords assigned to the AEDs or by naming the AEDs in the query portion of the utterance. It should be noted that each AED that is assigned an implicit proximity-based group hotword may execute a hotword detection model to detect the presence of the group hotword in the streaming audio to trigger a wake-up process and begin voice recognition on the audio. Because the implicit group hotword in this example is proximity-based, multiple AEDs may detect the group hotword in the captured streaming audio, but each of these AEDs may then process the audio to determine their respective proximity values ​​to the user, and then use these proximity values ​​to perform arbitration across the multiple AEDs to select one or more of these AEDs to perform the action specified by the user's query. Here, AEDs that are outside some upper distance threshold from the user may be ineligible to perform the query.Optionally, AEDs within some lower distance threshold may also be ineligible to respond to the query, such as a smartphone-AED in the user's pocket that detects the proximity-based group hotword "Hey nearby device(s)." The user also has the option to add / remove AEDs to / from selected groups that have been assigned a proximity-based group hotword.

[0030] Additionally or alternatively, the one or more AEDs selected to respond to the user's query may be based on the type of query and / or the respective device characteristics associated with each AED, such that only the one or more AEDs that are best suited to fulfill the query are selected. In this case, the device characteristics associated with each AED may include processing capabilities, device type, user-configurable device settings, power usage, battery level, the physical location of the AED, or network capabilities, among others. Thus, if the query is a single-device query such as "Hey nearby device, set a timer," device arbitration may determine that the closest AED to the user is ineligible to fulfill the query because the AED is a battery-powered smart speaker and the battery capacity is very low (e.g., less than 5%). Thus, the next closest AED with an implicit proximity-based group hotword assigned may fulfill the query.

[0031] In some implementations, an implicit group hotword is assigned to AEDs in a selected group associated with the same device type. For example, an implicit device type group hotword may include "Hey smart speakers" to target all AEDs associated with a user, including a smart speaker device type. Similarly, another implicit device type group hotword may include "Hey smart lights" to target all AEDs, including a smart lighting device type. It should be noted that the device type group hotword uniquely identifies the device type associated with the AEDs in the selected group, thereby providing a context indicating which AED the user wishes to target. Each AED may execute a pre-trained hotword detection model to detect the presence of the device type group hotword in streaming audio without performing voice recognition on the audio data.

[0032] In further implementations, implicit group hotwords are assigned to AEDs in a selected group that share a common attribute. For example, implicit attribute-based group hotwords can include "Hey blue devices" to target all AEDs associated with a user that are labeled as having a blue color, or "Hey red devices" to target all AEDs associated with a user that are labeled as having a red color. Attribute-based group hotwords can similarly specify any other attribute, such as size, e.g., "Hey large devices" or "Hey small devices." Of note, attribute-based group hotwords can further refine the AEDs in a particular group that a user desires to target. In a non-limiting example where an implicit device-type group hotword "Hey smart speakers" targets all smart speakers throughout a user's home and a manually enabled group hotword "Hey family room speakers" targets only the four smart speakers located in the living room of the user's home, an implicit attribute-based group hotword "Hey blue devices" can be used to target only two of the four smart speakers located in the user's living room that are labeled as having a red appearance.

[0033] An implicit group hotword may be enabled / disabled via the GUI of the digital assistant application. Similarly, the AEDs in the group that have been assigned the implicit group hotword may be specified / selected via the GUI of the digital assistant application. The AEDs in the selected group that have been assigned the implicit group hotword may be updated by adding additional AEDs to the group and / or removing AEDs from the group, as described above.

[0034] In a further implementation, the digital assistant automatically creates and assigns a group hotword to the selected group of AEDs performing the long-lived action while the long-lived action is in progress. For example, a user may speak a voice query / command that instructs the digital assistant to perform the long-lived action on two or more AEDs. In a non-limiting example, a voice query / command of "Hey Assistant, play party music playlist on speaker 1 and speaker 2" causes the digital assistant to perform the long-lived action by streaming the user's party music playlist as an audible playback from speaker 1 and speaker 2. In this example, the digital assistant is configured to automatically create an action-specific group hotword of "Party music" for the user to use in follow-up queries related to the long-lived action. Thus, speaker 1 and speaker 2 each receive an assignment instruction that assigns the group hotword of "Party music" automatically created by the digital assistant. The user may then target a long-lived action to be performed on Speaker 1 and Speaker 2 by simply saying "Party music." For example, the user may speak the utterance "Party music, next song" to advance to the next track in a playlist, or may say "Party music, turn up the volume" to command Speaker 1 and Speaker 2 to each increase the speaker volume.To inform the user of the action-specific group hotword created by the digital assistant, the digital assistant may output a synthesized voice corresponding to a response for audible playback from one of the AEDs (e.g., Speaker 1 or Speaker 2) to indicate that execution of the long-lived action is in progress and the automatically created group hotword for use in follow-up queries related to the long-lived action. In the above example, the response may include a synthesized voice saying, "Got it, now playing that. In the future, you can control playback using the 'party music' hotword." The digital assistant may cancel the use of the automatically created group hotword when the long-lived action ends.

[0035] 1A-1C illustrate a system 100 for assigning a group hotword 50g to two or more assistant-enabled devices (AEDs) 104 of a selected group associated with a user 102 to enable the user 102 to target two or more AEDs of the selected group in a single query by speaking the group hotword 50g. Briefly, as described in more detail below, FIG. 1A shows the user 102 manually enabling the group hotword to be assigned to two or more AEDs 104, 104a-c of a selected group associated with the user 102 by speaking 106 the following utterance: "Hey Assistant, have device 1 and device 2 respond to downstairs speakers." In response to the utterance 106, a digital assistant 105 executing on the AED 104 (and, optionally, a remote server 120 in communication with the AEDs) provides an assignment instruction that assigns the group hotword "downstairs speakers" to a selected group of AEDs 104 including a first AED 104a named "device 1" and a second AED 104b named "device 2." Each AED 104a, AED 104b to which the group hotword is assigned is configured to wake up from a low power state when the group hotword is detected in streaming audio by at least one of the AEDs in the selected group of AEDs 104a, AED 104b.For example, when the user speaks the subsequent utterance 126, "Downstairs speakers, play my playlist," the first AED 104a and the second AED 104b detect the group hotword "Downstairs speakers" in the audio data corresponding to the utterance 126 which triggers each AED 104a, AED 104b to execute a collaboration routine 150 in which they collaborate with each other to wake up from a low power state and begin playing the song 122 from the user's 102 playlist (e.g., track #1).

[0036] In the illustrated embodiment, the system 100 includes three AEDs 104a-c associated with a user 102 and running a digital assistant 105 with which the user 102 may interact through voice. Although three AEDs 104 are shown, the user 102 may include any number of AEDs 104 located throughout a voice-enabled environment associated with the user 102. The AEDs 104 are all smart speakers, but the AEDs 104 may include other computing devices, such as, without limitation, smartphones, tablets, smart displays, desktops / laptops, smart watches, smart appliances, headphones, or vehicle infotainment devices, without departing from the scope of this disclosure. Each AED 104 includes data processing hardware 10 and memory hardware 12 that stores instructions that, when executed on the data processing hardware 10, cause the data processing hardware 10 to perform operations. Each AED 104 includes an array of one or more microphones 16 configured to capture acoustic sounds, such as voice, directed to the AED 104. Each AED 104 may also include or be in communication with an audio output device (e.g., a speaker) 18 that may output audio, such as songs 122 and / or synthesized voice 450 (FIG. 4) from the digital assistant 105.

[0037] FIG. 1A shows that the user 102 speaks an utterance 106, "Hey Assistant, have device 1 and device 2 respond to downstairs speakers," in the vicinity of at least the first AED 104a to request the digital assistant 105 to activate the group hotword "downstairs speakers" and assign the group hotword to a selected group of AEDs including a first AED 104a named "device 1" and a second AED 104b named "device 2." The microphone 16 of the first AED 104a receives the utterance 106 and processes audio data 20 corresponding to the utterance 106. The initial processing of the audio data 20 may include filtering the audio data 20 and converting the audio data 20 from an analog signal to a digital signal. After the first AED 104a processes the audio data 20, the first AED 104a may store the audio data 20 in a buffer in the memory hardware 12 for further processing. With the audio data 20 in the buffer, the first AED 104a may use a hotword detector 108 to detect whether the audio data 20 includes a predefined generic hotword 50 of "Hey Assistant" assigned to each AED associated with the user 102. The hotword detector 108 is configured to identify hotwords included in the audio data 20 without performing voice recognition on the audio data 20. The hotword detector 108 may include an initial hotword detection stage that loosely listens for the presence of the hotword 50 and, if detected, triggers a second hotword detection stage that confirms the presence of the hotword 50. The initial hotword detection stage may run on a low-power digital signal processor (DSP) of the data processing hardware 10, while the second hotword detection stage may run on a more computationally expensive application processor (AP) (e.g., a system-on-chip (SoC)) to enable more accurate hotword detection.

[0038] In some implementations, the hot word detector 108 is configured to identify hot words in an early portion of the utterance 106. In this example, the hot word detector 108 may determine that the utterance 106, "Hey Assistant, have device 1 and device 2 respond to downstairs speakers," contains the predefined generic hot word 50, "Hey Assistant," if the hot word detector 108 detects acoustic features in the audio data 20 that are characteristic of the hot word 50. The acoustic features may be Mel-frequency cepstral coefficients (MFCCs), which are a representation of the short-term power spectra of the utterance 106, or may be Mel-scale filter bank energies of the utterance 106. For example, the hot word detector 108 may detect that the utterance 106, "Hey Assistant, have device 1 and device 2 respond to downstairs speakers," includes the hot word 50, "Hey Assistant," based on generating MFCCs from the audio data 20 and classifying the MFCCs as including MFCCs similar to MFCCs characteristic of the hot word "Hey Assistant" stored in a hot word model of the hot word detector 108. As another example, the hot word detector 108 may detect that the utterance 106, "Hey Assistant, have device 1 and device 2 respond to downstairs speakers," includes the hot word 50, "Hey Assistant," based on generating mel-scale filter bank energies from the audio data 402 and classifying the mel-scale filter bank energies as including mel-scale filter bank energies similar to mel-scale filter bank energies characteristic of the hot word "Hey Assistant" stored in a hot word model of the hot word detector 108.

[0039] If the hotword detector 108 determines that the audio data 20 corresponding to the utterance 106 includes the predefined generic hotword 50, the AED 104 may trigger a wake-up process that initiates speech recognition on the audio data 20 corresponding to the utterance 106. For example, a speech recognizer 116 running on the AED 104 may perform speech recognition and / or semantic interpretation on the audio data 20 corresponding to the utterance 106. The speech recognizer 116 may perform speech recognition on the audio data 20 to generate automated speech recognition (ASR) results for the utterance 106, and then perform query interpretation on the ASR results to identify names of group hotwords to be activated and group hotword queries 118 that specify each AED 104 in the selected group of AEDs to which the group hotwords should be assigned. In this example, the speech recognizer 116 may perform query interpretation on the ASR results that include the phrase "have devices 1 and 2 respond to downstairs devices" as the group hot word query 118 that specifies the name of the group hot word "downstairs speakers" and "device 1 and device 2," which are each AED 104a, 104b in the selected group of AEDs 104 to which the group hot word should be assigned.

[0040] In some implementations, the speech recognizer 116 is located on the server 120 in addition to or instead of being located on the AED 104. When the hot word detector 108 triggers the AED 104a to wake up in response to detecting a predefined generic hot word 50 in the utterance 106, the AED 104a may transmit the audio data 20 corresponding to the utterance 106 to the server 120 via the network 132. The AED 104a may transmit the portion of the audio data 20 including the hot word 50 so that the server 120 confirms the presence of the generic hot word 50. Alternatively, the AED 104a may transmit only the portion of the audio data 20 corresponding to the portion of the utterance 106 after the generic hot word 50 to the server 120. The server 120 executes the speech recognizer 116 to perform speech recognition and returns a transcription of the audio data 20 to the AED 104a. The AED 104a then identifies the words in the utterance 106, and the AED 104a performs semantic interpretation and identifies the group hotword query 118. The AED 104a (and / or the server 120) may identify the group hotword query 118 so that the digital assistant 105 activates and provides an assignment instruction to assign the group hotword "downstairs speakers" to the selected group of AEDs 104, including the first AED 104a and the second AED 104b. In the illustrated embodiment, the digital assistant 105 begins to execute a long-lived action of playing the song 122 as playback audio from the speaker 18 of the AED 104. The digital assistant 105 may stream the song 122 from a streaming service (not shown), or the digital assistant 105 may instruct the AED 104 to play a song stored on the AED 104.

[0041] After the group hot word 50g "downstairs devices" is enabled and assigned to the first AED 104a and the second AED 104b, the respective hot word detectors 108 running on the first AED 104a and the second AED 104b are configured to identify the group hot word 50g "downstairs devices" in the audio data 20 corresponding to the subsequent utterance 126. Here, the respective hot word detectors 108 may activate the respective group hot word models 114 running on the AEDs 104a and 104b, which are trained to detect the subsequent utterance 126 of the group hot word 50g "downstairs speakers" in the streaming audio captured by the AEDs 104a and 104b, respectively, without performing speech recognition on the captured audio. The group hot word models 114 may be stored on the memory hardware 12 of the AEDs 104 or the server 120. When stored on the server 120, the AED 104 may request the server to obtain the group hotword model 114 for the corresponding group hotword 50g and provide the obtained group hotword model 114 so that the AED 104 can activate the group hotword model 114. In some embodiments, the group hotword 50g is predefined and available as a suggested group hotword that the user 102 may enable and assign to the AEDs 104 of a selected group. In these embodiments, the corresponding group hotword model 114 is pre-trained to detect the group hotword 50g in streaming audio. In other embodiments, the group hotword 50g is a custom group hotword created by the user. In these other embodiments, the user 102 may train the corresponding group hotword model 114 to detect the custom group hotword 50g by speaking a training utterance that includes the user 102 speaking the custom group hotword 50g.

[0042] In a further implementation, assigning the group hot word to the selected group of AEDs 104 causes one or more of the AEDs 104 to run the speech recognizer 116 in a low power and low fidelity state, where the speech recognizer 116 is constrained or biased to recognize the group hot word 50g assigned to the AED 104 only when spoken in a subsequent utterance 126 captured by the AED 104. Because the speech recognizer 116 only recognizes a limited number of terms / phrases, the number of parameters of the speech recognizer 116 may be significantly reduced, thereby reducing the memory requirements and the number of calculations required to recognize the group hot word 50g in speech. Thus, the low power and low fidelity characteristics of the speech recognizer 116 may be suitable for execution on a digital signal processor (DSP). In these implementations, a speech recognizer 116 running on at least one of the AEDs 104 may recognize utterances 106 of enabled group hot words 50g in streaming audio captured by the at least one AED 104 instead of using the group hot word model 114.

[0043] One or more of the AEDs 104 may store a hotword registry 500 locally on the memory hardware 12. The hotword registry 500 includes a list of one or more hotwords 50 each assigned to one or more AEDs 104 associated with the user 102. The digital assistant 105 and / or the AEDs 104 in the selected group may include an enabled group hotword 50g in the list of one or more hotwords and add to the hotword registry 500 to identify each AED 104 in the selected group of AEDs 104 to which the group hotword 50g is assigned. Upon enabling and assigning the group hotword 50g "downstairs speakers" to a first AED 104a named device 1 and a second AED 104b named device 2, FIG. 1A shows the digital assistant 105 updating the hotword registry 500 to specify the assignment of the group hotword 50g "downstairs speakers" to device 1 and device 2.

[0044] In some implementations, after the first AED 104a and the second AED 104b in the selected group of AEDs receive the assignment instruction to assign the group hotword 50g, the first AED 104a and the second AED 104b execute a leader selection process 300 to select one or more AEDs from the selected group to listen for the presence of the group hotword 50g in the streaming audio on behalf of the AEDs in the selected group based on the respective device characteristics 302 associated with each AED 104. FIG. 3 shows an exemplary leader selection process 300 configured to receive as input the respective device characteristics 302 associated with each AED 104 in the selected group of AEDs and generate as output a selection instruction 310 to select one or more of the AEDs to listen for the presence of the group hotword 50g. The device characteristics 302 associated with each AED 104 may include at least one of processing capabilities, device type, user configurable device settings, power usage, battery level, physical location of the AED, or network capabilities, without limitation. In the example of FIG. 1A, the device characteristics 302 associated with the second AED 104b, named device 2, may indicate that the second AED 102b is a portable device and is currently powered by a battery, while the device characteristics 302 associated with the first AED 104a, named device 1, may indicate that the first AED 102a is a stationary device powered by an external power source, e.g., a power outlet. Thus, the selection instructions 310 output by the leader selection process 300 may indicate that if the battery level of device 2 is not less than five percent (5%), the closer of device 1 or device 2 should respond to the user 102 speaking "downstairs devices" by performing voice recognition and semantic interpretation to identify the query.That is, when a subsequent utterance 126 including the group hotword 50g "downstairs devices" is detected in the streaming audio by each of the AEDs 104a, 104b, execution of the collaboration routine 150 by the AEDs 104a, 104b causes the second AED 104b, named device 2, to not respond if its battery level is less than 5 percent, even if device 2 is closest to the user 102. This allows device 2 to conserve power by not having to expend processing resources performing speech recognition and / or semantic interpretation on the audio data. Alternatively, if power conservation is not a concern, the leader selection process 300 may generally select the closest AED 104 to process the audio, since the audio captured by that AED 104 is more likely to have a higher quality, and therefore more accurate, speech recognition, than audio captured by AEDs 104 that are more distant.

[0045] In further embodiments, the leader selection process 300 can generate more granular selection instructions 310. For example, the selection instructions 310 may select only one of the AEDs 104 to trigger a second stage of hot word detection (i.e., using the computationally expensive hot word detection model 114 or using the speech recognizer 116) to confirm the presence of the group hot word 50g when the first stage hot word detector 108 initially detects the group hot word 50g. That is, the selection instructions 310 may inform each AED 104 in the selected group of AEDs to which the group hot word 50g is assigned that when each AED 104 detects the presence of the group hot word 50g using the first stage hot word detector 108, only the selected one of the AEDs 104 will trigger a second stage of hot word detection to confirm the presence of the group hot word 50g. Expanding further, the device characteristics 302 may indicate that one of the AEDs is configured to run the first stage hotword detector 108 on a battery-powered DSP chip that consumes low power at the expense of low fidelity coarsely listening for the group hotword 50g, and when the group hotword 50g is detected by the first stage hotword detector 108, an application processor (e.g., SoC chip) is triggered to wake up and run the second stage hotword detection (e.g., hotword model 114 or on-device ASR 116) to confirm the presence of the group hotword 50g. Thus, if the device characteristics 302 indicate that one or more other AEDs in the selected group of AEDs are non-battery-powered devices, it may be efficient to utilize those devices for at least the task of second stage hotword detection so that the battery-powered devices do not waste power by triggering the AP to wake up from a low power state.

[0046] Other scenarios may exist where the device characteristics 302 for the AEDs in the selected group of AEDs indicate that the AEDs can perform on-device voice recognition for a limited set of common queries / commands, while other AEDs in the selected group need to provide audio to the server 120 to perform server-side ASR. The leader selection process 300 may generate selection instructions 310 that cause the collaboration routine 150 to select an AED that can perform on-device voice recognition to first attempt to perform on-device voice recognition on the captured audio data 20 to determine whether one of the common queries / commands in the limited set is recognized in the captured audio data 20. If one of the common queries / commands is not recognized, the generated selection instructions 310 may allow the collaboration routine 150 to select one of the other AEDs to provide audio data 20 to the server 120 to perform server-side ASR on the audio data 20.

[0047] Continuing with reference to FIG. 3 , the AEDs 104 in the selected group of AEDs 104 may re-run the leader selection process 300 periodically and / or in response to certain events. In one embodiment, re-running the leader selection process 300 is in response to a device state change 304 in one of the AEDs in the selected group of AEDs 104. The device state change 304 may include, without limitation, an increase in the processing load on the AED 104 to a level that violates a processing threshold, a decrease in the processing load on the AED 104 to a level that no longer violates the processing level, a change in background noise levels, a drop in battery capacity below a battery capacity threshold, a loss of network connectivity, the AED 104 powering off, etc. The device state change 304 causes the leader selection process 300 to re-evaluate the respective device characteristics 302 associated with each AED 104 in the selected group to select the AED or AEDs currently best suited to listen for the group hotword.

[0048] In one embodiment, re-executing the leader selection process 300 is performed in response to an update 306 of the selected group of AEDs 104 that adds one or more additional AEDs 104 to the selected group of AEDs 104. For example, FIG. 1B shows that the user 102 speaks another utterance 136, "Downstairs speakers, add device 3," near at least the first AED 104a to request the digital assistant 105 to assign the group hotword 50g "downstairs speakers" to the third AED 104c named "device 3" in addition to the first AED 104a named device 1 and the second AED 104b named device 2. Here, the utterance 136 includes the group hotword 50g "downstairs speakers" that at least the first AED 104a detects in the audio data 20 corresponding to the utterance 136 using the hotword detection model 114 corresponding to the group hotword 50g to trigger the first AED 104a (i.e., based on the selection instruction 310) to wake up from a low power state. Upon wake-up, the first AED 104a instructs the speech recognizer 116 to perform speech recognition on the audio data 20 to perform query interpretation on the ASR results to generate an ASR result for the utterance 136 and identify a group hotword query 118 that specifies adding a device identifier "Device 3" for the further AED 102c to the selected group of AEDs 104 to which the group hotword 50g "downstairs devices" is assigned. Thus, the third AED 104c may receive an assignment instruction to assign the group hot word 50g "downstairs speakers" to the AEDs in the selected group, which is now updated to include the third AED 104c. The third AED 104c may activate the hot word detection model 114 that corresponds to the group hot word 50c, as described above with reference to FIG. 1A.The digital assistant 105 may update the hotword registry 500 to add a third AED 104c, named device 3, to the selected group of AEDs 104 that are assigned the group hotword 50g of "downstairs speakers." In response to the update 306 that adds the third AED 104c to the selected group of AEDs 104, the leader selection process 300 of FIG. 3 may be re-executed to take into account the respective device characteristics 302 associated with the third AED 104c. All three AEDs 104a-c may cooperate with each other to accomplish the execution of the long-lived action of streaming songs 122 from the user's playlist.

[0049] Additionally or alternatively, re-executing the leader selection process 300 may occur in response to an update 306 of the selected group of AEDs 104 that removes one or more AEDs 104 from the selected group of AEDs 104. For example, FIG. 1C shows the user 102 speaking another utterance 146, “Device 1, leave the downstairs speakers group,” at least near the first AED 104a to request the digital assistant 105 to remove the first AED 104a, named Device 1, from the selected group of AEDs 104 that are assigned the group hotword 50g, “downstairs speakers.” Here, the utterance 146 includes a device-specific hotword 50d, "Device 1," uniquely assigned to the first AED 104a and detected by the first AED 10a in the audio data 20 corresponding to the utterance 146, which triggers the first AED 104a to wake up from a low power state and process the audio data 20 to identify a group hotword query 118 that requests the digital assistant 105 to remove the first AED 104a from the selected group of AEDs 104 to which the group hotword 50g, "downstairs speakers," is assigned. Thus, the first AED 104a may deactivate the hotword detection model 114 corresponding to the group hotword 50g so that the first AED 104a no longer listens for the presence of the group hotword 50g in the audio data. The digital assistant 105 may update the hotword registry 500 to remove the first AED 104a, named device 1, from the selected group of AEDs 104 that were assigned the group hotword 50g "downstairs speakers." The leader selection process 300 of FIG. 3 may be re-run to determine updated selection instructions 310 based on the fact that device 1 is no longer a member of the selected group of AEDs.The second AED 104b and the third AED 104c may now cooperate with each other without the first AED 104a to accomplish the execution of the long-lived action specified by the query 128 in the utterance 126 spoken by the user 102 in FIG. 1A.

[0050] 1A, at least the first AED 104a uses the corresponding group hotword model 114 to detect the presence of the group hotword 50g "downstairs devices" in the audio data 20 corresponding to a subsequent utterance 126 spoken by the user 102, the subsequent utterance 126 including a query 128 specifying an action to be performed. Specifically, the example shows the user 102 speaks the subsequent utterance 126 "Downstairs speakers, play my playlist," and at least the first AED 104a uses the group hotword model 114 to detect the group hotword 50g "downstairs devices" in the corresponding audio data 20. Detecting the group hotword 50g in the audio data 20 triggers the first AED 104a (and, optionally, the second AED 104b) to wake up from a low power state and execute a collaboration routine 150 that causes the first AED 104a and each of the other AEDs 104 to which the group hotword 50g is assigned to collaborate with each other to accomplish the execution of the action specified by the query 128. Here, the query 128 specifies a long-lived action, and the first AED 104a and the second AED 104b collaborate with each other by pairing with each other for the duration of the long-lived action and coordinating the execution of a sub-action related to the long-lived action to play songs 122 from a user's playlist. For example, one AED 104 may execute a sub-action to connect to a remote song streaming service that streams the playlist and broadcast the streaming playlist to the other AED 104. In some embodiments, cooperating AEDs 104 may have different music playback responsibilities, such as one AED being responsible for the left audio channel and another AED being responsible for the right audio channel, resulting in a stereo configuration.FIG. 1A shows a first AED 104a, named device 1, and a second AED 104b, named device 2, executing a collaboration routine 150 in which they collaborate with each other to accomplish the execution of a long-lived action of playing a song 122 (e.g., track #1) from a user's playlist.

[0051] In some embodiments, in response to the first AED 104a detecting the group hot word 50g in the audio data 20, the first AED 104a calls each other AED 104 in the selected group of AEDs 104 that did not detect the group hot word 50g to wake up from a low power state and cooperate with the first AED 104a to perform the operation specified by the query 128. In these embodiments, in response to detecting the group hot word 50g, the first AED 104a may identify each of the one or more other AEDs 104 in the selected group to which the group hot word is assigned by accessing a hot word registry 500. Here, the hot word registry 500, which includes a list of one or more hot words, includes the group hot word 50g, “downstairs stairs,” assigned to the first AED 104a, named Device 1, and the second AED 104b, named Device 2. Thus, the first AED 104a may identify that the second AED 104b, named device 2, is also assigned the group hotword 50g, and as a result, may call the second AED 104b to collaborate with the first AED 104a to accomplish the execution of the action specified by the query 128 (e.g., streaming the song 122 from the user's 102 playlist).

[0052] Although the query 128 in the illustrated embodiment specifies a long-lived action to be performed, other embodiments may include a query that specifies a device-level action to be performed individually on each AED in the selected group of AEDs. That is, during execution of the collaboration routine 150, each AED in the selected group of AEDs collaborates by independently performing the execution of the device-level action. For example, if the first AED 104a and the second AED 104b correspond to smart light bulbs that are assigned the same group hotword 50g, a query that specifies a device-level action to turn off the light will cause each smart light bulb to independently perform the action of powering off.

[0053] 2A, in some implementations, a software application 205 associated with the digital assistant 105 executes on a user device to display a user-defined group hotword selection screen 200, 200a on a graphical user interface (GUI) 208 of the user device. In the illustrated embodiment, the user device includes an AED 104 corresponding to a smartphone (e.g., smartphone 104j of FIG. 4). The user-defined group hotword selection screen 200a allows the user to activate and assign a group hotword 50g to two or more AEDs 104 of the group selected by the user. The user 102 may activate and assign group hotwords using the group hotword selection screen 200a in addition to or instead of providing a voice input as described above with reference to FIGS. 1A-1C. In the illustrated embodiment, the group hotword selection screen 200a displays multiple objects 210, 210a-d in the GUI 208 with which the user may interact to activate the group hotword 50g and instruct the digital assistant 105 to select the AED 104 of the group to which the group hotword 50g should be assigned.

[0054] The GUI 208 may receive user input indications indicating a user interaction with a text field object 210a that allows the user to create a custom group hotword by typing the name of the custom group hotword the user desires to create. Optionally, the user 102 may select a voice input graphic (e.g., a graphical microphone) to provide voice input corresponding to the user 102 speaking the custom group hotword. When creating a custom group hotword, the group hotword selection screen 200a may prompt the user to speak several training examples that include the custom group hotword for use in training the group hotword detection model 114 to detect the custom group hotword in streaming audio.

[0055] On the other hand, the user 102 may activate the predefined group hotwords by providing a user input indication indicating user interaction with a drop-down object 210b that presents a list of available predefined group hotwords to select from. The drop-down object 210 may present commonly used group hotwords as available predefined group hotwords to select from, such as group hotwords describing device types, e.g., "Smart Speakers" and "Smart Lights," group hotwords describing general areas / areas in an environment, e.g., "Family Room Devices," and group hotwords describing both device types and areas / areas, e.g., "Bedroom Speakers." The user may interact with the drop-down object 210b to scroll through the list of available group hotwords. In some implementations, custom group hotwords can be added to the list of available group hotwords. In the illustrated embodiment, the GUI 208 receives a user input indication indicating a user interaction with a drop-down object 210 selecting a predefined group hotword "Family Room Devices" from a list of available predefined group hotwords. Here, the selection of the predefined group hotword may instruct the digital assistant 105 to activate the predefined group hotword. The assistant may also suggest group hotwords to be enabled / activated to be assigned to the group's AEDs 104. For example, the user may take care to manually query the devices of the group (e.g., sequentially or via the individual names of the devices) that all belong to the group in some sense.

[0056] Additionally, the group hotword selection screen 200a displays a number of selection objects 210c, each corresponding to a respective one of a number of AEDs 104 associated with the user 102. The user 102 may provide user input (e.g., touch) to select each AED 104 for inclusion in the group of AEDs 104 to which the group hotword 50g created via the text field object 210a or selected from the drop-down object 210b is to be assigned. In the illustrated example, the GUI 208 receives a user input indication indicating user interaction with the selection objects 210c, which correspond to the AEDs 104 named Speaker 1, Speaker 2, Speaker 3, Speaker 4, and Smart TV, to include these AEDs in the group of AEDs selected to be assigned the group hotword "Family Room Devices." To instruct the digital assistant 105 to activate and assign the group hotword "Family Room Devices" to the selected group of AEDs 104, including speakers 1-4 and the smart TV, the user 102 may provide a user input instruction indicating a user interaction with the activation object 210d. Assuming that the activation object 210d is selected, the digital assistant 105 provides an assignment instruction to the selected group of AEDs, including speakers 1-4 and the smart TV, indicating the assignment of the group hotword "Family Room Speakers" to the selected group of AEDs. The digital assistant may also add the group hotword and the selected group of AEDs to the hotword registry 500 shown in FIG. 5.

[0057] 2B, in some implementations, the software application 205 associated with the digital assistant 105 is configured to display an implicit group hotword selection screen 200, 200b in the GUI 208 of the AED 104. The implicit group hotword selection screen 200b displays a plurality of available implicit group hotwords and allows the user 102 to select an AED of a group to which each implicit group hotword should be assigned. For each implicit group hotword, the implicit group hotword selection screen 200b may list all eligible AEDs to which the implicit group hotword can be assigned based on attributes associated with the AED. For example, all of the AEDs 104 associated with the user 102 are listed as eligible AEDs to which the proximity-based group hotwords "Hey nearby devices" and / or "Hey nearby device" should be assigned. Thus, a user 102 may target one or more AEDs that are closest to the user 102 at any given time by simply uttering the proximity-based group hotword "Hey nearby devices" or "Hey nearby device" in a single query, such that the AEDs that detect the spoken group hotword cooperate with each other by performing arbitration to select the device or devices that are closest to the user 102 to perform the action specified by the query. Advantageously, the proximity-based group hotword allows the user 102 to target only the subset of one or more AEDs that are currently closest in proximity to the user 102, without requiring the user to explicitly identify any particular AED among the subset of one or more AEDs.

[0058] In the illustrated embodiment, a proximity-based group hotword "Hey nearby devices" is assigned to all AEDs associated with the user by default. The user may interact with a selection object to remove any AED from the selected group of AEDs to which the proximity-based group hotword is assigned. For example, the GUI 208 may receive a user input indication indicating a user interaction with a selection object 210c corresponding to the AED 104 named Smartphone to remove the smartphone from the group to which the proximity-based group hotword is assigned. Thus, the smartphone will not detect or respond to the user speaking "Hey Nearby Devices" even if the smartphone is the nearest AED to the user 102.

[0059] Other implicit group hotwords include device type group hotwords that can be assigned to a selected group of AEDs that all share the same device type. In the illustrated embodiment, the implicit group hotword selection screen 200b lists only Speakers 1-7 as eligible AEDs to which the implicit device type group hotword "Hey smart speakers" should be assigned because the AEDs named Speakers 1-7 all contain the same device type of smart speaker. Thus, the user 102 may interact with a selection object 210c displayed in the GUI 208 to select the AEDs in the group to which the group hotword "Hey smart speakers" should be assigned (or deselect the AEDs from the group), and then speak an utterance that includes the group hotword "Hey smart speakers" to target all AEDs associated with the user 102 that contain a device type of smart speaker in a single query.

[0060] The implicit group hotword selection screen 200b also displays two different implicit attribute-based hotwords that may each be assigned to a respective selected group of AEDs 104 that share a common attribute. For example, a first attribute-based group hotword includes "Blue Speakers," which the user 102 may assign to speaker 1 and speaker 2 to allow the user to address all smart speakers that share the attribute of having the color blue (or otherwise labeled "Blue") in a single query. Similarly, a second attribute-based group hotword includes "Red Speakers," which the user may assign to speaker 3 and speaker 4 to allow the user to address all smart speakers that share the attribute of having the color red (or otherwise labeled "Red") in a single query. As will become apparent with reference to FIG. 4 below, the attribute-based group hotwords may further narrow down the particular group of AEDs that the user desires to address.

[0061] 4 illustrates an exemplary voice-enabled environment 400 including multiple AEDs 104 associated with a user 102. In the illustrated embodiment, the voice-enabled environment 400 is the home of the user 102 having multiple rooms and areas including a living room, a kitchen, and a bedroom. Although the voice-enabled environment 400 is shown as a home in FIG. 4, the voice-enabled environment 400 may include any environment that implements a network of multiple AEDs, such as an educational environment, a business, or an automobile.

[0062] The AED 104 includes seven smart speakers 104a-g (SP1-7), a smart display 104h, a smart TV 104i, and a smartphone 104j positioned throughout the voice-enabled environment. Smart speakers SP1 104a, SP2 104b, SP3 104c, SP4 104d, and smart TV 104i are positioned in the living room of the voice-enabled environment 400, in addition to smartphone 104j, which is portable / mobile and may be moved throughout various rooms / areas in the voice-enabled environment 400. Smart speakers SP1-SP4 and smart TV 104i may be combined or otherwise paired together to form a respective area named "Family Room." Further, the first smart speaker SP1 and the second smart speaker SP2 may be labeled as "Blue" devices to describe their physical attributes of being blue, and the third smart speaker SP3 and the fourth smart speaker SP4 may be labeled as "Red" devices to describe their attributes of being red. Other attributes may be used, such as size (e.g., large vs. small), type / brand (e.g., high fidelity speaker), or any other label that a user uses to identify / group AEDs within a particular area or across multiple areas in the voice-enabled environment 400. The voice-enabled environment 400 also shows a smart speaker SP5 104e and a smart display 104h located in the kitchen and coupled / paired together to form a respective area named "Kitchen". Similarly, smart speaker SP6 104f and smart speaker SP7 104g may be coupled / paired together to form a respective area named "Bedroom."

[0063] 5 illustrates an exemplary hotword registry 500 including a list of hotwords 50 each assigned to a respective selected group of AEDs 104 disposed in the voice-enabled environment 400. One or more of the AEDs 104 may each store the hotword registry 500 on their respective local memory hardware 12. An AED 104 that does not store the hotword registry 500 may access the hotword registry 500 on the other AEDs 104 to discover other AEDs 104 in the network and ascertain which hotwords are assigned to which AEDs. Additionally or alternatively, the hotword registry 500 may be stored on a centralized device in communication with one or more of the AEDs. For example, the hotword registry 500 may be stored on a remote server, such as a remote server associated with the digital assistant 105 that associates the hotword registry with a profile for the user 102.

[0064] Each of the AEDs 104 is assigned a default hotword 50 called "Hey Assistant" that, when detected in the streaming audio by one or more of the AEDs, triggers the AEDs 104 to wake up from a low power state and invoke the first digital assistant 105 to start processing one or more other terms following the default hotword 50. Here, the first digital assistant may be affiliated with a first voice assistant service (e.g., GOOGLE's Assistant). In addition, the smart speaker SP2 104b, the smart display 104h, and the smartphone 104j are also assigned another default hotword called "Other Assistant" that, when detected in the streaming audio by any one of the AEDs 104b, 104h, 104j, triggers the AEDs to invoke the second digital assistant to start processing one or more other terms following the other default hotword. Here, the second digital assistant may be affiliated with a second voice assistant service (e.g., AMAZON's Alexa or APPLE's Siri) different from the first voice assistant service. Furthermore, each AED 104 may be assigned a unique device-specific hotword that is configured to be detected in the streaming audio only by the corresponding AED if the user desires to address only the corresponding AED. For example, the unique device-specific hotword assigned to the first smart speaker SP1 104a in the environment 400 may include an identifier of the AED, such as "Hey Device 1" or simply "Device 1", or may include a device type and / or other attributes associated with the AED, such as "Hey Smart Speaker 1" or simply "Smart Speaker 1".

[0065] As previously described, the group hot words assigned to the AEDs of each selected group may include manually enabled hot words 50 assigned by the user 102 to the AEDs 104 of each selected group. The manually enabled hot words may be custom hot words created by the user 102 and / or predefined hot words available to be selected by the user 102. The predefined hot words may be associated with a predefined hot word model trained to detect the associated hot word. However, custom hot words created by the user 102 may require the user to train a custom hot word detection model to detect the custom hot word. For example, the user 102 may speak one or more utterances that include the custom hot word.

[0066] In some embodiments, the user 102 provides a voice input (e.g., speech) 106 (FIG. 1A) to select each AED that the user wants to include in the selected group of AEDs, and assigns the manually enabled group hotword 50g to the selected group of AEDs 104. Similarly, the user may provide subsequent voice inputs 136, 146 (FIGS. 1B and 1C) to update the selected group of AEDs 104 by adding one or more additional AEDs to the existing selected group of AEDs (FIG. 1B) and / or removing one or more AEDs from the existing selected group of AEDs (FIG. 1C). Additionally or alternatively, the user may provide a user input instruction indicating a user interaction with one or more objects displayed in the GUI 208, such as the user-defined group hotword selection screen 200a of FIG. 2A, to instruct the digital assistant 105 to enable the manual group hotword and select the AEDs in the group to which the group hotword should be assigned. The user 102 may provide subsequent user-interactive instructions to the GUI 208 to update the selected group of AEDs 104 by adding additional AEDs to the existing selected group and / or removing AEDs from the existing selected group.

[0067] In an exemplary hotword registry 500 for the voice-enabled environment 400, the user 102 activates and assigns a manual group hotword "Family Room Devices" to each selected group of AEDs including smart speakers SP1-SP4 104a-d and smart TV 104i located in an area named "Family Room". The user 102 also activates and assigns a manual group hotword "Kitchen Devices" to smart speaker SP5 104e and smart display 104h located in an area named "Kitchen". Similarly, a manual group hotword "Bedroom Speakers" is activated and assigned by the user to smart speakers SP6, SP7 104f-g located in an area named "Bedroom". Here, each manually activated group hotword may describe a location / area within the voice-enabled environment 400 (e.g., the user's home) where each selected group of AEDs with the corresponding group hotword 50 assigned thereto are located. Of note, the manually enabled group hotword “Bedroom Speakers” assigned to smart speakers SP6 and SP7 describes the device type (e.g., smart speaker) associated with the AEDs in each selected group.

[0068] In the illustrated example, the user 102 has not assigned any manually enabled group hotwords to the smartphone 104j, but if a corresponding group hotword preceding the query is detected in the streaming audio, one or more of the AEDs in the selected group may be updated to add / include the smartphone 104j to enable the smartphone 104j to cooperate with other AEDs in the respective group to perform the action specified by the query.

[0069] The example hotword registry 500 of FIG. 5 also illustrates that a plurality of different implicit group hotwords 50g are each assigned to a different respective selected group of AEDs 104. As described above with reference to FIG. 2B, the software application 205 associated with the digital assistant 105 may render an implicit group hotword selection screen 200b in the GUI 208, and the user 102 may interact with the screen 200b to browse the available implicit group hotwords and select the AEDs of the group to be assigned to the implicit group hotword. For example, a proximity-based group hotword "Hey nearby devices" and / or "Hey nearby devices" may be assigned to all of the AEDs 104a-j associated with the user 102 located in the voice-enabled environment 400 of FIG. 4. Thus, a user 102 may target one or more AEDs in the voice-enabled environment 400 that are closest to the user 102 at any given time by simply uttering the proximity-based group hotword "Hey nearby devices" or "Hey nearby device" in a single query, such that the AEDs that detect the spoken group hotword cooperate with each other by performing arbitration to select the device or devices that are closest to the user 102 to perform the action specified by the query. Advantageously, the proximity-based group hotword allows the user 102 to target only the subset of one or more AEDs that are currently closest in proximity to the user 102, without requiring the user to explicitly identify any particular AED among the subset of one or more AEDs.

[0070] Each AED 104 assigned an implicit proximity-based group hotword may trigger a wake-up process to execute a hotword detection model to detect the presence of the group hotword in the streaming audio to initiate speech recognition on the audio. Since the implicit group hotword in this example is proximity-based, multiple AEDs 104 may detect the group hotword in the captured streaming audio, but each of these AEDs 104 may then process the audio to determine their respective proximity values ​​to the user 102, and then perform arbitration using these proximity values ​​across the multiple AEDs 104 to select one or more of these AEDs 104 to perform the action specified by the user's query. Here, AEDs 104 that are outside some upper distance threshold from the user may be ineligible to perform the query. Optionally, AEDs 104 that are within some lower distance threshold, such as a smartphone AED in the user's pocket that detects the proximity-based group hotword "Hey nearby device(s)", may also be ineligible to respond to the query. The lower distance threshold can be applied depending on the type of query. For example, if the query is a search query where nearby devices are to provide search results as synthesized voice, the fact that the smartphone 104j is too close to the user 102, indicating that the smartphone 104j is in the user's pocket, disqualifies the smartphone 104j from fulfilling the query, since the synthesized voice will be muted and will not be understood / heard by the user 102. The user also has the option to add / remove the AED to / from selected groups that have been assigned proximity-based group hotwords.Additionally, the selected device closest to the user 102 may perform speech recognition and query interpretation to determine whether "nearby device" was spoken by the user 102 to indicate that the user 102 desires only the single device closest to the user to complete the query, or whether "nearby devices" was spoken by the user 102 to indicate that the user desires two or more nearby devices to complete the query.

[0071] 5 also illustrates two different implicit device type group hotwords, each assigned to a respective selected group of AEDs 104 in the voice-enabled environment 400 associated with a respective same device type. For example, a first device type group hotword includes "Smart Speakers" that is assigned to all smart speakers SP1-SP7 in the voice-enabled environment 400 to allow a user to target all AEDs 104a-h associated with the user 102 that include a device type of smart speaker in a single query. Here, the device type group hotword "Smart Speakers" targets four smart speakers SP1-SP4 located in an area named "Family Room", a smart speaker SP5 located in an area named "Kitchen", and smart speakers SP6, SP7 located in an area named "Bedroom". It should be noted that a manually enabled group hotword called "Family Room Devices" is also assigned to smart speakers SP1 to SP4, a manually enabled group hotword called "Kitchen Devices" is also assigned to smart speaker SP5, and a manually enabled group hotword called "Bedroom Speakers" is also assigned to smart speakers SP5 and SP7.

[0072] Additionally, a second device type group hotword includes "Smart Displays" assigned to each selected group of AEDs including the smart display 104h located in the area named "Kitchen" and the smart TV 104i located in the area named "Family Room." Notably, a manually enabled group hotword of "Family Room Devices" is also assigned to the smart TV 104i, and a manually enabled group hotword of "Kitchen Devices" is also assigned to the smart display 104h.

[0073] With continued reference to the voice-enabled environment 400 of FIG. 4 and the exemplary hotword registry 500 of FIG. 5, two different implicit attribute-based hotwords are each assigned to a respective selected group of AEDs 104 in the voice-enabled environment 400 that share a common attribute. For example, a first attribute-based group hotword includes "Blue Speakers" assigned to a first smart speaker SP1 and a second smart speaker SP2 located in an area named "Family Room" of the environment 400 to allow a user to target all smart speakers 104a-b that share the attribute of having the color blue (or otherwise labeled as "Blue") in a single query. Similarly, a second attribute-based group hotword includes "Red Speakers" assigned to all smart speakers 104c-d that share the attribute of having the color red (or otherwise labeled as "Red") in a single query. It should be noted that the first and second smart speakers SP1 and SP2, which are assigned the group hotword "Blue Speakers," and the third and fourth smart speakers SP3 and SP4, which are assigned the group hotword "Red Speakers," are also among the seven (seven) smart speakers 104a-g in the selected group that are assigned the device type group hotword "Smart Speakers," and the five (five) AEDs 104a-d, 104i in the selected group that are assigned the manually enabled group hotword "Family Room Devices," including smart speakers SP1-4 and the smart TV 104i. Thus, attribute-based group hotwords can further narrow down the specific group of AEDs that the user desires to target.

[0074] 4, in some implementations, a user 102 located in an area of ​​the voice-enabled environment 400 named Living Room speaks the utterance 406 "Speaker 1 & Smart TV, Play music videos," which corresponds to a command 418 for the digital assistant 105 to execute a long-lived action of streaming a music video for playback on the first smart speaker SP1 104a and the smart TV 104i. The digital assistant 105 may execute across all of the AEDs 104 in the voice-enabled environment 400. The terms "Speaker 1" and "Smart TV" are prepended to the command 418 to correspond to the respective device-specific hotwords 50a and 50b assigned to the first smart speaker SP1 104a and "Smart TV," respectively. Here, the first smart speaker SP1 104a executes the hot word detection model trained to detect the hot word 50a "Speaker 1" in the audio data corresponding to the utterance 14 to trigger the SP1 104a to wake up from the low power state and start processing on the audio data. At the same time, the smart TV 104i executes the hot word detection model trained to detect the hot word 50b "Smart TV" in the audio data corresponding to the utterance 14 to trigger the smart TV 104i to wake up from the low power state and start processing on the audio data. After processing the audio data by performing speech recognition to generate ASR results and performing query interpretation on the ASR results to identify commands 418 that perform long-lived actions on the first smart speaker SP1 104a and the smart TV 104i, SP1 and the smart TV cooperate with each other to perform the long-lived actions. For example, the smart TV 104i may stream video data to display the video portion of a music video, while the SP1 may stream audio data to audibly output the audio portion of the music video.

[0075] Continuing with the example, the digital assistant 105 is also configured to automatically create an action-specific group hotword while the long-lived action is in progress, and assign the action-specific group hotword to a selected group of AEDs including the first smart speaker SP1 104a and the smart TV 104i that are performing the long-lived action. The user 102 may use the action-specific group hotword in a follow-up query related to the long-lived action of playing music videos on the first smart speaker SP1 104a and the smart TV 104i. Thus, the AEDs corresponding to the first smart speaker SP1 104a and the smart TV 104i each receive an assignment instruction that assigns the action-specific group hotword "Music Videos" automatically created by the digital assistant 105. The user 102 may then address the long-lived action to be executed on the first smart speaker SP1 104a and the smart TV 104i by simply speaking the phrase "Music Videos" followed by a query / command that controls the long-lived action. For example, the user 102 may say "Music Videos, next song" or "Music Videos, turn up the volume" to instruct the first smart speaker SP1 to advance to the next music video or to increase the volume. In response to creating the action-specific group hotword and providing the assignment command to the first smart speaker SP1 104a and the smart TV 104i, the digital assistant 105 may update the hotword registry 500 of FIG. 5 to indicate that the action-specific group hotword "Music Videos" is assigned to the selected group of AEDs that includes the first smart speaker SP1 104a and the smart TV 104i.

[0076] In some implementations, the first smart speaker SP1 104a outputs for audible playback a synthesized voice corresponding to a response from the digital assistant 105 indicating that execution of the long-lived action is in progress and an action-specific group hotword that was automatically created for use in follow-up queries related to the long-lived action. For example, FIG. 4 shows SP1 104a outputting a synthesized voice corresponding to a response 450 from the digital assistant 105 including "Ok, playing music videos now...In the future, you can control playback using the 'Music Videos' hotword." The digital assistant 105 is configured to cancel the use of the action-specific group hotword related to the long-lived action when the long-lived action is terminated. Thus, when the long-lived action is terminated, the digital assistant 105 may update the hotword registry 500 to remove the action-specific hotword and may notify the selected group of AEDs that no longer respond to the action-specific group hotword. The user 102 may decline to use an action-specific group hotword at any time by providing a voice input or via the GUI of FIGS. 2A and 2B.

[0077] 6 is a flow chart of an example configuration of operations for a method 600 of activating and assigning a group hotword to a selected group of assistant-enabled devices (AEDs) 104. In operation 602, the method 600 includes receiving an assignment instruction in the data processing hardware 10 of a first AED 104a to assign a group hotword 50g to the AEDs 104 of the selected group associated with a user. The AEDs 104 of the selected group include the first AED 104a and one or more other AEDs 104b-n. Each AED in the selected group of AEDs is configured to wake up from a low power state when the group hotword 50g is detected in streaming audio by at least one of the AEDs in the selected group of AEDs.

[0078] At operation 604, the method 600 includes receiving, at the data processing hardware 10, audio data 20 corresponding to an utterance 126 spoken by the user 102. The audio data 20 includes a query 128 that specifies an action to be performed. At operation 606, the method 600 includes detecting, by the data processing hardware 10, the group hot word 50g in the audio data 20 using the hot word detection model 114.

[0079] At operation 608, in response to detecting the group hot word 50g in the audio data 20, the method 600 includes triggering, by the data processing hardware 10, the first AED 104a to wake up from a low power state, and executing, by the data processing hardware 10, a collaboration routine 150 that causes the first AED 104a and each of the other AEDs 104 in the selected group of AEDs to collaborate with each other to accomplish performance of the action specified by the query 128.

[0080] A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform tasks. In some embodiments, a software application may be referred to as an "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0081] Non-transient memory may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0082] 7 is a schematic diagram of an exemplary computing device 700 that may be used to implement the systems and methods described herein. The computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components, component connections and relationships, and component functions illustrated herein are intended to be merely exemplary and are not intended to limit the implementation of the inventions described and / or claimed herein.

[0083] Computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 connected to memory 720 and high-speed expansion port 750, and a low-speed interface / controller 760 connected to low-speed bus 770 and storage device 730. Each of components 710, 720, 730, 740, 750, and 760 may be connected to each other using various buses and implemented on a common motherboard or in other manners as appropriate. Processor 710 can process instructions for execution within computing device 700, including instructions stored in memory 720 or on storage device 730 to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 780 coupled to high-speed interface 740. In other implementations, multiple processors and / or multiple buses may be used along with multiple memories and multiple types of memories as appropriate. Also, multiple computing devices 700 may be connected together, with each device providing a portion of the required operations (eg, as a server bank, as a group of blade servers, or as a multi-processor system).

[0084] The memory 720 stores information non-transiently within the computing device 700. The memory 720 may be a computer-readable medium, a volatile memory unit(s), or a non-volatile memory(s). The non-transient memory 720 may be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disks or tapes.

[0085] The storage device 730 can provide mass storage for the computing device 700. In some implementations, the storage device 730 is a computer-readable medium. In various different implementations, the storage device 730 can be an array of devices, including a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. In further implementations, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 720, the storage device 730, or a memory on the processor 710.

[0086] While the high-speed controller 740 manages bandwidth-intensive operations for the computing device 700, the low-speed controller 760 manages less bandwidth-intensive operations. Such allocation of duties is merely exemplary. In some implementations, the high-speed controller 740 is coupled to a memory 720, a display 780 (e.g., via a graphics processor or accelerator), and a high-speed expansion port 750 that may accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to a storage device 730 and a low-speed expansion port 790. The low-speed expansion port 790, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or, for example, via a network adapter, to a networking device, such as a switch or a router.

[0087] The computing device 700 may be implemented in a number of different forms, as shown in the figure: For example, the computing device 700 may be implemented as a standard server 700a, or a group of multiple duplicates of such a server 700a, a laptop computer 700b, or part of a rack server system 700c.

[0088] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0089] These computer programs (also known as programs, software, software applications or codes) contain machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, computer-readable apparatus and / or computer-readable device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0090] The processes and logic flows described herein can be executed by one or more programmable processors, also referred to as data processing hardware, which execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by special purpose logic circuits, such as FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits). Processors suitable for executing computer programs include, by way of example, both general purpose and special purpose microprocessors, and any one or more processors of any kind of digital computer. In general, a processor receives instructions and data from a read-only memory or from a random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. In general, a computer also includes one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or is operatively coupled to receive data from such mass storage devices, send data to such mass storage devices, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, non-volatile media, and non-volatile memory devices, including, by way of example, semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices, magnetic disks, e.g., internal hard disks or removable disks, magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0091] To enable interaction with a user, one or more aspects of the present disclosure can be implemented on a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices can also be used to enable interaction with the user, for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and input from the user can be received in any form, including acoustic input, speech input, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from devices used by the user, for example, by sending web pages to a web browser on the user's client device in response to requests received from the web browser.

[0092] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0093] 10 Data Processing Hardware 12 Memory Hardware 14, 106, 126, 136, 146, 406 utterances 16 Microphones 18 Audio Output Devices 20 Audio Data 50, 50a, 50b, 50c, 50d, 50e, 50f, 50g Hotwords 100 Systems 102 users 104, 104b, 104c, 104d, 104e, 104f, 104g, 104h, 104i, 104j Assistant Enabled Devices 105 Digital Assistant 108 Hotwords Detector 114 Hot Word Model 116 Voice Recognizer 118 Query 120 Servers 122 songs 132 Network 150 Collaborative Routines 200, 200a, 200b Group hotword selection screen 205 Software Applications 208 Graphical User Interface 210, 210a, 210b, 210c, 210d objects 300 Leader Selection Process 302 Device characteristics 304 Device state change 306 Update 310 Selection Command 400 Voice-enabled environment 418 Command 450 Audio 500 Hotword Registry 700, 700a, 700b, 700c Computing Devices 710 Processor 720 Memory 730 Storage Devices 740, 760 Interface / Controller 750, 790 Expansion Port 770 Bus 780 Display

Claims

1. receiving, in data processing hardware of a first assistant-enabled device (AED), an assignment command to assign a group hotword to a selected group of AEDs associated with a user, the selected group of AEDs including the first AED and one or more other AEDs, each AED in the selected group of AEDs configured to execute a trained hotword detection model to detect the presence of the group hotword in streaming audio without performing speech recognition, and to wake up from a low power state when the group hotword is detected in the streaming audio by at least one of the AEDs in the selected group of AEDs, and other AEDs associated with the user that are not in the selected group of AEDs are configured not to wake up from a low power state when the group hotword is detected in the streaming audio by the at least one of the AEDs in the selected group of AEDs; receiving, in the data processing hardware, audio data corresponding to utterances spoken by the user, the audio data including the group hotwords and a query specifying an action to be performed; detecting, by the data processing hardware, the group hot word in the audio data using the hot word detection model operating in the first AED without performing speech recognition on the audio data; in response to detecting the group hot word in the audio data, triggering the first AED to wake up from the low power state by the data processing hardware; wake up each other AED in the selected group of AEDs that did not detect the group hotword in the audio data by the data processing hardware without performing voice recognition on the audio data, to cooperate with all AEDs in the selected group of AEDs to accomplish performance of the action specified by the query; and executing, by said data processing hardware, a collaboration routine that causes said first AED and each other AED in said selected group of AEDs to collaborate with each other to accomplish the execution of said action specified by said query, wherein multiple AEDs in said selected group of AEDs perform different responsibilities related to said action; The method includes:

2. and after receiving the assignment command, executing, by the data processing hardware, a leader selection process for selecting, from the AEDs of the selected group, one or more AEDs to listen for the presence of the group hotword in the streaming audio on behalf of the AEDs of the selected group based on respective device characteristics associated with each AED in the selected group; The method of claim 1 , wherein at least the first AED in the selected group of AEDs is selected by the leader selection process to listen for the presence of the group hotword in the streaming audio.

3. updating the selected group of AEDs to which the group hotword is assigned to add one or more additional AEDs to the selected group of AEDs and / or remove one or more of the AEDs from the selected group of AEDs; or the occurrence of a device state change in one of the AEDs in the selected group of AEDs.

3. The method of claim 2, further comprising the step of re-executing, by the data processing hardware, the leader selection process in response to at least one of:

4. 3. The method of claim 2, wherein the respective device characteristics associated with each AED in the selected group of AEDs include at least one of processing capability, device type, user configurable device settings, power usage, battery level, physical location of the AED, or network capabilities.

5. in response to detecting the group hot word in the audio data, identifying, by the data processing hardware, each of the one or more other AEDs in the selected group of AEDs to which the group hotword has been assigned by accessing a hotword registry, the hotword registry including a list of one or more hotwords each assigned to one or more AEDs associated with the user, the list of one or more hotwords including the group hotword; 2. The method of claim 1, wherein calling each other AED in the selected group of AEDs is based on each of the one or more other AEDs being assigned the group hotword identified by accessing the hotword registry.

6. The hotword registry includes: the first AED; At least one other AED associated with the user; or A server in communication with the first AED. The method of claim 5, wherein the at least one of the following is stored in the at least one memory:

7. The step of receiving an assignment command to assign the group hotword to the AEDs of the selected group includes: Receiving a voice input from the user requesting a digital assistant to activate the group hotword and assign the group hotword to the AEDs of the selected group; instructing a speech recognizer to perform speech recognition on the audio data to generate an automated speech recognition (ASR) result for the speech input; performing query interpretation on the ASR results for the speech input to identify a group hotword query that specifies names of the group hotwords to be enabled and each AED in the selected group of AEDs to which the group hotwords should be assigned; 2. The method of claim 1, comprising:

8. 2. The method of claim 1, wherein the step of receiving the assignment instruction to assign the group hotword to the AEDs of the selected group includes receiving a user input instruction indicating a user interaction with one or more objects displayed in a graphical user interface to instruct a digital assistant to activate the group hotword and select the AEDs of the group to which the group hotword is to be assigned.

9. The method further includes receiving, in the data processing hardware, previous audio data spoken by the user corresponding to a command for a digital assistant to perform a long-lived action at two or more AEDs associated with the user, the digital assistant being configured to automatically create the group hotword for use in a follow-up query related to the long-lived action; 2. The method of claim 1, wherein receiving the assignment instruction to assign the group hotword to the selected group of AEDs includes receiving the group hotword automatically created by the digital assistant, and the selected group of AEDs to which the group hotword is assigned includes the two or more AEDs that perform the long-lived action.

10. outputting, by the data processing hardware, for audible playback from the first AED, a synthesized speech corresponding to a response from the digital assistant indicating that execution of the long-lived action is in progress and the automatically created group hotword for use in a follow-up query related to the long-lived action; The method of claim 9 , wherein the digital assistant revokes use of the automatically created group hotword when the long-lived action ends.

11. each AED in the selected group of AEDs is associated with the same device type or shares a common attribute; the group hotwords assigned to the AEDs of the selected group include an implied hotword that uniquely identifies the device type or the common attribute associated with the AEDs of the selected group; The method of claim 1 , wherein the hot word detection model is pre-trained to detect the presence of the group hot word in streaming audio without performing speech recognition on the audio data.

12. the group hotwords assigned to the selected group of AEDs include proximity-based hotwords that address only a subset of one or more AEDs from the selected group of AEDs that are currently closest in proximity to the user, without explicitly identifying any particular AED within the subset of one or more AEDs; the hot word detection model is pre-trained to detect the presence of the group hot word in streaming audio without performing speech recognition on the audio data; 2. The method of claim 1, wherein the step of executing the collaboration routine causes the first AED and each other AED in the selected group of AEDs to determine a respective proximity value to the user and to perform arbitration using the respective proximity values ​​across the selected group of AEDs to select the subset of the one or more AEDs that should perform the operation specified by the query.

13. The group hot words are: Custom hot words created by said user; or A predefined hotword selected from a list of one or more available group hotwords The method of claim 1 , comprising one of:

14. During execution of the collaboration routine, the first AED and each other AED in the selected group of AEDs: generating a speech recognition result for the audio data; performing query interpretation on the speech recognition result to determine that the speech recognition result identifies the query that specifies the action to be performed; and Sharing the query interpretation performed on the speech recognition results with the other AEDs in the selected group of AEDs.

2. The method of claim 1, wherein the AEDs cooperate with each other by designating one of the AEDs in the selected group for a first AED operation.

15. the query specifying the action to be performed includes a query for AEDs in the selected group that perform long-lived actions; During execution of the collaboration routine, each AED in the selected group of AEDs: pairing with one another for the duration of said long-lived operation; and Coordinating the execution of sub-actions related to said long-lived actions. The method of claim 1 , wherein the first and second inputs cooperate with each other by:

16. the actions specified by the query include device-level actions to be performed on each AED in the selected group of AEDs; 2. The method of claim 1, wherein during execution of the collaboration routine, each AED in the selected group of AEDs collaborates with each other by independently performing execution of the device-level operations.

17. Data processing hardware; memory hardware in communication with said data processing hardware; A first assistant-enabled device (AED) including: The memory hardware, when executed on the data processing hardware, causes the data processing hardware to receiving an assignment instruction to assign a group hotword to a selected group of AEDs associated with a user, the selected group of AEDs including the first AED and one or more other AEDs, each AED in the selected group of AEDs configured to execute a trained hotword detection model to detect the presence of the group hotword in streaming audio without performing voice recognition and to wake up from a low power state when the group hotword is detected in the streaming audio by at least one of the AEDs in the selected group of AEDs, and other AEDs associated with the user that are not in the selected group of AEDs configured not to wake up from a low power state when the group hotword is detected in the streaming audio by the at least one of the AEDs in the selected group of AEDs; receiving audio data corresponding to utterances spoken by the user, the audio data including the group hotword and a query specifying an action to be performed; detecting the group hot word in the audio data without performing speech recognition on the audio data using the hot word detection model operative in the first AED; in response to detecting the group hot word in the audio data, triggering the first AED to wake up from the low power state; wake up each other AED in the selected group of AEDs that did not detect the group hotword in the audio data without performing voice recognition on the audio data, from the low power state, and invoke all AEDs in the selected group of AEDs to cooperate with each other AED in the selected group of AEDs to accomplish performance of the action specified by the query; and executing a collaboration routine that causes the first AED and each other AED in the selected group of AEDs to cooperate with each other to accomplish the performance of the action specified by the query, wherein multiple AEDs in the selected group of AEDs perform different responsibilities related to the action; A first assistant-enabled device (AED) that stores instructions to perform an action including:

18. The operations further include, after receiving the assignment command, performing a leader selection process to select one or more AEDs from the selected group of AEDs to listen for the presence of the group hotword in the streaming audio on behalf of the AEDs in the selected group based on respective device characteristics associated with each AED in the selected group of AEDs; 18. The first AED of claim 17, wherein at least the first AED in the selected group of AEDs is selected by the leader selection process to listen for the presence of the group hotword in the streaming audio.

19. The operation includes: updating the selected group of AEDs to which the group hotword is assigned to add one or more additional AEDs to the selected group of AEDs and / or remove one or more of the AEDs from the selected group of AEDs; or the occurrence of a device state change in one of the AEDs in the selected group of AEDs.

20. The first AED of claim 18, further comprising: re-running the leader selection process in response to at least one of:

20. 20. The first AED of claim 18, wherein the respective device characteristics associated with each AED in the selected group of AEDs include at least one of processing capability, device type, user configurable device settings, power usage, battery level, physical location of the AED, or network capabilities.

21. The operation, in response to detecting the group hot word in the audio data, comprises: identifying each of the one or more other AEDs in the selected group of AEDs to which the group hotword is assigned by accessing a hotword registry, the hotword registry including a list of one or more hotwords each assigned to one or more AEDs associated with the user, the list of one or more hotwords including the group hotword; 18. The first AED of claim 17, wherein calling each other AED in the selected group of AEDs is based on each of the one or more other AEDs being assigned the group hotword identified by accessing the hotword registry.

22. The hotword registry includes: the first AED; At least one other AED associated with the user; or A server in communication with the first AED. The first AED of claim 21, wherein the first AED is stored in at least one of the following:

23. Receiving the assignment instruction to assign the group hotword to the AEDs of the selected group includes: Receiving a voice input from the user requesting a digital assistant to activate the group hotword and assign the group hotword to the AEDs of the selected group; instructing a speech recognizer to perform speech recognition on the audio data to generate an automated speech recognition (ASR) result for the speech input; performing query interpretation on the ASR results for the voice input to identify a group hotword query that specifies names of the group hotwords to be enabled and each AED in the selected group of AEDs to which the group hotwords should be assigned; 18. The first AED of claim 17, comprising:

24. 18. The first AED of claim 17, wherein receiving the assignment instruction to assign the group hotword to the AEDs of the selected group includes receiving a user input instruction indicating a user interaction with one or more objects displayed in a graphical user interface to instruct a digital assistant to activate the group hotword and select the AEDs of the group to which the group hotword is to be assigned.

25. The operations further include receiving previous audio data spoken by the user and corresponding to a command for a digital assistant to perform long-lived actions at two or more AEDs associated with the user, the digital assistant being configured to automatically create the group hotword for use in follow-up queries related to the long-lived actions; 18. The first AED of claim 17, wherein receiving the assignment instruction to assign the group hotword to the AEDs of the selected group includes receiving the group hotword automatically created by the digital assistant, and the AEDs of the selected group to which the group hotword is assigned include the two or more AEDs that perform the long-lived action.

26. The operations further include outputting, for audible playback from the first AED, a synthesized speech corresponding to a response from the digital assistant indicating that execution of the long-lived action is in progress and the automatically created group hotword for use in a follow-up query related to the long-lived action; 26. The first AED of claim 25, wherein the digital assistant revokes use of the automatically created group hotword when the long-lived action ends.

27. each AED in the selected group of AEDs is associated with the same device type or shares a common attribute; the group hotwords assigned to the AEDs of the selected group include an implied hotword that uniquely identifies the device type or the common attribute associated with the AEDs of the selected group; 20. The first AED of claim 17, wherein the hot word detection model is pre-trained to detect the presence of the group hot word in streaming audio without performing speech recognition on the audio data.

28. the group hotwords assigned to the selected group of AEDs include proximity-based hotwords that address only a subset of one or more AEDs from the selected group of AEDs that are currently closest in proximity to the user, without explicitly identifying any particular AED within the subset of one or more AEDs; the hot word detection model is pre-trained to detect the presence of the group hot word in streaming audio without performing speech recognition on the audio data; 18. The first AED of claim 17, wherein executing the collaboration routine causes the first AED and each other AED in the selected group of AEDs to each determine a respective proximity value to the user and to perform arbitration using the respective proximity values ​​across the selected group of AEDs to select the subset of the one or more AEDs that should perform the operation specified by the query.

29. The group hot words are: Custom hot words created by said user; or A predefined hotword selected from a list of one or more available group hotwords 18. The first AED of claim 17, comprising one of:

30. During execution of the collaboration routine, the first AED and each other AED in the selected group of AEDs: generating a speech recognition result for the audio data; performing query interpretation on the speech recognition result to determine that the speech recognition result identifies the query that specifies the action to be performed; and Sharing the query interpretation performed on the speech recognition results with the other AEDs in the selected group of AEDs.

18. The first AED of claim 17, wherein the first AED cooperates with the second AED by designating one of the AEDs in the selected group for a second AED.

31. the query specifying the action to be performed includes a query for AEDs in the selected group that perform long-lived actions; During execution of the collaboration routine, each AED in the selected group of AEDs: pairing with one another for the duration of said long-lived operation; and Coordinating the execution of sub-actions related to said long-lived actions. The first AED of claim 17, which cooperate with each other by

32. the actions specified by the query include device-level actions to be performed on each AED in the selected group of AEDs; 18. The first AED of claim 17, wherein during execution of the collaboration routine, each AED in the selected group of AEDs collaborates with each other by independently performing execution of the device-level operations.

Images