hot phrase

Abstract

The present invention relates to a method (600) comprising a first assistant-enabled device (AED) (104) receiving a selected group assignment instruction to assign a hot word group (50g) to an AED comprising a first AED and one or more other AEDs. Each AED is configured to wake up from a low-power state when the hot word group is detected in streaming audio by at least one AED. The method further comprises receiving audio data (20) corresponding to a utterance by a user (102) and a query (128) including an action to be performed. In response to the detection of the hot word group in the audio data, the method further comprises triggering the first AED to wake up from a low-power state and execute a cooperation routine (150) such that the first AED and each other AED in the selected group of AEDs cooperate with each other to perform the action specified by the query.

Description

Technical Field

[0001] This disclosure involves hot phrases. Background Technology

[0002] Voice-enabled environments (e.g., home, workplace, school, car, etc.) allow users to speak queries or commands aloud to a computer-based system, which processes and answers the queries and / or performs functions based on the commands. Voice-enabled environments can be implemented using a network of connected microphone devices distributed across various rooms or areas of the environment. These devices can use hot words to help identify when a given utterance is addressed to the system, rather than to another person present in the environment. Therefore, the device can operate and wake up from sleep or hibernation only when the detected utterance contains hot words. Once awakened, the device can continue performing more expensive processing, such as fully on-device automatic speech recognition (ASR) or server-based ASR. Summary of the Invention

[0003] One aspect of this disclosure provides a method for enabling and assigning a hot word group to a selected group of two or more assistant-enabled devices (AEDs). The method includes receiving, at the data processing hardware of a first assistant-enabled device (AED), an assignment instruction to assign the hot word group to a selected group of AEDs associated with a user. The selected group of AEDs includes the first AED and one or more other AEDs. Each AED in the selected group is configured to wake up from a low-power state when the hot word group is detected in streaming audio by at least one of the AEDs in the selected group. The method further includes receiving, at the data processing hardware, audio data corresponding to speech spoken by the user, the audio data including a query specifying an operation to be performed. In response to detecting the hot word group in the audio data, the method includes triggering the first AED to wake up from the low-power state by the data processing hardware. The method further includes executing a cooperation routine by the data processing hardware to cause the first AED and each of the other AEDs in the selected group to cooperate with each other to perform the operation specified by the query.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the method further includes performing a leader election process by the data processing hardware to elect one or more AEDs from the selected group of AEDs based on corresponding device attributes associated with each AED in the selected group of AEDs, to listen for the presence of the hot word group in the streaming audio on behalf of the selected group of AEDs. Here, at least the first AED in the selected group of AEDs is selected by the leader election process to listen for the presence of the hot word group in the streaming audio. In these implementations, the method may further include re-executing the leader election process by the data processing hardware in response to at least one of the following: updating the selected group of AEDs assigned the hot word group, adding one or more additional AEDs to the selected group of AEDs and / or deleting one or more of the AEDs from the selected group of AEDs; or a device state change occurring at one of the AEDs in the selected group of AEDs. Optionally, the corresponding device attributes associated with each AED in the selected group of AEDs may include at least one of processing power, device type, user-configurable device settings, power consumption, battery level, physical location of the AED, or network functionality.

[0005] In response to the detection of the hot word group in the audio data, the method may further include the data processing hardware invoking each other AED in a selected group of the AED that did not detect the hot word group in the audio data to wake up from the low-power state and cooperate with all the AEDs in the selected group of the AED to perform the operation specified by the query. In some embodiments, in response to the detection of the hot word group in the audio data, the method further includes the data processing hardware identifying each of the one or more other AEDs in the selected group of the AED assigned the hot word group by accessing a hot word registry. The hot word registry contains a list of one or more hot words, each hot word being assigned to one or more AEDs associated with the user (or user group), and the list of one or more hot words includes the hot word group. In these embodiments, invoking each other AED in the selected group of the AED is based on each of the one or more other AEDs assigned the hot word group by accessing the hot word registry. In some examples, the hot word registry is stored in at least one of the following: the first AED, at least one other AED associated with the user, or a server communicating with the first AED.

[0006] Optionally, receiving the assignment instruction to assign the hot word group to a selected group of AEDs may include: receiving voice input from the user requesting a digital assistant to enable the hot word group and assign it to a selected group of AEDs; instructing a speech recognizer to perform speech recognition on the audio data to generate an automatic speech recognition (ASR) result for the voice input; and performing query interpretation on the ASR result for the voice input to identify a hot word group query, the hot word group query specifying the hot word group to be enabled and the name of each AED in the selected group to which the hot word group is to be assigned. In some embodiments, receiving the assignment instruction to assign the hot word group to a selected group of AEDs includes: receiving a user input indication that instructs a user to interact with one or more objects displayed in a graphical user interface to instruct a digital assistant to enable the hot word group and select the group of AEDs to which the hot word group is to be assigned.

[0007] The method may further include receiving, at the data processing hardware, previous audio data spoken by the user and corresponding to a command from a digital assistant to perform a prolonged action on two or more AEDs associated with the user. The digital assistant is configured to automatically create the hotword group for subsequent queries related to the prolonged action. Here, receiving the assignment instruction to assign the hotword group to a selected group of AEDs includes receiving the hotword group automatically created by the digital assistant. The selected group of AEDs to which the hotword group is assigned includes the two or more AEDs performing the prolonged action. In some embodiments, the method further includes, by the data processing hardware, outputting synthesized speech corresponding to a response from the digital assistant for audible playback from the first AED, indicating that the prolonged action is being performed, and the automatically created hotword group for subsequent queries related to the prolonged action. In these embodiments, the digital assistant deactivates the use of the automatically created hotword group at the end of the prolonged action. In some examples, each AED in the selected group of AEDs is associated with the same device type or shares common attributes. The hot word group assigned to the selected group of AEDs includes implicit hot words that uniquely identify the device type or common attribute associated with the selected group of AEDs. The hot word detection model is pre-trained to detect the presence of the hot word group in the streaming audio without performing speech recognition on the audio data.

[0008] In some implementations, the hot word set assigned to the selected group of AEDs includes proximity-based hot words that address only a subset of one or more AEDs currently closest to the user within the selected group of AEDs, without explicitly identifying any specific AED within the subset of one or more AEDs. The hot word detection model is pre-trained to detect the presence of the hot word set in streaming audio without performing speech recognition on the audio data, and executes the collaborative routine to cause the first AED and each other AED in the selected group of AEDs to determine a corresponding proximity value relative to the user, and performs arbitration using the corresponding proximity value within the selected group of AEDs to elect the subset of one or more AEDs to perform the operation specified by the query. Optionally, the hot word set may include one of the following: custom hot words created by the user or predefined hot words selected from a list of one or more available hot word sets. During the execution of the collaboration routine, the first AED and each other AED in the selected group of AEDs collaborate with each other by specifying one of the AEDs in the selected group of AEDs to perform the following: generate speech recognition results for the audio data; perform query interpretation on the speech recognition results to determine that the speech recognition results identify the query specifying the operation to be performed; and share the query interpretation performed on the speech recognition results with the other AEDs in the selected group of AEDs.

[0009] In some implementations, the query specifying the operation to be performed includes a query for performing a long-term operation on a selected group of AEDs, and during the execution of the cooperative routine, each AED in the selected group of AEDs cooperates with each other by pairing up and coordinating the execution of sub-actions related to the long-term operation during the long-term operation. The operation specified by the query may include a device-level operation performed on each AED in the selected group of AEDs, and during the execution of the cooperative routine, each AED in the selected group of AEDs cooperates with each other by independently performing the device-level operation.

[0010] Another aspect of this disclosure provides a first assistant enabling device (AED) for enabling and assigning hot word groups to a selected group of two or more assistant enabling devices. The first AED includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving assignment instructions to assign hot word groups to a selected group of AEDs associated with a user. The selected group of AEDs includes the first AED and one or more other AEDs. Each AED in the selected group of AEDs is configured to wake up from a low-power state when at least one of the AEDs in the selected group of AEDs detects the hot word group in streaming audio. The operations also include receiving audio data corresponding to uttered words by the user and including a query specifying an operation to be performed. In response to detecting the hot word group in the audio data, the operations include triggering the first AED to wake up from the low-power state. The operations also include executing a cooperation routine such that the first AED and each of the other AEDs in the selected group of AEDs cooperate with each other to perform the operation specified by the query.

[0011] Implementations of this disclosure may include one or more of the following optional features. In some implementations, the operation further includes performing a leader election process to elect one or more AEDs from the selected group of AEDs based on corresponding device attributes associated with each AED in the selected group of AEDs, to listen for the presence of the hot word group in the streaming audio on behalf of the selected group of AEDs. Here, at least the first AED in the selected group of AEDs is elected by the leader election process to listen for the presence of the hot word group in the streaming audio. In these implementations, the operation may also include re-executing the leader election process in response to at least one of the following: updating the selected group of AEDs assigned the hot word group, adding one or more additional AEDs to the selected group of AEDs and / or deleting one or more AEDs from the selected group of AEDs, or a device state change occurring at one of the AEDs in the selected group of AEDs. Optionally, the corresponding device attributes associated with each AED in the selected group of AEDs may include at least one of processing power, device type, user-configurable device settings, power consumption, battery level, physical location of the AED, or network functionality.

[0012] In response to detecting the hot word group in the audio data, the operation may further include invoking each other AED in the selected group of the AED that did not detect the hot word group in the audio data to wake up from the low-power state and cooperate with all the AEDs in the selected group of the AED to perform the operation specified by the query. In some embodiments, in response to detecting the hot word group in the audio data, the operation further includes identifying each of the one or more other AEDs in the selected group of the AED assigned the hot word group by accessing a hot word registry. The hot word registry contains a list of one or more hot words, each hot word being assigned to one or more AEDs associated with the user (or user group), and the list of one or more hot words includes the hot word group. In these embodiments, invoking each other AED in the selected group of the AED is based on each of the one or more other AEDs assigned the hot word group by accessing the hot word registry. In some examples, the hot word registry is stored in at least one of the following: the first AED, at least one other AED associated with the user, or a server communicating with the first AED.

[0013] Optionally, receiving the assignment instruction to assign the hot word group to a selected group of AEDs may include receiving voice input from the user requesting a digital assistant to enable the hot word group and assign it to a selected group of AEDs, instructing a speech recognizer to perform speech recognition on the audio data to generate an automatic speech recognition (ASR) result for the voice input, and performing a query interpretation on the ASR result for the voice input to identify the hot word group, which specifies the hot word group to be activated and the name of each AED in the selected group to which the hot word group is to be assigned. In some embodiments, receiving the assignment instruction to assign the hot word group to a selected group of AEDs includes receiving a user input indication that instructs the user to interact with one or more objects displayed in a graphical user interface to instruct the digital assistant to enable the hot word group and select the group of AEDs to which the hot word group is to be assigned.

[0014] In some examples, the operation may further include receiving prior audio data at the data processing hardware, the prior audio data being spoken by the user and corresponding to a command from a digital assistant to perform a prolonged action on two or more AEDs associated with the user. The digital assistant is configured to automatically create the hotword set for subsequent queries related to the prolonged action. Here, receiving the assignment instruction to assign the hotword set to a selected group of AEDs includes: receiving the hotword set automatically created by the digital assistant, the selected group of AEDs to which the hotword set is assigned includes the two or more AEDs performing the prolonged action. In some embodiments, the operation further includes outputting synthesized speech corresponding to a response from the digital assistant for audible playback from the first AED, indicating that the prolonged action is being performed, and the automatically created hotword set for subsequent queries related to the prolonged action. In these embodiments, the digital assistant revoks the use of the automatically created hotword set at the end of the prolonged action. In some examples, each AED in the selected group of AEDs is associated with the same device type or shares common attributes, the hot word group assigned to the selected group of AEDs includes implicit hot words that uniquely identify the device type or common attribute associated with the selected group of AEDs, and the hot word detection model is pre-trained to detect the presence of hot word groups in streaming audio without performing speech recognition on the audio data.

[0015] In some implementations, the hot word set assigned to the selected group of AEDs includes proximity-based hot words that address only a subset of one or more AEDs currently closest to the user within the selected group of AEDs, without explicitly identifying any specific AED within the subset of one or more AEDs. The hot word detection model is pre-trained to detect the presence of the hot word set in streaming audio without performing speech recognition on the audio data, and executes the collaborative routine to cause the first AED and each other AED in the selected group of AEDs to determine a corresponding proximity value relative to the user, and performs arbitration within the selected group of AEDs using the corresponding proximity value to select the subset of one or more AEDs to perform the operation specified by the query. Optionally, the hot word set may include one of the following: custom hot words created by the user or predefined hot words selected from a list of one or more available hot word sets. During the execution of the collaborative routine, the first AED and each other AED in the selected group of AEDs collaborate with each other by specifying one of the AEDs in the selected group of AEDs to perform the following: generate speech recognition results for the audio data, perform query interpretation on the speech recognition results to determine the speech recognition results identifier of the query to be performed, and share the query interpretation performed on the speech recognition results with the other AEDs in the selected group of AEDs.

[0016] In some implementations, the query specifying the operation to be performed includes a query for performing a long-term operation on a selected group of AEDs, and during the execution of the cooperative routine, each AED in the selected group cooperates with each other in such a way that they pair up with each other during the duration of the long-term operation and coordinate the execution of sub-actions associated with the long-term operation. The operation specified by the query may include a device-level operation performed on each AED in the selected group of AEDs, and during the execution of the cooperative routine, each AED in the selected group of AEDs cooperates with each other by independently performing the device-level operation.

[0017] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will become apparent from the specification, drawings, and claims.

[0018] Attached Figure Description

[0019] Figure 1A - 1C is a schematic diagram of a system for enabling hot word groups and assigning hot word groups to selected groups of assistant-enabled devices.

[0020] Figure 2A and2B This is a schematic diagram of an example graphical user interface for enabling hotword groups and assigning hotword groups to selected groups of assistant-enabled devices.

[0021] Figure 3 This is a schematic diagram of an example of a leadership election process used to elect one or more assistants to enable devices to listen for the presence of hot word pairs in streaming audio.

[0022] Figure 4 This is an example of a voice-enabled environment, which includes multiple assistant-enabled devices located in different areas of the environment.

[0023] Figure 5 This is a sample hot word registry, which contains a list of hot words, each of which is assigned to... Figure 4 The assistant in the voice environment enables different selected groups of devices.

[0024] Figure 6 This is a flowchart of an example arrangement of operations for enabling hot word groups and assigning them to selected groups of assistant-enabled devices.

[0025] Figure 7 This is an example computing device.

[0026] Similar reference symbols in different diagrams indicate similar elements. Detailed Implementation

[0027] Ideally, when conversing with a digital assistant interface, users should be able to communicate as if they were talking to another person by enabling the device to make verbal requests towards the assistant running the interface. The digital assistant interface then feeds these verbal requests to an automatic speech recognizer to process and recognize them in order to perform an action. However, in practice, ensuring that the device consistently responds to these verbal requests is challenging, as continuously running speech recognition on resource-constrained voice-enabled devices such as smartphones or smartwatches is very expensive.

[0028] To create a user experience that supports always-on voice, assistant-enabled devices typically run a compact hotword detection model. This model is configured to identify audio features that characterize a limited set of phrases, and when the user speaks these features, fully automatic speech recognition (ASR) is initiated for any subsequent speech uttered by the user. Advantageously, the hotword detection model can run on low-power hardware, such as a digital signal processor (DSP) chip, and can respond to a variety of fixed phrase commands, such as "Hey Google" or "Hey living room speaker".

[0029] As the number of assistant-enabled devices in a user's environment (e.g., home or office) increases, users may want to trigger multiple assistant-enabled devices simultaneously, such as adjusting the volume level of a group of assistant-enabled smart speakers or adjusting the lighting level of a group of assistant-enabled smart lights. Currently, when a user wants to trigger multiple different assistant-enabled devices, they need to issue a different query to each device independently. For example, to turn off the kitchen and dining room lights in a user's home, the user must say different queries, such as "Hey, kitchen light bulb, turn off" and "Hey, dining room light bulb, turn off".

[0030] The implementation described herein aims to allow users to issue a single query to a group of Assistant Enabled Devices (AEDs) to enable faster and more natural interaction with multiple different AEDs that the user may wish to control simultaneously. Specifically, the implementation aims to create hotword pairs and assign them to groups of two or more AEDs selected by the user, so that when a hotword pair is detected in streaming audio, each device will respond to a voice query containing the hotword pair by triggering from a low-power state. That is, within the selected groups of AED pairs assigned hotwords, each AED can run a trained hotword detection model to detect the presence of the hotword pair in the streaming audio without performing speech recognition. In some implementations, hotword pairs are predefined for assigning to selected groups of AED pairs, thereby pre-training corresponding hotword detection models to detect the presence of predefined hotword pairs. Alternatively, users can also create custom hotword pairs containing any words or phrases that the user wishes to use to access a specific group of AED pairs in a single query. Here, it may be necessary for the user to provide one or more training utterances from the custom hotwords spoken by the user to train the corresponding hotword detection model to detect the custom hotwords.

[0031] In some examples, a user selects a group of AEDs using a digital assistant interface and manually enables hotword groups (e.g., predefined or custom) to simultaneously assign the selected group of AEDs that the user wishes to access in a single query. AEDs in the selected group can receive assignment instructions from the digital assistant interface to assign hotword groups to the selected group of AEDs, thus configuring each AED in the selected group to wake up from a low-power state when at least one AED in the selected group detects the hotword group in streaming audio. For example, a user could assign the hotword group “home room speakers” to a group of four smart speakers located in a home room, allowing the user to access all four smart speakers by speaking a phrase containing the hotword group “home room speakers” and then issuing a single query, such as “play that 12-6-97 Phish program,” to perform a specific action. In this example, at least one smart speaker in a group of four smart speakers that detects the hot phrase "home room speaker" in the user's utterance will trigger the corresponding smart speaker to wake up from a low-power state and execute a collaboration routine, causing each smart speaker in the group of four smart speakers to cooperate with each other to complete the operation specified by the query. For example, the four home room speakers could collaborate to play music corresponding to a concert performed by the band Phish on December 6, 1997. In this example, one speaker is assigned to stream music from local storage, network storage, or a remote streaming service, and then broadcast the music to the other speakers for audio playback from the speakers. Optionally, during the collaborative completion of the operation, some smart speakers can perform different playback duties related to the operation; for example, two smart speakers could play audio corresponding to the left channel, and the other two smart speakers could play audio corresponding to the right channel, thus providing a stereo arrangement.

[0032] Continuing with this same example, other AEDs not assigned to a hotword group—such as those corresponding to device types other than smart speakers, or smart speakers located in rooms outside the user's home—will not respond to the hotword group and will remain in sleep mode when the user says "home room speaker." Furthermore, each AED can be assigned a unique device-specific hotword, and only that AED is configured to be detected in streaming audio when the user only wants to access it. For example, the unique device-specific hotword assigned to an AED could include the AED's identifier, such as "Hey Device 1," or it might include the device type and / or other attributes associated with the AED, such as "Hey Smart Speaker 1." Additionally, a group of four selected smart speakers located in the home room, as well as any other AEDs associated with the user but not assigned a hotword group, can also be configured to respond to a global default hotword, such as "Hey Assistant." In some examples, at least one AED associated with the user may be assigned two or more hotword groups simultaneously, such that at least one AED will be a member of a different selected group of AEDs, each assigned one of two or more corresponding hotword groups. In these examples, the selected group for each AED can include combinations of AED pairs assigned to the corresponding hot word group, and the combinations of AEDs are different from combinations of AEDs assigned to different corresponding hot words.

[0033] In some implementations, the user manually enables hotword groups to assign them to selected groups of AEDs. For example, the user can access a digital assistant application that displays a graphical user interface (GUI) allowing the user to configure and adjust settings for all AEDs associated with the user. Here, the GUI can provide a hotword screen displaying various graphical objects (text fields, buttons, drop-down menus) for creating and enabling hotword groups and selecting the AEDs to which the user wishes to assign the hotword group. In this way, the selected group of AEDs can receive assignment instructions to assign the hotword group in response to receiving user input instructions that instruct the user to interact with one or more objects displayed in the GUI to instruct the digital assistant to enable the hotword group and each AED in the selected group, so that the AED group is assigned the hotword group. The user can update the selected group of AEDs via the GUI by selecting one or more additional AEDs to add to the group and / or selecting one or more AEDs to remove from the group.

[0034] Users can also select AED groups and enable hot word groups to be assigned to the selected AED group via voice input corresponding to a hot word group query. Here, the user can speak a voice input request to the digital assistant to enable hot word groups and assign them to the selected AED group. For example, the voice input spoken by the user to enable hot word groups for assignment to a first and second AED located in the downstairs area of ​​the user's home could include "Device 1 and Device 2, respond to downstairs device". Here, the term "Device 1" spoken by the user includes the corresponding device-specific hot word assigned to the first AED, and the term "Device 2" spoken by the user includes a different corresponding device-specific hot word assigned to the second AED, such that the first and second AEDs will respectively detect the corresponding device-specific hot words and wake up to process the audio data corresponding to the hot word group query "respond to downstairs device". Therefore, at least one of the first or second AEDs can instruct a speech recognizer (e.g., on-device ASR or server-side ASR) to perform speech recognition on the audio data to generate an ASR result for the voice input, and then interpret the ASR result to recognize the hot word group query. The hot word group query identified by the query interpretation performed on the ASR results specifies the name of the hot word group to be enabled (e.g., "downstairs devices"), and assigns the hot word group to each AED in the selected group. Similarly, users can directly invoke the digital assistant using a global hot word, similarly providing voice input corresponding to the hot word group query. For example, a user could say the hot word group query "Hey assistant, have device 1 and device 2 respond to the downstairs devices." In this example, any AED associated with the user can detect the predefined default hot word "Hey assistant," wake up, initiate speech recognition on the audio data to generate the ASR results, perform speech recognition to identify the hot word group, and assign the hot word group to each AED in the selected group.

[0035] Similar to the GUI example above, users can also update selected AED groups via subsequent voice input, specifying one or more additional AEDs to be added to the group and / or selecting one or more AEDs to be removed from the group. For example, a user can say "Hey downstairs devices, add device 3" to add the third AED 104c (device 3) to the AED group assigned the hotword group "downstairs devices". Similarly, a user can say "Hey device 1, leave downstairs devices group" to remove the first AED 104a (device 1) from the group, so that the first AED is no longer assigned a hotword group and will not be triggered when the user says "Hey downstairs devices". Users can provide voice confirmation to confirm (or undo) updates to the AED groups. Furthermore, once all devices leave the selected group, the hotword may no longer exist, requiring the user to recreate or re-enable the hotword group.

[0036] In other examples, hotword phrases can be used implicitly. For instance, a user could say the hotword phrase "Hey nearby devices" or "Hey nearby devices" to address only AEDs closest to the user. The hotword detector can detect both singular and plural hotword phrases simultaneously, or only the singular form, relying on speech recognition to identify the suffix "s". This type of implicit hotword phrase includes proximity-based hotword phrases. Users can access a digital assistant application and interact with the GUI to specify which AEDs should be assigned proximity-based implicit hotword phrases. Thus, the hotword phrases in this example provide context to specify that the user only wants to invoke one or more AEDs currently closest to the user, without requiring the user to explicitly identify which AEDs, whether by assigning them a corresponding unique hotword or naming the AED in the query portion of the statement. Notably, each AED assigned an implicit proximity-based hotword phrase can run a hotword detection model to detect the presence of the hotword phrase in the streaming audio to trigger the wake-up process and initiate speech recognition on the audio. Because the implicit hot phrases in the example are based on proximity, even if multiple AEDs can detect hot phrases in the captured streaming audio, these AEDs can subsequently process the audio to determine the corresponding proximity values ​​relative to the user. Arbitration can then be performed among the multiple AEDs using these proximity values ​​to select one or more of them to complete the action specified in the user's query. Here, AEDs located far from the user's highest distance threshold may not be eligible to complete the query. Optionally, AEDs within a lower distance threshold, such as a smartphone AED in the user's pocket, which detects the distance-based hot phrase "Hey, nearby devices(s)", may also not be eligible to respond to the query. The user can also choose to add / remove AEDs from a selected group assigned proximity-based hot phrases.

[0037] Alternatively, one or more AEDs selected to respond to a user query may be chosen based on the query type and / or the corresponding device attributes associated with each AED, so that only one or more of the most suitable AEDs are selected to complete the query. Here, the device attributes associated with each AED may include processing power, device type, user-configurable device settings, power consumption, battery level, AED's physical location, or network capabilities, etc. Therefore, when the query is a single-device query, such as "Hey nearby devices, set a timer," device arbitration may determine that the AED closest to the user is not qualified to complete the query because the AED is a battery-powered smart speaker and the battery capacity is very low (e.g., less than 5%). Therefore, the next nearest AED, assigned an implicit proximity-based hot phrase, can complete the query.

[0038] In some implementations, implicit hotword groups are assigned to AEDs in a selected group associated with the same device type. For example, an implicit device type hotword group might include "Hey Smart Speaker" to address all AEDs associated with the user that include the smart speaker device type. Similarly, another implicit device type hotword group might include "Hey Smart Light" to address all AEDs that include the smart light device type. Notably, device type hotword groups provide context by uniquely identifying the device type associated with a selected group of AEDs, indicating which AEDs the user wants to access. Each AED can run a hotword detection model, pre-trained to detect the presence of device type hotword groups in streaming audio without performing speech recognition on the audio data.

[0039] In other implementations, implicit hotword groups are assigned to AEDs in a selected group that share common attributes. For example, implicit attribute-based hotword groups could include “Hey Blue Devices” to address all blue-marked AEDs associated with the user, or “Hey Red Devices” to address all red-marked AEDs associated with the user. Attribute-based hotword groups can similarly specify any other attribute, such as size, for example, “Hey Big Devices” or “Hey Small Devices.” Notably, attribute-based hotword groups can further narrow down the AEDs a user wants to access to a specific group. In a non-limiting example, where the implicit device type hotword group “Hey Smart Speakers” would address all smart speakers in the user’s home, and the manually enabled hotword group “Hey Family Room Speakers” would address only the four smart speakers located in the family rooms of the user’s home, the implicit attribute-based hotword group “Hey Blue Devices” could be used to address two of the four red-marked smart speakers located in the user’s family rooms.

[0040] Implicit keyword groups can be enabled / disabled via the GUI of the digital assistant application. Similarly, AED groups assigned implicit keyword groups can be specified / selected via the GUI of the digital assistant application. The selected group of AEDs assigned implicit keyword groups can be updated by adding additional AEDs to the group and / or removing AEDs from the aforementioned group.

[0041] In other implementations, the digital assistant automatically creates hotword phrases and assigns them to a selected group of AEDs performing a long-term action while the long-term action is in progress. For example, a user can issue a voice query / command instructing the digital assistant to perform a long-term action on two or more AEDs. In a non-limiting example, the voice query / command “Hey assistant, play the party music playlist on speaker 1 and speaker 2” causes the digital assistant to perform a long-term action by making the user’s party music playlist an audible streaming media playlist on speakers 1 and 2. In this example, the digital assistant is configured to automatically create the action-specific hotword phrase “party music” for the user to use in subsequent queries related to the long-term action. Therefore, each of speaker 1 and speaker 2 receives the task instruction automatically created by the digital assistant to assign the hotword phrase “party music”. The user can then access the long-term action performed on speakers 1 and 2 by simply saying “party music”. For example, the user might say something like “party music, next song” or “party music, turn up the volume” to go to the next track in the playlist or instruct speakers 1 and 2 to each increase their volume. To notify the user of action-specific hotword phrases created by the digital assistant, the digital assistant can output synthesized speech corresponding to a response that can be played audibly in one of the AEDs (e.g., speaker 1 or speaker 2) to indicate the ongoing long-term action and the automatically created hotword phrase for subsequent queries related to the long-term action. In the example above, the response could include synthesized speech conveying "Understood, play it now." In the future, you could use the hotword "party music" to control playback. When the long-term action ends, the digital assistant can revert the use of the automatically created hotword phrase.

[0042] Figure 1A - 1C illustrates a system 100 for assigning a hot word group 50g to a selected group of two or more Assistant Activated Devices (AEDs) 104 associated with a user 102, allowing the user 102 to address the selected group of two or more AEDs in a single query by speaking the hot word group 50g. In short, as described in more detail below, Figure 1AUser 102 manually enables a hot word group by uttering the phrase "Hey assistant, get devices 1 and 2 to respond to the downstairs speakers," assigning two or more AEDs 104, 104a-c associated with user 102 to a selected group. In response to the phrase 106, a digital assistant 105 (and optionally a remote server 120 communicating with the AEDs) executing on AED 104 provides an assignment instruction that assigns the hot word group "downstairs speakers" to the selected group of AEDs 104, which includes a first AED 104a named "Device 1" and a second AED 104b named "Device 2." Each AED 104a, 104b assigned a hot word group is configured to wake from a low-power state when the hot word group is detected in streaming audio by at least one AED in the selected group of AEDs 104a, 104b. For example, when the user utters the following words 126, “Downstairs speaker, play my playlist,” the first AED 104a and the second AED 104b detect the hot phrase “downstairs speaker” in the audio data corresponding to the words 126. This triggers each AED 104a and 104b to wake up from a low-power state and execute a collaborative routine 150 to cooperate with each other to start playing music 122 from the user 102’s playlist (e.g., track #1).

[0043] In the example shown, system 100 includes three AEDs 104a-c associated with user 102, which act as a digital assistant 105 that user 102 can interact with via voice. While three AEDs 104 are described, user 102 may include any number of AEDs 104 located throughout the entire voice-enabled environment associated with user 102. Although all AEDs 104 correspond to smart speakers, AEDs 104 may include other computing devices that do not depart from the scope of this disclosure, such as, but not limited to, smartphones, tablets, smart displays, desktop / laptop computers, smartwatches, smart appliances, headphones, or vehicle infotainment devices. Each AED 104 includes data processing hardware 10 and memory hardware 12 storing instructions that, when executed on the data processing hardware 10, cause the data processing hardware 10 to perform operations. Each AED 104 includes an array of one or more microphones 16 configured to capture acoustic sounds, such as voice directed at the AED 104. Each AED 104 may also include an audio output device (e.g., a speaker) 18, or communicate with an audio output device (e.g., a speaker) 18, which can output audio, such as music 122 and / or synthesized speech 450, from the digital assistant 105. Figure 4 ).

[0044] Figure 1AThe image shows user 102 speaking uttering 106, “Hey assistant, get devices 1 and 2 responding to the downstairs speakers,” near at least the first AED 104a, to request digital assistant 105 to activate the hotword group “downstairs speakers” and assign the hotword group to a selected group of AEDs, including the first AED 104a named “Device 1” and the second AED 104b named “Device 2”. The microphone 16 of the first AED 104a receives utterance 106 and processes audio data 20 corresponding to utterance 106. Initial processing of the audio data 20 may include filtering the audio data 20 and converting the audio data 20 from an analog signal to a digital signal. While the first AED 104a processes the audio data 20, the first AED 104a may store the audio data 20 in a buffer of memory hardware 12 for further processing. If audio data 20 is present in the buffer, the first AED 104a may use a hotword detector 108 to detect whether the audio data 20 includes the predefined global hotword 50 “Hey assistant” assigned to each AED associated with user 102. Hot word detector 108 is configured to identify hot words contained in audio data 20 without performing speech recognition on the audio data 20. Hot word detector 108 may include an initial hot word detection phase that coarsely listens for the presence of hot word 50, and if detected, triggers a second hot word detection phase to confirm the presence of hot word 50. The initial hot word detection phase may be executed on a low-power digital signal processor (DSP) of data processing hardware 10, while the second hot word detection phase may run on a more computationally intensive application processor (AP) (e.g., a system-on-a-chip (SoC)) to provide more accurate hot word detection.

[0045] In some implementations, hot word detector 108 is configured to identify hot words located in the initial portion of utterance 106. In this example, if hot word detector 108 detects an acoustic feature in audio data 20 that serves as a feature of hot word 50, hot word detector 108 can determine that utterance 106 “Hey assistant, let devices 1 and 2 respond to the downstairs speaker” includes the predefined global hot word 50 “Hey assistant”. The acoustic feature may be Mel-frequency cepstral coefficients (MFCCs) representing the short-term power spectrum of utterance 106, or the Mel-scale filter bank energy of utterance 106. For example, hot word detector 108 may detect that utterance 106 “Hey assistant, devices 1 and 2 respond to the downstairs speaker” includes hot word 50 “Hey assistant” based on MFCCs generated from audio data 20 and MFCCs classified as including MFCCs similar to those that serve as features of the hot word “Hey assistant” stored in the hot word model of hot word detector 108. As another example, the hot word detector 108 can detect that the speech 106 “Hey assistant, device 1 and device 2 respond to the downstairs speaker” includes the hot word 50 “Hey assistant” based on the Mel scale filter bank energy generated from the audio data 402 and the Mel scale filter bank energy classified into Mel scale filter bank energies that include features similar to the Mel scale filter bank energy that is a feature of the hot word “Hey assistant” stored in the hot word model of the hot word detector 108.

[0046] When the hot word detector 108 determines that the audio data 20 corresponding to the utterance 106 includes predefined global hot words 50, the AED 104 can trigger a wake-up process to initiate speech recognition on the audio data 20 corresponding to the utterance 106. For example, a speech recognizer 116 running on the AED 104 can perform speech recognition and / or semantic interpretation on the audio data 20 corresponding to the utterance 106. The speech recognizer 116 can perform speech recognition on the audio data 20 to generate an automatic speech recognition (ASR) result for the utterance 106, and then perform query interpretation on the ASR result to identify a hot word group query 118 specifying the name of the hot word group to be enabled, and each AED 104 in the selected group of AEDs is assigned a hot word group. In this example, the speech recognizer 116 can query and interpret the ASR results, which include the phrase “make devices 1 and 2 respond to downstairs devices” as a hot word query 118. The hot word query 118 specifies the name of the hot word group “downstairs speaker” and assigns the hot word group to each AED 104a, 104b in the selected group of AED 104, “device 1 and device 2”.

[0047] In some implementations, the speech recognizer 116 is located on a server 120, other than or in place of the AED 104. When the hot word detector 108 triggers the AED 104a to wake up in response to detecting a predefined global hot word 50 in utterance 106, the AED 104a can transmit audio data 20 corresponding to utterance 106 to the server 120 via network 132. The AED 104a can transmit a portion of the audio data 20 including the hot word 50 to the server 120 to confirm the presence of the global hot word 50. Alternatively, the AED 104a can transmit only a portion of the audio data 20 corresponding to the portion of utterance 106 following the global hot word 50 to the server 120. The server 120 executes the speech recognizer 116 to perform speech recognition and returns a transcription of the audio data 20 to the AED 104a. The AED 104a sequentially recognizes the words in utterance 106, and performs semantic interpretation and identifies the hot word group query 118. AED 104a (and / or server 120) can identify hot phrase query 118 for digital assistant 105 to enable and provide assignment instructions, assigning the hot phrase "downstairs speaker" to a selected group of AED 104, which includes first AED 104a and second AED 104b. In the example shown, digital assistant 105 initiates a long-term action of playing music 122 from speaker 18 of AED 104. Digital assistant 105 can stream music 122 from a streaming service (not shown), or digital assistant 105 can instruct AED 104 to play music stored on AED 104.

[0048] After the hot word group 50g "downstairs device" is enabled and assigned to the first AED 104a, second AED 104b, the corresponding hot word detector 108 running on the first AED 104a, second AED 104b is configured to recognize the hot word group 50g "downstairs device" corresponding to the subsequent utterance 126 in the audio data 20. Here, each corresponding hot word detector 108 can activate a corresponding hot word group model 114 to run on the corresponding AED 104a, 104b, which is trained to detect the subsequent utterance 126 of the hot word group 50g "downstairs speaker" in the streaming audio captured by the corresponding AED 104a, 104b, without performing speech recognition on the captured audio. The hot word group model 114 can be stored on the memory hardware 12 of the AED 104 or on the server 120. If stored on server 120, AED 104 can request the server to retrieve a hot word group model 114 for the corresponding hot word group 50g and provide the retrieved hot word group model 114 so that AED 104 can activate the hot word group model 114. In some examples, hot word group 50g is predefined and available as a recommended hot word group; user 102 can enable the hot word group and assign it to a selected group of AED 104. In these examples, the corresponding hot word group model 114 is pre-trained to detect hot word group 50g in streaming audio. In other examples, hot word group 50g is a user-defined hot word group. In these other examples, user 102 can train the corresponding hot word group model 114 to detect the custom hot word group 50g by speaking the training utterance, which includes user 102 saying the custom hot word group 50g.

[0049] In another embodiment, hot word groups are assigned to a selected group of AED 104s such that one or more AED 104s execute speech recognizer 116 in a low-power and low-fidelity state, wherein speech recognizer 116 is constrained or biased to recognize only the hot word groups 50g assigned to AED 104 when spoken in subsequent utterances 126 captured by AED 104. Since speech recognizer 116 recognizes only a limited number of terms / phrases, the number of parameters of speech recognizer 116 can be significantly reduced, thereby reducing the memory requirements and computational load required to recognize hot word groups 50g in speech. Therefore, the low-power and low-fidelity characteristics of speech recognizer 116 are suitable for execution on a digital signal processor (DSP). In these embodiments, speech recognizer 116 executing on at least one AED 104 can recognize utterances 106 with enabled hot word groups 50g in streaming audio captured by at least one AED 104, instead of using hot word group model 114.

[0050] One or more AEDs 104 can locally store a hotword registry 500 on memory hardware 12. The hotword registry 500 contains a list of one or more hotwords 50, each hotword assigned to one or more AEDs associated with user 102. Digital assistants 105 and / or AEDs 104 in selected groups can populate the hotword registry 500 to include enabled hotword groups 50g in the list of one or more hotwords, and identify each AED 104 in the selected group to which hotword group 50g is assigned. After enabling and assigning the hotword group 50g "speaker downstairs" to a first AED 104a named Device 1 and a second AED 104b named Device 2, Figure 1A The digital assistant 105 updated the hot word registry 500 to specify that the hot word group 50g “downstairs speaker” should be assigned to device 1 and device 2.

[0051] In some examples, the first AED 104a and the second AED 104b in the selected group of AEDs receive an assignment instruction to assign a hot word group 50g. The first AED 104a and the second AED 104b perform a leader election process 300, electing one or more AEDs from the selected group to listen for the presence of the hot word group 50g in the streaming audio, based on the corresponding device attributes 302 associated with each AED 104. Figure 3 An example leader election process 300 is shown, configured to receive as input corresponding device attributes 302 associated with each AED 104 in a selected group of AEDs, and generate as output an election instruction 310 to elect one or more AEDs to listen for the presence of hot word group 50g. The device attributes 302 associated with each AED 104 may include, but are not limited to, at least one of the following: processing power, device type, user-configurable device settings, power consumption, battery level, AED's physical location, or network capabilities. Figure 1AIn the example, device attribute 302 associated with the second AED 104b, named Device 2, could indicate that the second AED 104b is a portable device and is currently battery-powered, while device attribute 302 associated with the first AED 104a, named Device 1, could indicate that the first AED 102a is a stationary device powered by an external power source (e.g., a power outlet). Therefore, the election instruction 310 output by the leadership election process 300 could instruct that the more recent of Device 1 or Device 2 relative to the user 102 who said “downstairs device” should respond by performing speech recognition and semantic interpretation to identify the query, unless Device 2’s battery level is less than five percent (5%). In other words, when each of AEDs 104a and 104b detects a subsequent utterance 126 containing the hot phrase "downstairs device" in the streaming audio, the collaborative routine 150 executed by AEDs 104a and 104b will cause a second AED 104b, named device 2, to remain unresponsive when its battery level is below 5%, even if device 2 is closest to user 102. This allows device 2 to save power by not consuming processing resources to perform speech recognition and / or semantic interpretation of the audio data. Otherwise, without considering energy saving, the leadership election process 300 would typically choose the nearest AED 104 to process the audio, as the audio captured by AED 104 is likely to be of higher quality than that captured by a more distant AED 104, and thus provide more accurate speech recognition.

[0052] In another example, the leadership election process 300 can generate more granular election instructions 310. For example, election instruction 310 can select only one of the AEDs 104 to trigger a second-stage hot word detection (i.e., using a computationally intensive hot word detection model 114 or a speech recognizer 116) to determine the presence of hot word group 50g when the first-stage hot word detector 108 initially detects it. That is, election instruction 310 can notify each AED 104 in a selected group of AEDs assigned hot word group 50g that when each AED 104 detects the presence of hot word group 50g using the first-stage hot word detector 108, only the elected AED 104 will trigger the second-stage hot word detection to confirm the presence of hot word group 50g. Further extending this, device attribute 302 can indicate that one of the AEDs is battery-powered and configured to run a first-stage hot word detector 108 on a DSP chip. This chip consumes low power at the expense of low fidelity to coarsely listen for hot word group 50g. Once the first-stage hot word detector 108 detects hot word group 50g, it triggers an application processor (e.g., a SoC chip) to wake up and run a second-stage hot word detection (e.g., hot word model 114 or on-device ASR 116) to confirm the presence of hot word group 50g. Therefore, if device attribute 302 indicates that one or more other AEDs in a selected group of AEDs are non-battery-powered devices, the task of utilizing these devices for at least the second-stage hot word detection can be effective, thus avoiding wasting power by triggering the AP to wake up from a low-power state.

[0053] Another possible scenario is that the device attribute 302 of the AEDs in the selected group indicates that the AED is capable of performing speech recognition on the device for a limited set of common queries / commands, while other AEDs in the selected group need to provide audio to the server 120 to perform server-side ASR. The leader election process 300 can generate an election instruction 310 that causes the collaboration routine 150 to elect an AED capable of performing speech recognition on the device, first attempting to perform speech recognition on the captured audio data 20 to confirm whether one of the limited set of common queries / commands has been recognized in the captured audio data 20. If one of the common queries / commands is not recognized, the generated election instruction 310 can allow the collaboration routine 150 to elect one of the other AEDs to provide audio data 20 to the server 120 to perform server-side ASR on the audio data 20.

[0054] Continue to refer to Figure 3The AEDs 104 in the selected group of AEDs 104 can periodically and / or re-execute the leader election process 300 in response to specific events. In one example, the re-execution of the leader election process 300 occurs in response to a device state change 304 of one of the AEDs in the selected group of AEDs 104. The device state change 304 may include, but is not limited to, an increase in the load on the processing AED 104 to a level that violates a processing threshold, a decrease in the load on the processing AED 104 to a level that no longer violates a processing level, a change in background noise level, a decrease in battery capacity below a battery capacity threshold, a loss of network connectivity, or a power-off of the AED 104. The device state change 304 allows the leader election process 300 to re-evaluate the corresponding device attributes 302 associated with each AED 104 in the selected group to elect one or more AEDs that are currently best suited for the listening hot word group.

[0055] In one example, in response to an update 306 to the selected group of AED 104 due to the addition of one or more additional AED 104s to the selected group, the leadership election process 300 is re-executed. For example, Figure 1B The user 102 speaks another phrase 136, “Downstairs speaker, add device 3,” near at least the first AED 104a, to request the digital assistant 105 to assign the hot word group 50g, “downstairs speaker,” to a third AED 104c named “device 3,” in addition to the first AED 104a and the second AED 104b, which are named device 1 and device 2. Here, phrase 136 includes at least the first AED 104a (i.e., based on election instruction 310) detecting the hot word group 50g, “downstairs speaker,” in the audio data 20 corresponding to phrase 136 using a hot word detection model 114 corresponding to the hot word group 50g, to trigger the first AED 104a to wake up from a low-power state. Once activated, the first AED 104a instructs the speech recognizer 116 to perform speech recognition on the audio data 20 to generate an ASR result of the dialogue 136, and to perform query interpretation on the ASR result to identify a hot word query 118 specifying the device identifier "Device 3". The hot word query 118 is used to add an additional AED 102c to the selected group of AED 104, which has been assigned the hot word 50g "downstairs device". Therefore, the third AED 104c can receive an assignment instruction to assign the hot word 50g "downstairs speaker" to the selected group of the AED that has been updated to now include the third AED 104c. The third AED 104c can activate the aforementioned reference... Figure 1AThe hot word detection model 114 corresponds to the hot word group 50c. The digital assistant 105 can update the hot word registry 500 to add a third AED 104c named Device 3 to the selected group of AED 104 assigned the hot word group 50g "speaker downstairs". This can be re-executed. Figure 3 The leadership election process 300 takes into account the corresponding device attribute 302, which is associated with the third AED 104c. The AED 104c responds to the update 306 of adding the third AED 104c to the selected group of AED 104. All three AEDs 104a-c can cooperate with each other to perform the long-term action of streaming music 122 from the user's playlist.

[0056] Alternatively or additionally, in response to an update 306 to a selected group of AEDs 104, which removes one or more AEDs 104 from the selected group, the leadership election process 300 may be re-executed. For example, Figure 1C The user 102 speaks another phrase 146, “Device 1, leave the downstairs speaker group,” near at least the first AED 104a, to request the digital assistant 105 to remove the first AED 104a named “Device 1” from the selected group of AEDs 104a assigned the hotword group 50g, “downstairs speaker.” Here, phrase 146 includes the device-specific hotword 50d, “Device 1,” which is uniquely assigned to the first AED 104a and detected by the first AED 104a in the audio data 20 corresponding to phrase 146, triggering the first AED 104a to wake up from a low-power state and process the audio data 20 to identify the hotword group query 118, which requests the digital assistant 105 to remove the first AED 104a from the selected group of AEDs 104a assigned the hotword group 50g, “downstairs speaker.” Therefore, the first AED 104a can deactivate the hot word detection model 114 corresponding to the hot word group 50g, so that the first AED 104a no longer listens for the presence of the hot word group 50g in the audio data. The digital assistant 105 can update the hot word registry 500 to remove the first AED 104a named Device 1 from the selected group of AED 104s assigned the hot word group 50g "the speaker downstairs". Figure 3 The leadership election process 300 can be re-executed to determine updated election instructions 310 based on the fact that device 1 is no longer a member of the selected group of AEDs. Now, the second and third AEDs 104b, 104c can cooperate with each other without the first AED 104a to complete... Figure 1A The execution of the long-term action specified by query 128 in the statement 126 spoken by user 102.

[0057] Reference Figure 1A At least the first AED 104a uses the corresponding hot phrase model 114 to detect the presence of the hot phrase 50g "downstairs device" in the audio data 20, which corresponds to the subsequent utterance 126 spoken by the user 102, including a query 128 specifying the operation to be performed. Specifically, the example shows the user 102 saying the subsequent utterance 126 "downstairs speaker, play my playlist" and at least the first AED 104a uses the hot phrase model 114 to detect the hot phrase 50g "downstairs device" in the corresponding audio data 20. Detecting the hot phrase 50g in the audio data 20 triggers the first AED 104a (and optionally the second AED 104b) to wake up from a low-power state and execute a collaboration routine 150, enabling the first AED 104a to collaborate with each other AED 104 assigned the hot phrase 50g to perform the operation specified by the query 128. Here, query 128 specifies a long-term action, and the first and second AEDs 104a and 104b cooperate by pairing up with each other and coordinating the execution of sub-actions associated with the long-term action to play music 122 from the user's playlist during the long-term action. For example, one AED 104 may perform a sub-action connecting to a remote music streaming service to stream the playlist and broadcast the streamed playlist to another AED 104. In some examples, the cooperating AEDs 104 may assume different music playback responsibilities, such as one AED assuming the role of the left audio channel while the other AED assuming the role of the right audio channel to provide a stereo arrangement. Figure 1A The diagram shows a first AED 104a named Device 1 and a second AED 104b named Device 2 executing a collaborative routine 150 to cooperate with each other to complete a long-term action of playing music 122 (e.g., track #1) from a user's playlist.

[0058] In some examples, in response to the first AED 104a detecting a hot word group 50g in the audio data 20, the first AED 104a invokes each of the other AEDs 104s in the selected group of AEDs 104s that did not detect the hot word group 50g to wake up from a low-power state and cooperate with the first AED 104a to perform the operation specified by query 128. In these examples, in response to detecting the hot word group 50g, the first AED 104a can identify each of one or more other AEDs 104s in the selected group to which the hot word group was assigned by accessing the hot word registry 500. Here, the hot word registry 500, which contains a list of one or more hot words, includes the hot word group 50g “downstairs stairs” assigned to the first AED 104a named Device 1 and the second AED 104b named Device 2. Therefore, the first AED 104a can recognize that the second AED 104b, named device 2, has also been assigned the hot word group 50g, thereby invoking the second AED 104b to cooperate with the first AED 104a to complete the execution of the operation specified by query 128 (e.g., streaming music 122 from the user's playlist 102).

[0059] While query 128 in the illustrated example specifies a long-term action to be performed, other examples may include a query specifying a device-level operation to be performed individually for each AED in the selected group of AEDs. That is, in the execution of the cooperative routine 150, each AED in the selected group of AEDs cooperates by independently performing a device-level operation. For example, if the first AED 104a and the second AED 104b corresponding to smart bulbs are assigned the same hotword group 50g, a query specifying a device-level power-off operation would cause each smart bulb to independently perform a power-off operation.

[0060] refer to Figure 2A In some implementations, a software application 205 associated with the digital assistant 105 executes on a user device to display a user-defined hotword selection screen 200, 200a in the user device's graphical user interface (GUI) 208. In the illustrated example, the user device includes an interface with a smartphone (such as...). Figure 4 The smartphone 104j corresponds to the AED 104. The user-defined hot word group selection screen 200a allows the user to enable a hot word group 50g and assign it to a group consisting of two or more AEDs 104 selected by the user. The user 102 can use the hot word group selection screen 200a, in addition to or instead of providing the above reference. Figure 1A-1CVoice input is used to enable and assign hot words. In the example shown, the hot word selection screen 200a displays multiple objects 210, 210a-d in the GUI 208, which the user can interact with to instruct the digital assistant 105 to enable hot word 50g and select the AED 104 group to which hot word 50g has been assigned.

[0061] GUI 208 can receive user input instructions, directing the user to interact with text field object 210a, which allows the user to create a custom hotword group by typing the name of the custom hotword group they want to create. Optionally, user 102 can select a voice input graph (e.g., a graphic microphone) to provide voice input corresponding to the custom hotword group spoken by the user. When creating a custom hotword group, hotword group selection screen 200a can prompt the user to speak multiple training examples including the custom hotword group for training hotword group detection model 114 to detect custom hotword groups in streaming audio.

[0062] On the other hand, user 102 can enable predefined hotword groups by providing a user input instruction that instructs the user to interact with dropdown object 210b, which displays a list of available predefined hotword groups. Dropdown object 210b can present commonly used hotword groups as available predefined hotword groups for selection, such as hotword groups describing device types, like "smart speakers" and "smart lights," hotword groups describing public areas / regions in the environment, like "home room devices," and hotword groups describing both device type and area / region, like "bedroom speakers." The user can interact with dropdown object 210b to scroll through the list of available hotword groups. In some examples, custom hotword groups can be added to the list of available hotword groups. In the example shown, GUI 208 receives a user input instruction instructing the user to interact with dropdown object 210b to select the predefined hotword group "home room devices" from the list of available predefined hotword groups. Here, the selection of a predefined hotword group can instruct digital assistant 105 to enable the predefined hotword group. The assistant can also suggest enabling / activating hot word groups to be assigned to AED 104 groups. For example, users may prefer to manually search for a group of devices that all belong to the semantic group (e.g., ordered or sorted by their respective names).

[0063] In addition, the hotword selection screen 200a displays multiple selection objects 210c, each corresponding to one of a plurality of AEDs 104 associated with the user 102. The user 102 can provide user input (e.g., touch) to select each AED 104 to include in an AED group, assigning a hotword group 50g created via text field object 210a or selected from drop-down object 210b to the AED 104 group. In the illustrated example, GUI 208 receives user input instructions instructing the user to interact with the selection objects 210c corresponding to the AEDs 104 named Speaker 1, Speaker 2, Speaker 3, Speaker 4, and Smart TV to include these AEDs in the selected group of AEDs and assign them the hotword group “Home Room Devices”. To instruct the digital assistant 105 to enable the hotword group “Home Room Devices” and assign it to the selected group of AEDs 104 including Speakers 1-4 and Smart TV, the user 102 can provide user instructions instructing the user to interact with the enabled object 210d. Assuming target 210d is selected, digital assistant 105 will provide assignment instructions to the selected group of the AED (including speakers 1-4 and the smart TV), instructing the hot word group "Home Room Speakers" to be assigned to the selected group of the AED. For example... Figure 5 As shown, the digital assistant can also add hot word groups and selected groups of AEDs to the hot word registry 500.

[0064] refer to Figure 2BIn some implementations, the software application 205 associated with the digital assistant 105 is configured to display implicit hotword selection screens 200, 200b in the GUI 208 of the AED 104. The implicit hotword selection screen 200b displays multiple available implicit hotwords and allows the user 102 to select each implicit hotword from the selected group to be assigned to the AED. For each implicit hotword, the implicit hotword selection screen 200b may list all eligible AEDs that can be assigned the implicit hotword based on attributes associated with the AED. For example, all AEDs 104 associated with the user 102 are listed as eligible AEDs and assigned the proximity-based hotwords “Hey, nearby devices” and / or “Hey, nearby devices”. Therefore, in a single query, user 102 can access one or more AEDs closest to user 102 at any given time by simply saying the proximity-based hot phrase "Hey, nearby devices" or "Hey, nearby devices." This allows the AEDs that detect the spoken hot phrase to collaborate by performing arbitration to select the one or more devices closest to user 102 to complete the operation specified by the query. Advantageously, the proximity-based hot phrase allows user 102 to address only a subset of the one or more AEDs currently closest to user 102, without requiring the user to explicitly identify any specific AED within that subset.

[0065] In the example shown, by default, the proximity-based hotword phrase "Hey, nearby devices" is assigned to all AEDs associated with the user. The user can interact with a selection object to remove any AED from the selected group of AEDs assigned the proximity-based hotword phrase. For example, GUI 208 can receive user input instructing the user to interact with selection object 210c corresponding to AED 104 named "Smartphone" to remove the smartphone from the group assigned the proximity-based hotword phrase. Therefore, even if the smartphone is closest to the user 102, it will not detect or respond to the user saying "Hey, nearby devices."

[0066] Other implicit hotword groups include device type hotword groups, which can be assigned to selected groups of all AEDs sharing the same device type. In the example shown, the implicit hotword group selection screen 200b lists only Speakers 1-7 as eligible AEDs, and these AEDs will be assigned the implicit device type hotword group "Hey Smart Speaker" because AEDs named Speakers 1-7 all contain smart speakers of the same device type. Therefore, user 102 can interact with the selection object 210c displayed in GUI 208 to select (or deselect) the AED group to be assigned the hotword group "Hey Smart Speaker" and then speak a phrase containing the hotword group "Hey Smart Speaker" to access all AEDs with the smart speaker device type associated with user 102 in a single query.

[0067] The implicit hotword selection screen 200b also displays two different implicit attribute-based hotwords, each of which can be assigned to a selected group of corresponding AEDs sharing a common attribute. For example, the first attribute-based hotword group includes "blue speaker," which user 102 can assign to speaker 1 and speaker 2 to allow the user to access all smart speakers sharing the attribute of blue (or otherwise marked "blue") in a single query. Similarly, the second attribute-based hotword group includes "red speaker," which the user can assign to speaker 3 and speaker 4 to allow the user to access all smart speakers sharing the attribute of red (or otherwise marked "red") in a single query. (See below...) Figure 4 It is clear that attribute-based hot word groups can further narrow down the specific groups of AEDs that users want to access.

[0068] Figure 4 An example voice-enabled environment 400 is shown, including multiple AEDs 104 associated with user 102. In the example shown, the voice-enabled environment 400 is user 102's home, with multiple rooms and areas, including a family room, a kitchen, and a bedroom. While the voice-enabled environment 400 is depicted... Figure 4 The home is an example, but the voice-enabled environment 400 can include any environment that enables multiple AED networks, such as educational environments, commercial or automotive environments.

[0069] AED 104 includes seven smart speakers 104a-g (SP 1-7), one smart display 104h, one smart TV 104i, and one smartphone 104j, located throughout the voice-enabled environment. Smart speakers SP1 104a, SP2 104b, SP3 104c, SP4 104d and smart TV 104i are located in a home room within the voice-enabled environment 400. Additionally, smartphone 104j is portable and can be moved between rooms / areas of the voice-enabled environment 400. Smart speakers SP1-SP4 and smart TV 104i can be connected or otherwise paired together to form a corresponding partition called a "home room". Furthermore, the first smart speaker SP1 and the second smart speaker SP2 can be labeled "blue" devices to describe their blue physical attributes, and the third smart speaker SP3 and the fourth smart speaker SP4 can be labeled "red" devices to describe their red physical attributes. Other attributes can be used, such as size (e.g., large vs. small), type / brand (e.g., high-fidelity speaker), or any other tags that users use to identify / group AEDs within one or more areas in the voice-enabled environment 400. The voice-enabled environment 400 also describes smart speakers SP5 104e and smart displays 104h located in the kitchen, and which are paired / combined to form a corresponding area named "Kitchen". Similarly, smart speakers SP6 104f and SP7 104g can be combined / paired together to form a corresponding area named "Bedroom".

[0070] refer to Figure 4 The environment that supports voice is described in section 400. Figure 5 An example hotword registry 500 is shown, comprising a list of hotwords 50, each hotword 50 assigned to a corresponding selected group of AEDs 104 located in a voice-enabled environment 400. One or more AEDs 104 may each store the hotword registry 500 on their respective local memory hardware 12. An AED 104 that does not store a hotword registry 500 can discover other AEDs 104 in the network and access their hotword registry 500 to determine which hotwords are assigned to which AEDs. Alternatively or additionally, the hotword registry 500 may be stored on a centralized device and communicate with one or more AEDs. For example, the hotword registry 500 may be stored on a remote server, such as a remote server associated with a digital assistant 105, which associates the hotword registry with a user 102's profile.

[0071] Each AED 104 is assigned a default hotword 50, "Hey Assistant." When this hotword is detected in streaming audio by one or more AEDs, it triggers the AED 104 to wake up from a low-power state and invoke the first digital assistant 105 to initiate processing of one or more other terms following the default hotword 50. Here, the first digital assistant may be associated with a first voice assistant service (e.g., Google Assistant). Additionally, the smart speaker SP2 104b, smart display 104h, and smartphone 104j are also assigned another default hotword, "Other Assistant." When any of the AEDs 104b, 104h, and 104j detects "Other Assistant" in streaming audio, this hotword triggers the AED to invoke a second digital assistant to initiate processing of one or more other terms following the other default hotword. Here, the second digital assistant may be associated with a second voice assistant service (e.g., Amazon's Alexa or Apple's Siri), rather than with the first voice assistant service. Furthermore, each AED 104 can be assigned a unique device-specific hotword, and only the corresponding AED is configured to detect the hotword in the streaming audio when the user only wants to access that specific AED. For example, the unique device-specific hotword assigned to the first smart speaker SP1 104a in the environment 400 may include the AED's identifier, such as "Hey Device 1" or simply "Device 1", or may include the device type and / or other attributes associated with the AED, such as "Hey Smart Speaker 1" or simply "Smart Speaker 1".

[0072] As previously stated, the hot word group assigned to the selected group of the corresponding AED may include manually enabled hot words 50 assigned by user 102 to the selected group of the corresponding AED 104. Manually enabled hot words may be custom hot words created by user 102 and / or predefined hot words selectable by user 102. Predefined hot words may be associated with a predefined hot word model, which is trained to detect associated hot words. However, custom hot words created by user 102 may require the user to train a custom hot word detection model to detect the custom hot words. For example, user 102 may utter one or more utterances containing custom hot words.

[0073] In some examples, user 102 provides voice input (e.g., speech) 106 ( Figure 1A This allows the user to select each AED they wish to include in a chosen group of AEDs, and assigns a manually enabled hot word group 50g to the chosen group of AED 104. Similarly, the user can provide subsequent voice input 136, 146 (…). Figure 1B and 1C To add one or more additional AEDs to a selected group of existing AEDs. Figure 1B) and / or remove one or more AEDs from a selected group of existing AEDs. Figure 1C This updates the selected group of AED 104. Alternatively, the user can provide user input instructions that instruct the user to interact with one or more objects displayed in GUI 208 (e.g., ...). Figure 2A The user-defined hotword selection screen 200a instructs the digital assistant 105 to enable manual hotword groups and select the AED group to which the hotword group is to be assigned. The user 102 can provide subsequent user interaction instructions to the GUI 208 to update the selected group of AED 104 by adding additional AEDs from the existing selected group and / or deleting AEDs.

[0074] In an example hotword registry 500 for a voice-enabled environment 400, user 102 enables and assigns the manual hotword group “Home Room Devices” to a selected group of corresponding AEDs, which includes smart speakers SP1-SP4 104a-d and a smart TV 104i located in an area named “Home Room”. User 102 also enables and assigns the manual hotword group “Kitchen Devices” to smart speakers SP5 104e and a smart display 104h located in an area named “Kitchen”. Similarly, the manual hotword group “Bedroom Speakers” is enabled and assigned by the user to smart speakers SP6, SP7 104f-g located in an area named “Bedroom”. Here, each manually enabled hotword group can describe a location / area within the voice-enabled environment 400 (e.g., the user’s home), where the selected group of the corresponding AED assigned the corresponding hotword group 50 is located in the location / area. Notably, the manually enabled hotword group “Bedroom Speakers” assigned to smart speakers SP6, SP7 describes the device type (e.g., smart speaker) associated with the selected group of the corresponding AED.

[0075] In the example shown, user 102 did not assign any manually enabled hot word groups to smartphone 104j. However, when the corresponding hot word group is detected before the query in the streaming audio, one or more of the selected groups of AEDs can be updated to add / include smartphone 104j, enabling smartphone 104j to collaborate with other AEDs in the corresponding group to perform the operation specified by the query.

[0076] Figure 5 The example hot word registry 500 also shows that multiple different implicit hot word groups 50g are assigned to corresponding selected groups of different AED 104s. See above, reference... Figure 2BThe software application 205 associated with the digital assistant 105 can present a subtle hotword selection screen 200b in the GUI 208, and the user 102 can interact with the screen 200b to view the available subtle hotwords and select the AED group to be assigned to the subtle hotword group. For example, the proximity-based hotword groups "Hey nearby devices" and / or "Hey nearby devices" can be assigned to devices that are close to the user. Figure 4 All AEDs 104a-j associated with user 102 located in the voice-enabled environment 400. Therefore, user 102 can, at any given time, address one or more AEDs closest to user 102 in the voice-enabled environment 400 in a query by simply saying the proximity-based hot phrase "Hey nearby devices" or "Hey nearby devices," such that the AEDs detecting the spoken hot phrase will cooperate by performing arbitration to select one or more devices closest to user 102 to perform the operation specified by the query. Advantageously, the proximity-based hot phrase allows user 102 to access only a subset of the one or more AEDs currently closest to user 102, without requiring the user to explicitly identify any specific AED within the one or more subsets of AEDs.

[0077] Each AED 104, assigned an implicit proximity-based hot word group, can run a hot word detection model to detect the presence of the hot word group in the streaming audio, triggering a wake-up process and initiating speech recognition on the audio. Since the implicit hot word group in this example is proximity-based, even if multiple AEDs 104 can detect the hot word group in the captured streaming audio, these AEDs 104 can each subsequently process the audio to determine a corresponding proximity value relative to the user 102, and then use these proximity values ​​to perform arbitration among the multiple AEDs 104 to select one or more of these AEDs 104 to complete the action specified by the user's query. Here, AEDs 104 exceeding a maximum distance threshold from the user may not be eligible to complete the query. Alternatively, AEDs 104 within a lower distance threshold, such as a smartphone AED in the user's pocket detecting the distance-based hot word group "Hey, nearby devices(s)," may also not be eligible to respond to the query. Lower distance thresholds can be applied depending on the type of query. For example, if the query is a search query where nearby devices provide the search results as synthesized speech, then the fact that smartphone 104j is so close to user 102 that it's in the user's pocket will disqualify smartphone 104j from completing the query because the synthesized speech will be muted and user 102 won't understand / hear it. The user can also choose to add / remove AEDs from a selected group of hot words assigned based on proximity. Furthermore, the selected device closest to user 102 can perform speech recognition and query interpretation to determine whether "nearby devices" is spoken by user 102 to indicate that user 102 only wants the single closest device to complete the query, or whether "nearby devices" is spoken to indicate that the user wants two or more nearby devices to complete the query.

[0078] also, Figure 5The example hotword registry 500 also displays two distinct implicit device type hotword groups, each assigned to a corresponding selected group of AEDs 104 associated with the same corresponding device type in the voice-enabled environment 400. For example, the first device type hotword group includes “Smart Speaker” assigned to all smart speakers SP1-SP7 in the voice-enabled environment 400, allowing the user to address all AEDs 104a-h associated with the user 102, including smart speakers, in a single query. Here, the device type hotword group “Smart Speaker” addresses four smart speakers SP1-SP4 located in the area named “Family Activity Room,” smart speaker SP5 located in the area named “Kitchen,” and smart speakers SP6 and SP7 located in the area named “Bedroom.” Notably, the manually enabled hotword group “Family Room Devices” is also assigned to smart speakers SP1-SP4, the manually enabled hotword group “Kitchen Devices” is also assigned to smart speaker SP5, and the manually enabled hotword group “Bedroom Speakers” is also assigned to smart speakers SP5 and SP7.

[0079] In addition, the second device type hot word group includes "smart display," which is assigned to selected groups of the corresponding AEDs. These groups include the smart display 104h located in the area named "kitchen" and the smart TV 104i located in the area named "family activity room." Notably, the manually activated hot word group "family room devices" is also assigned to the smart TV 104i, and the manually activated hot word group "kitchen devices" is also assigned to the smart display 104h.

[0080] Continue to refer to Figure 4 Support for voice environments 400 and Figure 5Example hotword registry 500 assigns two distinct implicit attribute-based hotwords to corresponding selected groups of AEDs 104 that share common attributes in a voice-enabled environment 400. For example, the first attribute-based hotword group includes “blue speaker”, which is assigned to the first and second smart speakers SP1, SP2 located in an area named “family room” in environment 400, allowing the user to access all smart speakers 104a-b that share the attribute of having the color blue (or being labeled “blue”) in a single query. Similarly, the second attribute-based hotword group includes “red speaker”, which is assigned to all smart speakers 104c-d that share the attribute of having the color red (or being labeled “red”) in a single query. It is worth noting that the first and second smart speakers SP1 and SP2, which were assigned the hot word group "blue speaker," and the third and fourth smart speakers SP3 and SP4, which were assigned the hot word group "red speaker," are also in the selected group of seven (7) smart speakers 104a-g assigned the device type hot word group "smart speaker," and the selected group of five (5) AEDs 104a-d and 104i assigned the manually enabled hot word group "home room device." The hot word group "home room device" includes smart speakers SP1-4 and smart TV 104i. Therefore, attribute-based hot word groups can further narrow down the specific groups of AEDs that users want to access.

[0081] refer to Figure 4In one example, a user 102, located in a region called the family room within a voice-enabled environment 400, utters the utterance 406 “Speaker 1 & Smart TV” corresponding to command 418. The command is then executed by a digital assistant 105 to perform a long-term action of streaming music video to play on a first smart speaker SP1 104a and a smart TV 104i. The digital assistant 105 can execute on all AEDs 104 within the voice-enabled environment 400. The terms “Speaker 1” and “Smart TV” serve as prefixes to command 418 and correspond to the respective device-specific hotwords 50a and 50b assigned to the first smart speaker SP1 104a and “Smart TV”, respectively. Here, the first smart speaker SP1 104a executes a hotword detection model trained to detect the hotword 50a “Speaker 1” corresponding to utterance 14 in the audio data, triggering SP1 104a to wake from a low-power state and initiate processing on the audio data. Simultaneously, the smart TV 104i executes a hot word detection model, which is trained to detect the hot word 50b "smart TV" corresponding to utterance 14 in the audio. This triggers the smart TV 104i to wake up from a low-power state and initiate audio data processing. After performing speech recognition processing on the audio data, an ASR result is generated, and the ASR result is interpreted to recognize command 418. After the first smart speaker SP1 104a and the smart TV 104i perform a long-term action, SP1 and the smart TV cooperate to complete the long-term action. For example, the smart TV 104i can stream video data to display the video portion of a music video, while SP1 can stream audio data to audibly output the audio portion of the music video.

[0082] Continuing this example, the digital assistant 105 is also configured to automatically create hotword sets for specific operations and assign these hotword sets to a selected group of AEDs, including the first smart speaker SP1 104a and the smart TV 104i, which perform the long-term action when it is in progress. In subsequent queries related to the long-term action of playing music videos on the first smart speaker SP1 104a and the smart TV 104i, the user 102 can use the hotword set for the specific operation. Therefore, the AEDs corresponding to the first smart speaker SP1 104a and the smart TV 104i respectively receive assignment instructions assigning the hotword set "music video" automatically created by the digital assistant 105. Subsequently, the user 102 can access the long-term action performed on the first smart speaker SP1 104a and the smart TV 104i by simply saying the phrase "music video," followed by a query / command to control that long-term action. For example, user 102 can say "Music video, next song" or "Music video, turn up the volume" to enter the next music video or instruct the first smart speaker SP1 to increase the volume. In response to creating hot phrases for specific actions and providing assignment instructions to the first smart speaker SP1 104a and the smart TV 104i, the digital assistant 105 can update... Figure 5 The hot word registry 500 indicates that the specific operation hot word group "music video" is assigned to a selected group of AEDs, which includes the first smart speaker SP1 104a and the smart TV 104i.

[0083] In some examples, for sound playback, the first smart speaker SP1 104a outputs synthesized speech corresponding to the response of the digital assistant 105 to indicate that a long-term action is in progress and automatically creates hot words for specific actions for subsequent queries related to the long-term action. For example, Figure 4 The synthesized speech output by SP1 104a is displayed, corresponding to the response 450 of digital assistant 105. The speech includes "Okay, now play the music video... In the future, you can use the hotword 'music video' to control playback." Digital assistant 105 is configured to deactivate the use of the group hotword associated with the specific action when the long-term action ends. Therefore, when the long-term action ends, digital assistant 105 can update the hotword registry 500 to delete the hotword for the specific action and notify the selected group of AEDs that it will no longer respond to the hotword group for the specific action. User 102 can respond at any time by providing voice input or by... Figure 2A and 2B The GUI refuses to use hotword groups for specific operations.

[0084] Figure 6This is a flowchart of an exemplary operational arrangement for a method 600 for enabling and assigning hot word groups to a selected group of Assistant Activation Devices (AEDs) 104. At operation 602, method 600 includes receiving an assignment instruction on the data processing hardware 10 of a first AED 104a, which assigns a hot word group 50g to a selected group of AEDs 104 associated with a user. The selected group of AEDs 104 includes the first AED 104a and one or more other AEDs 104b-n. Each AED in the selected group of AEDs 104 is configured to wake up from a low-power state when at least one AED in the selected group of AEDs 104 detects the hot word group 50g in streaming audio.

[0085] In operation 604, method 600 includes receiving audio data 20 corresponding to utterance 126 spoken by user 102 at data processing hardware 10. Audio data 20 includes a query 128 specifying an operation to be performed. In operation 606, method 600 includes detecting hot word groups 50g in the audio data 20 using a hot word detection model 114 by data processing hardware 10.

[0086] In operation 608, in response to detecting hot word group 50g in audio data 20, method 600 includes triggering a first AED 104a to wake up from a low-power state by data processing hardware and executing a cooperation routine 150 by data processing hardware 10, causing the first AED 104a in a selected group of AEDs to cooperate with each other to perform the operation specified by query 128.

[0087] A software application (i.e., a software resource) can refer to computer software that instructs a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, streaming media applications, social networking applications, and game applications.

[0088] Non-transitory memory can be a physical device used to store programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0089] Figure 7 This is a schematic diagram of an exemplary computing device 700 that can be used to implement the systems and methods described in this document. The computing device 700 is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the description and / or claims of the invention in this document.

[0090] Computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 connected to the memory 720, a high-speed expansion port 750, and a low-speed interface / controller 760 connected to a low-speed bus 770 and the storage device 730. Each of components 710, 720, 730, 740, 750, and 760 is interconnected using various buses and can be mounted on a common motherboard or otherwise suitably mounted. Processor 710 can process instructions for execution within computing device 700, including instructions stored in memory 720 or storage device 730, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as a display 780 coupled to high-speed interface 740. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, may be suitably used. Additionally, multiple computing devices 700 may be connected, with each device providing some of the necessary operations (e.g., as a server group, a set of blade servers, or a multiprocessor system).

[0091] Memory 720 stores information non-transitorily within computing device 700. Memory 720 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-transitory memory 720 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) on a temporary or permanent basis for use by computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0092] Storage device 730 provides mass storage for computing device 700. In some embodiments, storage device 730 is a computer-readable medium. In various embodiments, storage device 730 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or a device array, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 720, storage device 730, or memory on processor 710.

[0093] High-speed controller 740 manages bandwidth-intensive operations for computing device 700, while low-speed controller 760 manages lower bandwidth-intensive operations. This assignment of responsibilities is merely illustrative. In some embodiments, high-speed controller 740 is coupled to memory 720, display 780 (e.g., via a graphics processor or accelerator), and high-speed expansion port 750, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 760 is coupled to storage device 730 and low-speed expansion port 790. Low-speed expansion port 790 may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet) and may be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or network devices, such as switches or routers, for example, via network adapters.

[0094] The computing device 700 can be implemented in a variety of different forms, as shown in the figure. For example, it can be implemented as a standard server 700a or multiple times in a set of such servers 700a, as a laptop 700b, or as part of a rack server system 700c.

[0095] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These different implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system, at least one input device, and at least one output device.

[0096] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level programming and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, instrumentation, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0097] The processes and logic flows described in this specification can be executed by one or more programmable processors (also known as data processing hardware) to perform functions by manipulating input data and producing outputs by executing one or more computer programs. Processes and logic flows can also be executed by special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). Processors suitable for executing computer programs include, for example, general-purpose microprocessors and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor that executes instructions and one or more storage devices that store instructions and data. Typically, a computer will also include, or be operatively coupled to, receiving data from or transferring data to one or more mass storage devices for storing data, or both, such as magnetic disks, magneto-optical disks, or optical disks. However, a computer does not require such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor storage devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory can be supplemented or combined with special-purpose logic circuits.

[0098] To provide interaction with a user, one or more aspects of this disclosure can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or a touchscreen for displaying information to the user) and optional keyboard and pointing devices (e.g., a mouse or trackball), through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending and receiving documents from the device used by the user; for example, by responding to a request received from a web browser by sending a webpage to a web browser on the user's client device.

[0099] Many embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the following claims.

Claims

1. A method for enabling and assigning hot word groups to selected groups of two or more assistant-enabled devices, characterized in that, include: At the data processing hardware of the first AED (Assistant Enabled Device), an assignment instruction is received to assign a hot word group to a selected group of AEDs associated with the user, the selected group of AEDs including the first AED and one or more other AEDs, each AED in the selected group of AEDs being configured to run a trained hot word detection model to detect the presence of the hot word group in the streaming audio without performing speech recognition, and to wake up from a low-power state when the hot word group is detected in the streaming audio by at least one of the AEDs in the selected group of AEDs, wherein other AEDs associated with the user that are not in the selected group of AEDs are configured not to wake up from a low-power state when the hot word group is detected in the streaming audio by at least one of the AEDs in the selected group of AEDs; At the data processing hardware, audio data corresponding to the words spoken by the user is received, the audio data including the hot words and a query specifying the operation to be performed; The data processing hardware uses the hot word detection model running on the first AED to detect the hot word groups in the audio data without performing speech recognition processing on the audio data; and In response to the detection of the hot word group in the audio data: The data processing hardware triggers the first AED to wake up from the low-power state; and Without performing speech recognition processing on the audio data, the data processing hardware invokes each other AED in a selected group of AEDs that did not detect the hot word group in the audio data to wake up from a low-power state, and cooperates with all AEDs in the selected group to perform the operation specified by the query; and The data processing hardware executes a collaboration routine to enable the first AED in the selected group of AEDs and each other AED to cooperate with each other to complete the operation specified by the query.

2. The method according to claim 1, characterized in that, After receiving the allocation instruction, the process further includes: The data processing hardware performs a leader election process to elect one or more AEDs from the selected group of AEDs based on the corresponding device attributes associated with each AED in the selected group, to listen for the presence of the hot word group in the streaming audio on behalf of the selected group of AEDs. In this process, at least the first AED in the selected group of AEDs is elected by the leadership election process to detect the presence of the hot word group in the streaming audio.

3. The method according to claim 2, characterized in that, Further, the data processing hardware re-executes the leadership election process in response to at least one of the following: For updates to the selected group of the AED assigned to the hot word group, add one or more additional AEDs to the selected group of the AED and / or delete one or more of the AEDs from the selected group of the AED; or A change in device status occurs at one of the AEDs in the selected group of AEDs.

4. The method according to claim 2, characterized in that, The corresponding device attributes associated with each AED in the selected group of AEDs include at least one of the following: processing power, device type, user-configurable device settings, power consumption, battery level, physical location of the AED, or network capabilities.

5. The method according to claim 1, characterized in that, In response to detecting the hot word group in the audio data, the method further includes: The data processing hardware identifies each of the one or more other AEDs in a selected group of the AED assigned the hot word group by accessing a hot word registry. The hot word registry contains a list of one or more hot words, each hot word being assigned to one or more AEDs associated with the user, and the list of one or more hot words includes the hot word group. Each of the other AEDs in the selected group of the AED is invoked based on each of the one or more other AEDs assigned to the hot word group, as identified by accessing the hot word registry.

6. The method according to claim 5, characterized in that, The hot word registry is stored in at least one of the following: The first AED; At least one other AED associated with the user; or The server that communicates with the first AED.

7. The method according to claim 1, characterized in that, Receiving the assignment instruction to assign the hot word group to a selected group of the AED includes: The system receives voice input from the user requesting the digital assistant to activate the hot word group and assign the hot word group to a selected group of the AED. Instruct the speech recognizer to perform speech recognition on the audio data to generate an ASR (Automatic Speech Recognition) result for the speech input; and A query interpretation is performed on the ASR results used for the voice input to identify hot word group queries, which specify the hot word groups to be enabled and the name of each AED in a selected group to be assigned the hot word groups.

8. The method according to claim 1, characterized in that, Receiving the assignment instruction to assign the hot word group to a selected group of the AED includes: receiving a user input indication that instructs a user to interact with one or more objects displayed in a graphical user interface to instruct the digital assistant to enable the hot word group and select a selected group of the AED to which the hot word group is to be assigned.

9. The method according to claim 1, characterized in that, Further includes: The data processing hardware receives previous audio data spoken by the user and corresponding to commands from a digital assistant to perform long-term actions on two or more AEDs associated with the user. The digital assistant is configured to automatically create the hotword phrases for subsequent queries related to the long-term actions. The assignment instruction to assign the hot word group to a selected group of AEDs includes: receiving the hot word group automatically created by the digital assistant, wherein the selected group of AEDs to which the hot word group is assigned includes the two or more AEDs that perform the long-term action.

10. The method according to claim 9, characterized in that, Further includes: The data processing hardware outputs synthesized speech corresponding to the response from the digital assistant for the audible playback from the first AED, indicating that the long-term action is being performed, and automatically generated hotword phrases for subsequent queries related to the long-term action. The digital assistant then deactivates the use of the automatically created hot word group at the end of the long-term action.

11. The method according to claim 1, characterized in that, Each AED in the selected group is associated with the same device type or shares common attributes; The hot word group assigned to the selected group of the AED includes implicit hot words that uniquely identify the device type or general attribute associated with the selected group of the AED; and The hot word detection model is pre-trained to detect the presence of hot word groups in streaming audio without performing speech recognition on the audio data.

12. The method according to claim 1, characterized in that, The hot word group assigned to the selected group of AEDs includes proximity-based hot words, which address only a subset of one or more AEDs in the selected group of AEDs that are currently closest to the user, without explicitly identifying any specific AED in the subset of one or more AEDs; The hot word detection model is pre-trained to detect the presence of hot word groups in streaming audio without performing speech recognition on the audio data; as well as The collaboration routine is executed to cause the first AED and each other AED in the selected group of AEDs to determine a corresponding proximity value relative to the user, and to perform arbitration in the selected group of AEDs using the corresponding proximity value to elect the subset of the one or more AEDs to complete the execution of the operation specified by the query.

13. The method according to claim 1, characterized in that, The hot word group includes one of the following: Custom hot words created by the user; or Select a predefined hot word from a list of one or more available hot word groups.

14. The method according to claim 1, characterized in that, During the execution of the cooperative routine, the first AED in the selected group of AEDs and each other AED cooperate with each other by specifying one of the AEDs in the selected group of AEDs to perform the following: Generate speech recognition results for the audio data; Perform a query interpretation on the speech recognition result to determine if the speech recognition result recognizes the query specifying the operation to be performed; as well as The query interpretation performed on the speech recognition results is shared with other AEDs in the selected group of the AED.

15. The method according to claim 1, characterized in that, The query specifying the operation to be performed includes a query to perform a long-term action on a selected group of the AED; and During the execution of the cooperative routine, each AED in the selected group of AEDs cooperates with each other in the following manner: They pair up with each other during the duration of the long-term action; and Coordinate the execution of sub-actions related to the long-term action.

16. The method according to claim 1, characterized in that, The operation specified by the query includes a device-level operation performed on each AED in the selected group of AEDs; and During the execution of the collaborative routine, each AED in the selected group of AEDs collaborates with each other by independently performing the device-level operations.

17. A first AED (Assistant Activation Device) for activating hot word groups and assigning them to a selected group of two or more assistant activation devices, characterized in that, include: Data processing hardware; as well as Memory hardware communicating with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, including: The system receives an assignment instruction to assign a hot word group to a selected group of AEDs associated with a user, the selected group of AEDs including the first AED and one or more other AEDs, each AED in the selected group of AEDs being configured to run a trained hot word detection model to detect the presence of the hot word group in streaming audio without performing speech recognition, and to wake up from a low-power state when at least one of the AEDs in the selected group of AEDs detects the hot word group in streaming audio, wherein other AEDs associated with the user that are not in the selected group of AEDs are configured not to wake up from a low-power state when the hot word group is detected in streaming audio by at least one of the AEDs in the selected group of AEDs; Receive audio data corresponding to the words spoken by the user, the audio data including the hot words and a query specifying the operation to be performed; Using the hot word detection model running on the first AED, the hot word groups are detected in the audio data without performing speech recognition processing on the audio data; and In response to the detection of the hot word group in the audio data: Trigger the first AED to wake up from the low-power state; and Without performing speech recognition processing on the audio data, the data processing hardware invokes each other AED in a selected group of AEDs that did not detect the hot word group in the audio data to wake up from a low-power state, and cooperates with all AEDs in the selected group to perform the operation specified by the query; and A collaboration routine is executed such that the first AED in the selected group of AEDs and each other AED cooperate with each other to perform the operation specified by the query.

18. The first AED according to claim 17, characterized in that, After receiving the allocation instruction, the operation further includes: A leadership election process is performed to elect one or more AEDs from the selected group of AEDs based on the corresponding device attributes associated with each AED in the selected group of AEDs, to listen for the presence of the hot word group in the streaming audio on behalf of the selected group of AEDs. In this process, at least the first AED in the selected group of AEDs is elected by the leadership election process to detect the presence of the hot word group in the streaming audio.

19. The first AED as described in claim 18, characterized in that, The operation further includes re-executing the leadership election process in response to at least one of the following: For updates to the selected group of the AED assigned to the hot word group, add one or more additional AEDs to the selected group of the AED and / or delete one or more of the AEDs from the selected group of the AED; or A change in device status occurs at one of the AEDs in the selected group of AEDs.

20. The first AED according to claim 18, characterized in that, The corresponding device attributes associated with each AED in the selected group of AEDs include at least one of the following: processing power, device type, user-configurable device settings, power consumption, battery level, physical location of the AED, or network capabilities.

21. The first AED according to claim 17, characterized in that, The operation further includes, in response to detecting the hot word group in the audio data, further including: By accessing a hot word registry, each of the one or more other AEDs in a selected group of the AED assigned the hot word group is identified. The hot word registry contains a list of one or more hot words, each hot word being assigned to one or more AEDs associated with the user, and the list of one or more hot words includes the hot word group. Each of the other AEDs in the selected group of the AED is invoked based on each of the one or more other AEDs assigned to the hot word group, as identified by accessing the hot word registry.

22. The first AED according to claim 21, characterized in that, The hot word registry is stored in at least one of the following: The first AED; At least one other AED associated with the user; or The server that communicates with the first AED.

23. The first AED according to claim 17, characterized in that, Receiving the assignment instruction to assign the hot word group to a selected group of the AED includes: The system receives voice input from the user requesting the digital assistant to activate the hot word group and assign the hot word group to a selected group of the AED. Instruct the speech recognizer to perform speech recognition on the audio data to generate an ASR (Automatic Speech Recognition) result for the speech input; and A query interpretation is performed on the ASR results used for the voice input to identify hot word group queries, which specify the hot word groups to be enabled and the name of each AED in a selected group to be assigned the hot word groups.

24. The first AED according to claim 17, characterized in that, Receiving the assignment instruction to assign the hot word group to a selected group of the AED includes: receiving a user input indication that instructs a user to interact with one or more objects displayed in a graphical user interface to instruct the digital assistant to enable the hot word group and select a selected group of the AED to which the hot word group is to be assigned.

25. The first AED according to claim 17, characterized in that, The operation further includes: The system receives previous audio data spoken by the user and corresponding to commands from a digital assistant to perform long-term actions on two or more AEDs associated with the user. The digital assistant is configured to automatically create the hotword phrases for subsequent queries related to the long-term actions. The assignment instruction to assign the hot word group to a selected group of AEDs includes: receiving the hot word group automatically created by the digital assistant, wherein the selected group of AEDs to which the hot word group is assigned includes the two or more AEDs that perform the long-term action.

26. The first AED according to claim 25, characterized in that, Further includes: For the audible playback from the first AED, synthesized speech corresponding to the response from the digital assistant is output to indicate that the long-term action is being performed, and automatically generated hotword phrases are used for subsequent queries related to the long-term action. The digital assistant then deactivates the use of the automatically created hot word group at the end of the long-term action.

27. The first AED according to claim 17, characterized in that, Each AED in the selected group is associated with the same device type or shares common attributes; The hot word group assigned to the selected group of the AED includes implicit hot words that uniquely identify the device type or general attribute associated with the selected group of the AED; and The hot word detection model is pre-trained to detect the presence of hot word groups in streaming audio without performing speech recognition on the audio data.

28. The first AED according to claim 17, characterized in that, The hot word group assigned to the selected group of AEDs includes proximity-based hot words, which address only a subset of one or more AEDs in the selected group of AEDs that are currently closest to the user, without explicitly identifying any specific AED in the subset of one or more AEDs; The hot word detection model is pre-trained to detect the presence of hot word groups in streaming audio without performing speech recognition on the audio data; as well as The collaboration routine is executed to cause the first AED and each other AED in the selected group of AEDs to determine a corresponding proximity value relative to the user, and to perform arbitration in the selected group of AEDs using the corresponding proximity value to elect the subset of the one or more AEDs to complete the execution of the operation specified by the query.

29. The first AED according to claim 17, characterized in that, The hot word group includes one of the following: Custom hot words created by the user; or Select a predefined hot word from a list of one or more available hot word groups.

30. The first AED according to claim 17, characterized in that, During the execution of the cooperative routine, the first AED in the selected group of AEDs and each other AED cooperate with each other by specifying one of the AEDs in the selected group of AEDs to perform the following: Generate speech recognition results for the audio data; Perform a query interpretation on the speech recognition result to determine if the speech recognition result recognizes the query specifying the operation to be performed; as well as The query interpretation performed on the speech recognition results is shared with other AEDs in the selected group of the AED.

31. The first AED according to claim 17, characterized in that, The query specifying the operation to be performed includes a query to perform a long-term action on a selected group of the AED; and During the execution of the cooperative routine, each AED in the selected group of AEDs cooperates with each other in the following manner: They pair up with each other during the duration of the long-term action; and Coordinate the execution of sub-actions related to the long-term action.

32. The first AED according to claim 17, characterized in that, The operation specified by the query includes a device-level operation performed on each AED in the selected group of AEDs; and During the execution of the collaborative routine, each AED in the selected group of AEDs collaborates with each other by independently performing the device-level operations.

Images