Network microphone device with command keyword conditioning

Abstract

In one aspect, a playback device includes a voice assistant service (VAS) wake word engine and a command keyword engine. The playback device detects, via the command keyword engine, a first command keyword and determines whether one or more playback conditions corresponding to the first command keyword are satisfied. Based on (a) detecting the first command keyword and (b) determining that the one or more playback conditions corresponding to the first command keyword are satisfied, the playback device executes a first playback command corresponding to the first command keyword. When the playback device detects, via the wake word engine, a wake word in a voice input, the playback device streams, to one or more remote servers associated with the VAS, sound data corresponding to at least a portion of the voice input.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 439,009, filed June 12, 2019, entitled “NETWORK MICROPHONE WITH COMMANDKEYWORD CONDITIONING”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This technology relates to consumer products, and more specifically, to methods, systems, products, features, services and other elements relating to voice-assisted control of media playback systems or some aspect thereof. Background Technology

[0004] Access to and listening to digital audio via external speakers was limited until SONOS began developing a new playback system in 2002. Sonos then filed one of its first patent applications in 2003 entitled "Method for Synchronizing Audio Playback between Multiple Networked Devices," and began offering its first media playback systems for sale in 2005. Sonos wireless home audio systems allow people to experience music from multiple sources via one or more networked playback devices. Through a software control app installed on a controller (e.g., a smartphone, tablet, computer, voice input device), people can play their desired content in any room with networked playback devices. Media content (e.g., songs, podcasts, video audio) can be streamed to the playback devices, allowing each room with a playback device to play different media content. Additionally, rooms can be grouped together to synchronously play the same media content, and / or the same media content can be listened to synchronously in all rooms. Attached Figure Description

[0005] The features, aspects, and advantages of the currently disclosed technology can be better understood by referring to the following description, appended claims, and drawings, in which:

[0006] The features, aspects, and advantages of the currently disclosed technology can be better understood by referring to the following description, appended claims, and drawings. Those skilled in the art will understand that the features shown in the drawings are for illustrative purposes, and variations in the arrangement of different and / or additional features are possible.

[0007] FIG. 1Ais a partial cutaway view of an environment with a media playback system configured in accordance with aspects of the disclosed technology.

[0008] FIG. 1B is FIG. 1A a schematic diagram of a media playback system and one or more networks.

[0009] FIG. 2A is a functional block diagram of an example playback device.

[0010] FIG. 2B is FIG. 2A a perspective view of an example housing of a playback device.

[0011] FIG. 2C is a diagram of an example voice input.

[0012] FIG. 2D is a diagram depicting example sound samples in accordance with aspects of the disclosure.

[0013] FIG. 3A , 3B , 3C, 3D, and 3E are diagrams showing example playback device configurations in accordance with aspects of the disclosure.

[0014] FIG. 4 is a functional block diagram of an example controller device in accordance with aspects of the disclosure.

[0015] FIG. 5A and FIG. 5B is a controller interface in accordance with aspects of the disclosure.

[0016] FIG. 6 is a message flow diagram of a media playback system.

[0017] FIG. 7A is a functional block diagram of certain components of a first example network microphone device in accordance with aspects of the disclosure.

[0018] FIG. 7B is a functional block diagram of certain components of a second example network microphone device in accordance with aspects of the disclosure.

[0019] FIG. 7C is a functional block diagram showing an example state machine in accordance with aspects of the disclosure.

[0020] FIG. 8 shows an example noise plot showing analyzed sound metadata associated with background speech.

[0021] FIG. 9A shows a first portion of a table showing example command keywords and associated conditions in accordance with aspects of the disclosure.

[0022] FIG. 9BA second portion of a table showing example command keywords and associated conditions in accordance with aspects of the present disclosure is shown.

[0023] FIG. 10 is a schematic diagram showing an example media playback system and cloud network in accordance with aspects of the present disclosure.

[0024] FIG. 11 A table showing an example playlist in accordance with aspects of the present disclosure is shown.

[0025] FIG. 12 is a flowchart of an example method of performing operations based on command keywords in accordance with aspects of the present disclosure.

[0026] FIG. 13 is a flowchart of an example method of performing operations based on command keywords in accordance with aspects of the present disclosure.

[0027] FIG. 14 is a flowchart of an example method of performing operations based on command keywords in accordance with aspects of the present disclosure. and

[0028] FIG. 15A , FIG. 15B , FIG. 15C and FIG. 15D Example output of an example NMD configured in accordance with aspects of the present disclosure is shown.

[0029] The accompanying drawings are for purposes of illustrating the example embodiments, but it is understood that the application is not limited to the arrangements and instrumentalities shown in the drawings. In the drawings, like reference numerals identify like elements throughout. To facilitate discussion, one or more most significant bits of any reference number refer to the figure in which that element is first introduced. For example, element 103a is first introduced with respect to FIG. 1. FIG. 1A Element 103a is introduced and discussed. DETAILED DESCRIPTION

[0030] I. SUMMARY

[0031] Example techniques described herein relate to a wake-word engine configured to detect commands. An example network microphone device (“NMD”) can implement such a wake-word engine in parallel with a wake-word engine that invokes a voice assistant service (“VAS”). While the VAS wake-word engine can involve a random wake-word, the command keyword engine is invoked by a command, e.g., “play” or “skip.”

[0032] Network microphone devices can be used to facilitate voice control of smart home devices, such as wireless audio playback devices, lighting devices, appliances, and home automation devices (e.g., thermostats, door locks, etc.). An NMD is a networked computing device that typically includes an arrangement of microphones (e.g., a microphone array) configured to detect sounds present in the NMD’s environment. In some examples, an NMD can be implemented within another device, such as an audio playback device.

[0033] Voice input to such NMDs typically includes a wake word followed by an utterance that includes a user request. In practice, the wake word is typically a predetermined random word or phrase used to “wake up” the NMD and cause it to invoke a particular voice assistant service (“VAS”) to interpret the intent of the voice input in the detected sounds. For example, a user can say the wake word “Alexa” to invoke the VAS, “Ok, Google” to invoke the VAS, “Hey, Siri” to invoke the VAS, or “Hey, Sonos” to invoke the VAS provided by and other examples. In practice, a wake word can also be referred to as, for example, an activation word, a trigger word, an arousal word, or phrase, and can take the form of any suitable word, combination of words (e.g., a particular phrase), and / or some other audio cue.

[0034] To identify whether the sounds detected by an NMD contain voice input that includes a particular wake word, the NMD typically utilizes a wake word engine that is typically onboard the NMD. The wake word engine can be configured to use one or more recognition algorithms to identify (i.e., “find” or “detect”) the particular wake word in the recorded audio. Such recognition algorithms can include pattern recognition trained to detect the frequency and / or time domain patterns created by speaking the wake word. This wake word recognition process is often referred to as “keyword spotting.” In practice, to help facilitate keyword spotting, an NMD can buffer sounds detected by the NMD’s microphones, and then use the wake word engine to process the buffered sounds to determine whether the wake word is present in the recorded audio.

[0035] When the wake-word engine detects a wake-word in the recorded audio, the NMD can determine that a wake-word event (i.e., a “wake-word trigger”) has occurred, which indicates that the NMD has detected a sound that includes a potential voice input. The occurrence of a wake-word event typically causes the NMD to perform additional processes involving the detected sound. For a VAS wake-word engine, these additional processes can include: extracting the detected sound data from the buffer, and other possible additional processes such as outputting an alert (e.g., an audible chime and / or an indicator light) indicating that a wake-word has been recognized. Extracting the detected sound can include: reading out and packaging the stream of detected sound according to a particular format, and sending the packaged sound data to an appropriate VAS for interpretation.

[0036] In turn, the VAS corresponding to the wake-word recognized by the wake-word engine receives the sent sound data from the NMD over a communication network. Traditionally, the VAS takes the form of a remote service implemented using one or more cloud servers configured to process voice inputs (e.g., AMAZON’s ALEXA, APPLE’s SIRI, MICROSOFT’s CORTANA, GOOGLE’s ASSISTANT, etc.). In some cases, certain components and functions of the VAS can be distributed across local and remote devices.

[0037] When the VAS receives the detected sound data, the VAS processes the data, which involves recognizing the voice input and determining the intent of the words captured in the voice input. The VAS can then provide a response back to the NMD using certain instructions based on the determined intent. Based on the instructions, the NMD can cause one or more smart devices to perform an operation. For example, in other examples, the NMD can cause a playback device to play a particular song, or cause a lighting device to turn on / off, according to instructions from the VAS. In some cases, the NMD or a media system with the NMD (e.g., a media playback system with a playback device equipped with the NMD) can be configured to interact with multiple VASs. In practice, the NMD can select one VAS over another based on a particular wake-word recognized in the sound detected at the NMD.

[0038] One challenge faced by traditional wake-word engines is that they are prone to false positives caused by “false wake-word” triggers. A false positive in the context of an NMD generally refers to a detected sound input that incorrectly invokes a VAS. For a VAS wake-word engine, a false positive can invoke a VAS even if, in fact, no user intended to speak a wake-word to the NMD.

[0039] For example, a false positive can occur when the wake-word engine recognizes the wake-word in the detected sound from audio (e.g., music, a podcast, etc.) being played by the NMD environment. The output audio can be played from a playback device in the vicinity of the NMD or by the NMD itself. For example, when a commercial for the AMAZON ALEXA service is output in the vicinity of the NMD, the word “Alexa” in the commercial can trigger a false positive. The word or phrase in the output audio that causes the false positive can be referred to herein as a “false wake-word.”

[0040] In other examples, a word that is phonetically similar to the actual wake-word causes a false positive. For example, when a commercial for a car is output in the vicinity of the NMD, the word “Lexus” can be a false wake-word that causes a false positive because the word is phonetically similar to “Alexa.” As other examples, a false positive can occur when a person says the VAS wake-word or a phonetically similar word in a conversation. In other examples, a word that is phonetically similar to the actual wake-word causes a false positive. For example, when a commercial for a car is output in the vicinity of the NMD, the word “Lexus” can be a false wake-word that causes a false positive because the word is phonetically similar to “Alexa.” As other examples, a false positive can occur when a person says the VAS wake-word or a phonetically similar word in a conversation.

[0041] The occurrence of false positives is undesirable because they can cause the NMD to consume additional resources or interrupt audio playback, among other possible negative consequences. Some NMDs can avoid false positives by requiring a button to be pressed to invoke the VAS, such as on an AMAZON FIRE TV remote or an APPLE TV remote. In practice, the impact of false positives produced by the VAS wake-word engine is often mitigated in part by the VAS processing the detected sound data and determining that the detected sound data does not include an identifiable voice input.

[0042] In contrast to a predetermined random wake-word that invokes the VAS, a keyword that invokes a command (referred to herein as a “command keyword”) can be a word or combination of words (e.g., a phrase) that is used as the command itself, such as a playback command. In some implementations, the command keyword can be used as both a wake-word and the command itself. That is, when the command keyword engine detects the command keyword in the recorded audio, the NMD can determine that a command keyword event has occurred and responsively execute the command corresponding to the detected keyword. For example, based on detecting the command keyword “pause,” the NMD causes playback to pause. One advantage of the command keyword engine is that the recorded audio does not necessarily need to be sent to the VAS for processing, which can result in faster responses to voice inputs and increased user privacy, among other possible benefits. In some implementations described below, the detected command keyword event can cause one or more subsequent actions, such as local natural language processing of the voice input. In some implementations, the command keyword event can be one of one or more other conditions that must be detected before causing such actions.

[0043] According to example techniques described herein, after detecting a command keyword, an example NMD can generate a command keyword event (and execute a command corresponding to the detected command keyword) only if certain conditions corresponding to the detected command keyword are satisfied. For example, after detecting the command keyword "skip," an example NMD generates a command keyword event (and skips to the next track) only if certain playback conditions indicating that a skip should be performed are satisfied. These playback conditions can include, for example, (i) a first condition that a media item is being played back; (ii) a second condition that a queue is active; and (iii) a third condition that the queue includes a media item after the media item being played back. If any of these conditions are not satisfied, then a command keyword event is not generated (and a skip is not performed).

[0044] By requiring (a) detection of a command keyword and (b) certain conditions corresponding to the detected command keyword before generating a command keyword event, the incidence of false positives can be reduced. For example, while playing TV audio, a conversation or other TV audio does not generate a false positive for the "skip" command keyword because the TV audio input is active (rather than a queue). Moreover, the NMD can continuously listen for command keywords (rather than needing to press a button to put the NMD in a state to receive voice input) as a condition related to the state in which a controlled device gate wake word event is generated.

[0045] Various aspects of the conditioning of keyword events can also apply to VAS wake word engines and other traditional random wake word engines. For example, such conditioning can make other wake word engines practical in addition to command keyword engines, which can be prone to false positives. For example, an NMD can include a streaming audio service wake word engine that supports certain wake words unique to a streaming audio service. For example, after detecting a streaming audio service wake word, an example NMD generates a streaming audio service wake word event only if a certain streaming audio service is satisfied. These playback conditions can include, for example, (i) a valid subscription to the streaming audio service; and (ii) an audio track from the streaming audio service in the queue; among other examples.

[0046] Moreover, command keywords can be single words or phrases. Phrases generally include more syllables, which generally makes command keywords more unique and easier to be recognized by command keyword engines. Thus, in some cases, command keywords that are phrases can be less prone to false positive detection. Moreover, using phrases can allow more intent to be incorporated into command keywords. For example, the command keyword "skip forward" indicates that a skip should be forward in the queue to a subsequent track, rather than backward to a previous track.

[0047] In addition, the NMD can include a local natural language unit (NLU). In contrast to an NLU implemented in one or more cloud servers capable of recognizing a wide variety of speech inputs, an example local NLU is capable of recognizing a relatively small library of keywords (e.g., 10,000 words and phrases), which facilitates practical implementation on the NMD. When the command keyword engine generates a command keyword event after detecting a command keyword in a speech input, the local NLU can process a speech utterance portion of the speech input to look up keywords from the library and determine an intent from the found keywords.

[0048] If the speech utterance portion of the speech input includes at least one keyword from the library, the NMD can execute a command corresponding to the command keyword according to one or more parameters corresponding to the at least one keyword. In other words, the keyword can change or customize the command corresponding to the command keyword. For example, the command keyword engine can be configured to detect "play" as a command keyword, and the local NLU library can include the phrase "at low volume." Then, if the user speaks "play music at low volume" as a speech input, the command keyword engine generates a command keyword event for "play" and uses the keyword "at low volume" as a parameter for the "play" command. Thus, the NMD not only causes playback based on the speech input, but also lowers the volume.

[0049] Example techniques involve customizing keywords in the library for a user of a media playback system. For example, the NMD can populate the library with names that have already been configured in the media playback system (e.g., zone names, smart device names, and user names). In addition, the NMD can populate the local NLU library with names of favorite playlists, internet radio stations, and the like. This customization allows the local NLU to more efficiently assist the user with voice commands. This customization can also be advantageous because the size of the local NLU library can be limited.

[0050] One possible advantage of the local NLU is increased privacy. By processing speech utterances locally, the user can avoid transmitting speech recordings to the cloud (e.g., to servers of a voice assistant service). In addition, in some implementations, the NMD can use a local area network to discover playback devices and / or smart devices connected to the network, which can avoid providing this data to the cloud. Furthermore, the user's preferences and customizations can remain local to the NMD in the home, possibly using the cloud only as an optional backup. Other advantages are also possible.

[0051] As described above, the example techniques are related to command keywords. A first example implementation involves a device that includes: a network interface; one or more processors; at least one microphone configured to detect sound; at least one speaker; a wake-word engine configured to receive input sound data representing sound detected by the at least one microphone and generate a voice assistant service (VAS) wake-word event when the wake-word engine detects a VAS wake-word in the input sound data, wherein, when the VAS wake-word event is generated, the device streams sound data representing sound detected by the at least one microphone to one or more servers of a voice assistant service; and a command keyword engine configured to receive input sound data representing sound detected by the at least one microphone and generate a command keyword event when (a) the second wake-word engine detects one of a plurality of command keywords supported by the second wake-word engine in the input sound data, and (b) one or more playback conditions corresponding to the detected command keyword are satisfied, wherein each of the plurality of command keywords is a respective playback command. The device detects, via the command keyword engine, a first command keyword, and determines whether one or more playback conditions corresponding to the first command keyword are satisfied. Based on (a) detecting the first command keyword, and (b) determining that the one or more playback conditions corresponding to the first command keyword are satisfied, the device generates, via the command keyword engine, a command keyword event corresponding to the first command keyword. In response to the command keyword event, and determining that the one or more playback conditions are satisfied, the device executes a first playback command corresponding to the first command keyword.

[0052] A second example implementation involves a device that includes: a network interface; one or more processors; at least one microphone configured to detect sound; at least one speaker; a wake-word engine configured to receive input sound data representing sound detected by the at least one microphone and generate a voice assistant service (VAS) wake-word event when the wake-word engine detects a VAS wake-word in the input sound data, wherein, when the VAS wake-word event is generated, the device streams sound data representing sound detected by the at least one microphone to one or more servers of a voice assistant service; a command keyword engine configured to receive input sound data representing sound detected by the at least one microphone. The device detects, via the command keyword engine, a first command keyword that is one of a plurality of command keywords supported by the device, and determines, via a local natural language unit (NLU), an intent based on the at least one keyword. After detecting the first command keyword event and determining the intent, the device executes a first playback command corresponding to the first command keyword in accordance with the determined intent.

[0053] While some embodiments described herein can refer to functions performed by a given actor (e.g., a“user” and / or other entity), it should be understood that this description is for explanatory purposes only. Unless the language of a claim expressly requires, no claim should be construed as requiring any such exemplar actor to act.

[0054] Further, some functions are described herein as being performed“based on” or“in response to” another element or function. To be“based on” is understood to mean that one element or function is related to another function or element. To be“in response to” is understood to mean that one element or function is a necessary result of another function or element. For brevity, functions are often described as being based on another function when a chain of functions exists; however, such disclosure is understood to disclose any type of functional relationship.

[0055] II. Example Operating Environment

[0056] FIG. 1A And FIG. 1B An example configuration of a media playback system 100 (or“MPS 100”) in which one or more embodiments disclosed herein can be implemented is shown. Reference is first made to FIG. 1A The illustrated MPS 100 is associated with an example home environment having multiple rooms and spaces, which can be collectively referred to as a“home environment,”“smart home,” or“environment 101.” The environment 101 includes a home having several rooms, spaces, and / or playback zones, including a master bathroom 101a, a master bedroom 101b (referred to herein as“Nick’s Room”), a second bedroom 101c, a family room or den 101d, an office 101e, a living room 101f, a dining room 101g, a kitchen 101h, and an outdoor patio 101i. While certain embodiments and examples are described below in the context of a home environment, the technology described herein can be implemented in other types of environments. In some embodiments, for example, the MPS 100 can be implemented in one or more commercial settings (e.g., a restaurant, a shopping mall, an airport, a hotel, a retail store, or other store), one or more vehicles (e.g., a sport utility vehicle, a bus, a car, a boat, a ship, an airplane), multiple environments (e.g., a combination of a home and a vehicle environment), and / or other suitable environments that can require multi-zone audio.

[0057] In these rooms and spaces, the MPS 100 includes one or more computing devices. Reference is made to FIG. 1A And 1BSuch computing devices can include playback devices 102 (identified individually as playback devices 102a-102o), network microphone devices 103 (identified individually as "NMDs" 103a-102i), and controller devices 104a and 104b (collectively, "controller devices 104"). Referring to FIG. 1B , the home environment can include additional and / or other computing devices, including local network devices such as one or more smart lighting devices 108 FIG. 1B ), a smart thermostat 110, and a local computing device 105 FIG. 1A . In embodiments described below, one or more of the various playback devices 102 can be configured as portable playback devices, while other playback devices can be configured as fixed playback devices. For example, earphones 102o FIG. 1B ) is a portable playback device, while a playback device 102d on a bookshelf can be a fixed device. As another example, a playback device 102c on a patio can be a battery-powered device that allows it to be transported to various areas within the environment 101 and outside of the environment 101 when not plugged into a wall outlet or the like.

[0058] Still referring to FIG. 1B , the various playback, network microphone, and controller devices 102, 103, and 104 of the MPS 100 and / or other network devices can be coupled to each other via point-to-point connections and / or through other connections via a network 111 (e.g., a LAN including a network router 109), which can be wired and / or wireless. For example, a playback device 102j in a study 101d FIG. 1A ) that can be designated as a "left" device can have a point-to-point connection with a playback device 102a that is also in the study 101d and can be designated as a "right" device. In related embodiments, the left playback device 102j can communicate with other network devices (e.g., a playback device 102b) that can be designated as a "front" device via point-to-point connections and / or via other connections via the network 111.

[0059] As further shown in FIG. 1B , the MPS 100 can be coupled to one or more remote computing devices 106 through a wide area network ("WAN") 107. In some embodiments, each remote computing device 106 can take the form of one or more cloud servers. The remote computing devices 106 can be configured to interact with the computing devices in the environment 101 in various ways. For example, the remote computing devices 106 can be configured to facilitate streaming and / or control playback of media content (e.g., audio) in the home environment 101.

[0060] In some implementations, the various playback devices, NMDs, and / or controller devices 102-104 can be communicatively coupled to at least one remote computing device associated with a VAS and at least one remote computing device associated with a media content service ("MCS"). For example, in the illustrated example of FIG. 1B FIG. 1, remote computing device 106a is associated with a VAS 190 and remote computing device 106b is associated with a MCS 192. Although only a single VAS 190 and a single MCS 192 are shown in the example of FIG. 1B FIG. 1 for clarity, the MPS 100 can be coupled to multiple different VASs and / or MCSs. In some implementations, the VAS can be operated by one or more of AMAZON, GOOGLE, APPLE, MICROSOFT, SONOS, or other voice assistant providers. In some implementations, the MCS can be operated by one or more of SPOTIFY, PANDORA, AMAZON MUSIC, or other media content services.

[0061] As further shown in FIG. 1B FIG. 1, the remote computing devices 106 also include a remote computing device 106c that is configured to perform certain operations, such as remotely facilitating media playback functions, managing device and system state information, directing communications between the devices of the MPS 100 and one or more VASs and / or MCSs, and other operations. In one example, the remote computing device 106c provides a cloud server for one or more SONOS wireless HiFi systems.

[0062] In various implementations, one or more of the playback devices 102 can take the form of or include an onboard (e.g., integrated) network microphone device. For example, the playback devices 102a-e include or are equipped with corresponding NMDs 103a-e, respectively. Unless otherwise indicated in the specification, a playback device that includes or is equipped with an NMD can be referred to interchangeably herein as a playback device or an NMD. In some cases, one or more of the NMDs 103 can be standalone devices. For example, NMDs 103f and 103g can be standalone devices. A standalone NMD can omit components and / or functionality that are typically included in a playback device (e.g., a speaker or related electronics). For example, in such cases, a standalone NMD can not produce audio output or can produce limited audio output (e.g., relatively low quality audio output).

[0063] The various playback and network microphone devices 102 and 103 of the MPS 100 can each be associated with a unique name that can be assigned to the respective device by a user, for example, during setup of one or more of the devices. For example, as shown in FIG. 1B“shelf” to playback device 102d, as it is physically located on a shelf. Similarly, the name “island” can be assigned to NMD 103f, as it is physically located on an island countertop in kitchen 101h FIG. 1A ) can be assigned names according to zones or rooms, e.g., playback devices 102e, 102i, 102m, and 102n, which are named “bedroom,” “dining room,” “living room,” and “office,” respectively. In addition, certain playback devices can have functionally descriptive names. For example, playback devices 102a and 102b are assigned the names “right” and “front,” respectively, as these two devices are configured to provide specific audio channels during media playback in the zone of study 101d FIG. 1A ) can be named “portable” as it is battery-powered and / or easily transported to different areas of environment 101. Other naming conventions are possible.

[0064] As described above, NMDs can detect and process sounds from their surrounding environment, e.g., sounds that include background noise mixed with speech spoken by a person in the vicinity of the NMD. For example, when a NMD detects a sound in an environment, the NMD can process the detected sound to determine whether the sound includes speech that contains speech input for the NMD and ultimately for a particular VAS. For example, the NMD can identify whether the speech includes a wake word associated with a particular VAS.

[0065] In the illustrated example of FIG. 1B , NMD 103 is configured to interact with VAS 190 over a network via network 111 and router 109. For example, when a NMD identifies a potential wake word in a detected sound, an interaction with VAS 190 can be initiated. The identification results in a wake word event, which in turn causes the NMD to begin sending detected sound data to VAS 190. In some implementations, various local network devices 102-105 FIG. 1A ) and / or remote computing devices 106c of MPS 100 can exchange various feedback, information, instructions, and / or related data with a remote computing device associated with a selected VAS. Such exchanges can be related to or independent of a message containing speech input. In some embodiments, the remote computing device and MPS 100 can exchange data over a communication path as described herein and / or using a metadata exchange channel as described in U.S. Application No. 15 / 438,749, filed February 21, 2017, entitled “Voice Control of a Media Playback System,” the entire contents of which are incorporated herein by reference.

[0066] Upon receiving the stream of sound data, the VAS 190 determines whether there is a voice input in the stream data from the NMD, and if so, the VAS 190 will also determine the potential intent in that voice input. The VAS 190 can next send a response back to the MPS 100, which can include sending the response directly to the NMD that caused the wake-word event. The response is typically based on the intent determined by the VAS 190 to exist in the voice input. As an example, in response to the VAS 190 receiving a voice input that is a voice prompt to "play Hey Jude by The Beatles," the VAS 190 can determine that the underlying intent of the voice input is to initiate playback, and further determine that the intent of the voice input is to play the specific song "Hey Jude." Upon these determinations, the VAS 190 can send a command to a specific MCS 192 to retrieve the content (i.e., the song "Hey Jude"), and subsequently, that MCS 192 provides this content (e.g., a stream) to the MPS 100 either directly or indirectly through the VAS 190. In some implementations, the VAS 190 can send a command to the MPS 100 that causes the MPS 100 itself to retrieve the content from the MCS 192.

[0067] In certain implementations, when a voice input is recognized in voice detected by two or more NMDs that are adjacent to each other, the NMDs can facilitate arbitration between each other. For example, the playback device 102d in the kitchen 101h of the environment 101( FIG. 1A ) is in close proximity to the living room playback device 102m that is equipped with an NMD, and both devices 102d and 102m can at least sometimes detect the same sound. In this case, arbitration can be required to determine which device is ultimately responsible for providing the detected sound data to a remote VAS. Examples of arbitrating between NMDs can be found, for example, in the previously cited U.S. Application No. 15 / 438,749.

[0068] In certain implementations, NMDs can be assigned to, or associated with, a designated or default playback device that can not contain an NMD. For example, the island NMD 103f in the kitchen 101h( FIG. 1A ) can be assigned to the dining room playback device 1021 that is relatively close to the island NMD 103f. In practice, in response to a remote VAS receiving a voice input from an NMD to play audio, the NMD can instruct the assigned playback device to play the audio, which the NMD can have sent the voice input to the VAS in response to a user speaking a command to play a specific song, album, playlist, etc. Additional details regarding assigning NMDs and playback devices as designated or default devices can be found, for example, in the previously cited U.S. Patent Application No.

[0069] Other aspects related to the different components of the example MPS 100 and how the different components interact to provide a media experience to a user can be found in the following sections. Although the discussion herein can generally refer to the example MPS 100, the techniques described herein are not limited to application in the above-described home environment, among other things. For example, the techniques described herein can be useful in other home environment configurations that include more or fewer of any of the playback, network microphone, and / or controller devices 102-104. For example, the techniques herein can be used in an environment with a single playback device 102 and / or a single NMD 103. In some examples of such a case, the network 111 FIG. 1B ) can be removed, and the single playback device 102 and / or the single NMD 103 can directly communicate with the remote computing devices 106a-d. In some embodiments, a telecommunications network (e.g., an LTE network, a 5G network, etc.) can communicate with the various playback, network microphone, and / or controller devices 102-104 independent of the LAN.

[0070] a. Example playback and network microphone devices

[0071] FIG. 2A is a functional block diagram illustrating certain aspects of a playback device 102 of the MPS 100 and FIG. 1A and FIG. 1B . As shown, the playback device 102 includes various components, each of which is discussed in further detail below, and the various components of the playback device 102 can be operatively coupled to each other by a system bus, a communications network, or some other connection mechanism. In the illustrated example of FIG. 2A , the playback device 102 can be referred to as a “NMD-equipped” playback device because it includes components that support NMD functionality, such as one of the NMDs 103 shown in FIG. 1A .

[0072] As shown, the playback device 102 includes at least one processor 212, which can be a clock-driven computing component configured to process input data according to instructions stored in a memory 213. The memory 213 can be a tangible, non-transitory computer-readable medium configured to store instructions executable by the processor 212. For example, the memory 213 can be a data storage device that can be loaded with software code 214 executable by the processor 212 to implement certain functionality.

[0073] In one example, the functionality can involve the playback device 102 obtaining audio data from an audio source, which can be another playback device. In another example, the functionality can involve the playback device 102 sending audio data, detected sound data (e.g., corresponding to voice input), and / or other information to another device on a network via the at least one network interface 224. In yet another example, the functionality can involve the playback device 102 causing one or more other playback devices to playback audio in synchronization with the playback device 102. In yet another example, the functionality can involve the playback device 102 facilitating pairing or otherwise binding with one or more other playback devices to create a multi-channel audio environment. Many other example functionalities are possible as well, some of which are discussed below.

[0074] As just mentioned, certain functionality can involve the playback device 102 playing back audio content in synchronization with one or more other playback devices. During synchronized playback, a listener can not perceive differences in time delay between the synchronized playback devices between playback of audio content. Some examples of audio playback synchronization between playback devices are provided in greater detail in U.S. Patent No. 8,234,395, filed April 4, 2004, and entitled “System and method for synchronizing operations among a plurality of independently clocked digital data processing devices,” which is incorporated by reference herein in its entirety.

[0075] To facilitate audio playback, the playback device 102 includes an audio processing component 216, which is generally configured to process audio prior to presentation by the playback device 102. In this regard, the audio processing component 216 can include one or more digital-to-analog converters (“DACs”), one or more audio pre-processing components, one or more audio enhancement components, one or more digital signal processors (“DSPs”), and / or the like. In some implementations, one or more of the audio processing components 216 can be subcomponents of the processor 212. In operation, the audio processing component 216 receives analog and / or digital audio and processes and / or intentionally alters the audio to produce audio signals for playback.

[0076] The produced audio signals can then be provided to one or more audio amplifiers 217 for amplification and playback through one or more speakers 218 operably coupled to the amplifiers 217. The audio amplifiers 217 can include components configured to amplify the audio signals to a level for driving the one or more speakers 218.

[0077] Each of the speakers 218 can include a separate sensor (e.g., a “driver”), or the speakers 218 can include a complete speaker system that includes an enclosure with one or more drivers. Particular drivers of the speakers 218 can include, for example, a subwoofer (e.g., for low frequencies), a midrange driver (e.g., for midrange frequencies), and / or a tweeter (e.g., for high frequencies). In some cases, the sensors can be driven by respective corresponding audio amplifiers of the audio amplifier 217. In some implementations, the playback device can not include speakers 218, but can include a speaker interface for connecting the playback device to external speakers. In certain embodiments, the playback device can include neither speakers 218 nor the audio amplifier 217, but can include an audio interface (not shown) for connecting the playback device to an external audio amplifier or an audio-video receiver.

[0078] In addition to generating audio signals for playback by the playback device 102, the audio processing component 216 can be configured to process audio to be transmitted over the network interface 224 to one or more other playback devices for playback. In an example scenario, audio content to be processed and / or played back by the playback device 102 can be received from an external source, e.g., through an audio line-in interface of the playback device 102 (not shown) (e.g., an auto-detecting 3.5 mm audio line-in connection) or through the network interface 224, as described below.

[0079] As shown, the at least one network interface 224 can take the form of one or more wireless interfaces 225 and / or one or more wired interfaces 226. The wireless interfaces can provide network interface functionality for the playback device 102 to communicate wirelessly with other devices (e.g., other playback devices, NMDs, and / or controller devices) according to a communication protocol (e.g., any of the wireless standards including IEEE 802.11a, 802.11b, 802.11g, 802.11n, 802.11ac, 802.15, 4G mobile communication standards, etc.). The wired interfaces can provide network interface functionality for the playback device 102 to communicate over a wired connection with other devices according to a communication protocol (e.g., IEEE 802.3). Although FIG. 2A Although the network interface 224 shown in FIG. 2 includes both wired and wireless interfaces, in some implementations the playback device 102 can include only a wireless interface or only a wired interface.

[0080] Typically, network interface 224 facilitates data flow between playback device 102 and one or more other devices on a data network. For example, playback device 102 may be configured to receive audio content from one or more other playback devices, network devices within a LAN, and / or audio content sources on a WAN (e.g., the Internet) via a data network. In one example, the audio content and other signals sent and received by playback device 102 may be transmitted in the form of digital packet data, which includes an Internet Protocol (IP)-based source address and an IP-based destination address. In this case, network interface 224 may be configured to parse the digital packet data so that data destined for playback device 102 is correctly received and processed by playback device 102.

[0081] like FIG. 2A As shown, the playback device 102 also includes a voice processing component 220 operatively coupled to one or more microphones 222. The microphones 222 are configured to detect sound (i.e., sound waves) in the environment of the playback device 102 and then provide it to the voice processing component 220. More specifically, each microphone 222 is configured to detect sound and convert it into a digital or analog signal representing the detected sound, which then enables the voice processing component 220 to perform various functions based on the detected sound, as described in more detail below. In one embodiment, the microphones 222 are arranged as a microphone array (e.g., an array of six microphones). In some embodiments, the playback device 102 includes more than six microphones (e.g., eight or twelve microphones) or fewer than six microphones (e.g., four microphones, two microphones, or a single microphone).

[0082] In operation, the speech processing component 220 is typically configured to detect and process sound received through the microphone 222, identify potential speech input in the detected sound, and extract the detected sound data to enable VAS (e.g., VAS190). FIG. 1B)) to process speech input recognized in the detected sound data. The speech processing components 220 can include one or more analog-to-digital converters, an acoustic echo canceller ("AEC"), a spatial processor (e.g., one or more multi-channel Wiener filters, one or more other filters, and / or one or more beamformer components), one or more buffers (e.g., one or more circular buffers), one or more wake-word engines, one or more speech extractors, and / or one or more speech processing components (e.g., components configured to identify speech of a particular user or a particular group of users associated with a home), among other example speech processing components. In example implementations, the speech processing components 220 can include or take the form of one or more DSPs or one or more DSP modules. In this regard, certain speech processing components 220 can be configured with particular parameters (e.g., gain and / or spectral parameters) that can be modified or tuned to achieve particular functionality. In some implementations, one or more of the speech processing components 220 can be subcomponents of the processor 212.

[0083] As FIG. 2A As further shown, the playback device 102 also includes a power supply component 227. The power supply component 227 includes at least an external power interface 228 that can be coupled to a power source (not shown) through a cable or the like that physically connects the playback device 102 to an electrical outlet or some other external power source. Other power supply components can include, for example, transformers, converters, and similar components configured to format the power supply.

[0084] In some implementations, the power supply component 227 of the playback device 102 can additionally include an internal power source 229 (e.g., one or more batteries) configured to power the playback device 102 in the absence of a physical connection to an external power source. When equipped with the internal power source 229, the playback device 102 can operate independently of an external power source. In some such implementations, the external power interface 228 can be configured to facilitate charging of the internal power source 229. As previously discussed, a playback device that includes an internal power source can be referred to herein as a "portable playback device." On the other hand, a playback device that operates using an external power source can be referred to herein as a "stationary playback device," although such a device can in fact be moved about a home or other environment.

[0085] The playback device 102 also includes a user interface 240 that can facilitate user interaction independently or in conjunction with user interaction facilitated by one or more controller devices 104. In various embodiments, the user interface 240 includes one or more physical buttons and / or supports a graphical interface provided on a touch-sensitive screen and / or surface, among others, to enable a user to provide input directly. The user interface 240 can also include one or more of a light (e.g., LED) and a speaker to provide visual and / or audio feedback to the user.

[0086] As an illustrative example, FIG. 2B An example housing 230 of the playback device 102 is shown, including a user interface in the form of a control area 232 at a top 234 of the housing 230. The control area 232 includes buttons 236a-c for controlling audio playback, volume level, and other functions. The control area 232 also includes a button 236d for toggling the microphone 222 to an on or off state.

[0087] As FIG. 2B Further shown, the control area 232 is at least partially surrounded by a hole formed in the top 234 of the housing 230 through which the microphone 222 (not visible in FIG. 2B ) receives sound in the environment of the playback device 102. The microphone 222 can be disposed in various locations along the top 234 and / or within the top 230 or other areas of the housing 230 so as to detect sound from one or more directions relative to the playback device 102.

[0088] By way of example, SONOS, Inc. currently offers (or has offered) for sale certain playback devices that can implement certain embodiments disclosed herein, including the “PLAY:1,” “PLAY:3,” “PLAY:5,” “PLAYBAR,” “CONNECT:AMP,” “PLAYBASE,” “BEAM,” “CONNECT,” and “SUB.” Any other past, present, and / or future playback devices can additionally or alternatively be used as playback devices to implement example embodiments disclosed herein. Moreover, it should be understood that the playback devices are not limited to the examples or SONOS product offerings shown. For example, the playback devices can include or take the form of a wired or wireless headphone set that can operate as part of the MPS 100 through a network interface, among others. In another example, the playback devices can include or interact with an extension base for a personal mobile media playback device. In yet another example, the playback devices can be integrated into another device or component, such as a television, lighting fixture, or some other device for use indoors or outdoors. FIG. 2A or FIG. 2B shown. For example, the playback devices can include or take the form of a wired or wireless headphone set that can operate as part of the MPS 100 through a network interface, among others. In another example, the playback devices can include or interact with an extension base for a personal mobile media playback device. In yet another example, the playback devices can be integrated into another device or component, such as a television, lighting fixture, or some other device for use indoors or outdoors.

[0089] FIG. 2C is a diagram of an example voice input 280 that can be processed by an NMD or a playback device equipped with an NMD. The voice input 280 can include a keyword portion 280a and an utterance portion 280b. The keyword portion 280a can include a wake word or a command keyword. In the case of a wake word, the keyword portion 280a corresponds to the detected sound that elicited the wake word. The utterance portion 280b corresponds to the detected sound that can include a user request following the keyword portion 280a. In response to the event elicited by the keyword portion 280a, the NMD can process the utterance portion 280b to identify the presence of any words in the detected sound data. In various implementations, a potential intent can be determined based on the words in the utterance portion 280b. In certain implementations, the potential intent can also be based on, or at least partially based on, certain words in the keyword portion 280a, such as when the keyword portion includes a command keyword. In any case, the words can correspond to one or more commands, as well as certain keywords for certain commands. The keywords in the voice utterance portion 280b can be, for example, words that identify a particular device or group in the MPS 100. For example, in the illustrated example, the keywords in the voice utterance portion 280b can be one or more words that identify one or more zones in which music is to be played (e.g., the living room and the dining room FIG. 1A ). In some cases, the utterance portion 280b can include additional information, such as a pause (e.g., a period of non-speech) between the detected words spoken by the user, as FIG. 2C illustrated. This pause can demarcate the location of separate commands, keywords, or other information spoken by the user within the utterance portion 280b.

[0090] Based on certain command criteria, the NMD and / or the remote VAS can take action as a result of identifying one or more commands in the voice input. The command criteria can be based on the inclusion of certain keywords in the voice input, among other possibilities. Additionally or alternatively, the command criteria for a command can involve the identification of one or more control state and / or zone state variables that are combined with the identification of one or more particular commands. The control state variables can include, for example, indicators that identify the volume level, the queue associated with one or more devices, and the playback state, such as whether the device is playing the queue, paused, etc. The zone state variables can include, for example, indicators that identify which zone players are grouped in a group.

[0091] In some implementations, the MPS 100 is configured to temporarily reduce the volume of audio content it is playing when a certain keyword (e.g., a wake word) is detected in the keyword portion 280a. The MPS 100 can restore the volume after processing the voice input 280. Such a process can be referred to as ducking, examples of which are disclosed in U.S. Patent Application No. 15 / 438,749, which is incorporated by reference in its entirety.

[0092] FIG. 2D An example sound sample is shown. In this example, the sound sample corresponds to a sound data stream (e.g., one or more audio frames) associated with a wake word or command keyword found in the keyword portion 280a of FIG. 2A As shown, the example sound sample includes: (i) sound detected in the environment of the NMD immediately prior to the wake word or command word being spoken, which can be referred to as a pre-roll portion (between times tO and ti); (ii) sound detected in the environment of the NMD while the wake word or command word is being spoken, which can be referred to as a wake-up portion (between times ti and t2); and / or (iii) sound detected in the environment of the NMD after the wake word or command word is spoken, which can be referred to as a post-roll portion (between times t2 and t3). Other sound samples are possible. In various implementations, aspects of the sound sample can be evaluated according to an acoustic model that aims to map mels / spectrogram features to phonemes in a given language model for further processing. For example, automatic speech recognition (ASR) can include such mapping for command keyword detection. In contrast, a wake word detection engine can be precisely tuned to recognize a particular wake word and invoke downstream actions of the VAS (e.g., by recognizing only random words in voice input processed by the playback device).

[0093] ASR for command keyword detection can be tuned to accommodate a wide range of keywords (e.g., 5, 10, 100, 1000, 10000 keywords). In contrast to wake word detection, command keyword detection can involve feeding ASR output to an onboard local NLU that, together with the ASR, determines when a command word event has occurred. In some implementations described below, the local NLU can determine an intent based on one or more other keywords in the ASR output produced by a particular voice input. In these or other implementations, the playback device can only act on a detected command keyword event when the playback device determines that certain conditions have been met (e.g., environmental conditions (e.g., low background noise)).

[0094] b. Example playback device configuration

[0095] FIG. 3A-3EAn example configuration of playback devices is shown. First, refer to FIG. 3A In some example instances, a single playback device can belong to one zone. For example, playback device 102c on the patio can belong to zone A. In some implementations described below, multiple playback devices can be "bound" to form a "bound pair" that together form a single zone. For example, playback device 102f (named "Bed 1") in the master bedroom 101h can be bound to playback device 102g (named "Bed 2") in the master bedroom 101h to form zone B. The bound playback devices can have different playback responsibilities (e.g., channel responsibilities). In another implementation described below, multiple playback devices can be merged to form a single zone. For example, playback device 102d (named "Bookshelf") can be merged with playback device 102m (named "Living Room") to form a single zone C. The merged playback devices 102d and 102m can not be specifically assigned different playback responsibilities. That is, the merged playback devices 102d and 102m can each play audio content as if unmerged, in addition to being able to play audio content in sync. FIG. 3A FIG. 1A FIG. 3A FIG. 1A

[0096] For control purposes, each zone in the MPS 100 can be represented as a single user interface ("UI") entity. For example, as displayed by the controller device 104, zone A can be provided as a single entity named "Portable," zone B can be provided as a single entity named "Stereo," and zone C can be provided as a single entity named "Living Room."

[0097] In various embodiments, a zone can take the name of one of the playback devices belonging to the zone. For example, zone C can take the name of the living room device 102m (as shown). In another example, zone C can take the name of the bookshelf device 102d. In another example, zone C can take the name of some combination of the bookshelf device 102d and the living room device 102m. The user can select the selected name through input at the controller device 104. In some embodiments, a zone can be named a different name than the devices belonging to the zone. For example, zone B in the master bedroom 101h is named "Stereo," but none of the devices in zone B have this name. In one aspect, zone B is a single UI entity representing a single device named "Stereo," which is composed of constituent devices named "Bed 1" and "Bed 2." In one implementation, the Bed 1 device can be playback device 102f in the master bedroom 101h and the Bed 2 device can also be playback device 102g in the master bedroom 101h. FIG. 3A FIG. 1A FIG. 1A

[0098] ​​​​​​​As noted above, bound playback devices can have different playback responsibilities, e.g., playback responsibilities for certain audio channels. For example, as shown in FIG. 3B Bed 1 and Bed 2 devices 102f and 102g can be bound to produce or enhance a stereo effect of the audio content. In this example, the Bed 1 playback device 102f can be configured to play left channel audio components, while the Bed 2 playback device 102g can be configured to play right channel audio components. In some implementations, such a stereo binding can be referred to as a “pairing.”

[0099] Additionally, playback devices configured to be bound can have additional and / or different respective speaker drivers. As shown in FIG. 3C the playback device 102b named “Front” can be bound with the playback device 102k named “SUB.” The Front device 102b can present a mid-high frequency range, while the SUB device 102k can present a low frequency, e.g., a subwoofer. When unbound, the Front device 102b can be configured to present the entire frequency range. As another example, FIG. 3D the Front device 102b and the SUB device 102k are shown further bound with the right playback device 102a and the left playback device 102j, respectively. In some implementations, the right device 102a and the left device 102j can form surround or “satellite” channels of a home theater system. The bound playback devices 102a, 102b, 102j, and 102k can form a single Zone (D) ( FIG. 3A ).

[0100] In some implementations, playback devices can also be “merged.” In contrast to certain bound playback devices, merged playback devices can not have assigned playback responsibilities, but can each present the full range of audio content that each playback device is capable of playing back. However, merged devices can be represented as a single UI entity (i.e., a Zone, as described above). For example, FIG. 3E merging of playback devices 102d and 102m in the living room is shown, which will result in these devices being represented by a single UI entity of Zone C. In one embodiment, the playback devices 102d and 102m can playback audio in synchrony, during which each playback device outputs the full range of audio content that each respective playback device 102d and 102m is capable of presenting.

[0101] In some embodiments, standalone NMDs can themselves be in a zone. For example, NMD 103h from FIG. 1A is named “CLOSET” and forms Zone C. FIG. 3ANMDs can also be bound or merged with other devices to form zones. For example, an NMD device 103f named "Island" can be bound with playback device 102i Kitchen, both together forming zone F, which is also named "Kitchen." Additional details regarding the assignment of NMDs and playback devices as designated or default devices can be found, for example, in the previously cited U.S. Patent Application No. 15 / 438,749. In some embodiments, a standalone NMD can not be assigned to a zone.

[0102] Zones of single, bound, and / or merged devices can be arranged to form a group of playback devices that synchronize playback of audio. Such a group of playback devices can be referred to as a "group," "zone group," "synchronization group," or "playback group." In response to input provided through a controller device 104, playback devices can be dynamically grouped and ungrouped to form new or different groups that synchronize playback of audio content. For example, referring to FIG. 3A , zone A can be grouped with zone B to form a zone group that includes playback devices of both zones. As another example, zone A can be grouped with one or more other zones C-I. Zones A-I can be grouped and ungrouped in a variety of ways. For example, three, four, five, or more (e.g., all) of zones A-I can be grouped together. As described in the previously referenced U.S. Patent No. 8,234,395, when grouped together, zones of single and / or bound playback devices can synchronize playback of audio with one another. Grouping devices that are bound is an example type of association between portable and fixed playback devices that can be caused in response to a triggering event, as discussed above and in more detail below.

[0103] In various implementations, a zone in an environment can be assigned a particular name that can be a default name of a zone within a zone group or a combination of names of zones within a zone group, for example, "Dining Room + Kitchen" as shown in FIG. 3A In some embodiments, a zone group can also be named a unique name selected by a user, for example, "Nick's Room" as shown in FIG. 3A The name "Nick's Room" can be a name selected by a user over a previous name of the zone group, for example, the room name "Master Bedroom."

[0104] Referring again to FIG. 2A , certain data can be stored in memory 213 as one or more state variables that are periodically updated and used to describe the state of a playback zone, playback devices, and / or associated zone groups. Memory 213 can also include data associated with the state of other devices of MPS 100, which can be shared between devices from time to time so that one or more devices have the most up-to-date data associated with the system.

[0105] In some embodiments, the memory 213 of the playback device 102 can store instances of various variable types associated with a zone. The variable instances can be stored with an identifier (e.g., a tag) corresponding to the type. For example, certain identifiers can be a first type "al" for identifying a zone, a second type "bl" for identifying a playback device that can be bound in the zone, and a third type "cl" for identifying a zone group that the zone can belong to. As a related example, in FIG. 1A the identifier associated with the patio can indicate that the patio is the only playback device for a particular zone, and not in a zone group. The identifier associated with the living room can indicate that the living room is not grouped with other zones, but includes the bound playback devices 102a, 102b, 102j, and 102k. The identifier associated with the dining room can indicate that the dining room is part of a dining room + kitchen group, and that devices 103f and 102i are bound. Since the kitchen is part of the dining room + kitchen zone group, the identifier associated with the kitchen can indicate the same or similar information. Other example zone variables and identifiers are described below.

[0106] In yet another example, as shown in FIG. 3A the MPS 100 can include other associated variables or identifiers for zones and zone groups, such as an identifier associated with an area. An area can relate to a cluster of zone groups and / or zones that are not within a zone group. For example, FIG. 3A a first area named "first area" and a second area named "second area" are shown. The first area includes zones and zone groups for the patio, the study, the dining room, the kitchen, and the bathroom. The second area includes zones and zone groups for the bathroom, Nick's room, the bedroom, and the living room. In one aspect, an area can be used to invoke a cluster of zone groups and / or zones that share one or more zones and / or zone groups of another cluster. In this regard, such an area is different from a zone group that does not share a zone with another zone group. Other examples of techniques for implementing areas can be found in U.S. Application No. 15 / 682,506, filed August 21, 2017, entitled "Room Association Based on Name," and U.S. Patent No. 8,483,853, filed September 11, 2007, entitled "Controlling and manipulating groupings in a multi-zone media system." Each of these applications is incorporated by reference herein in its entirety. In some embodiments, the MPS 100 can not implement areas, in which case the system can not store variables associated with areas.

[0107] Memory 213 can also be configured to store other data. Such data can pertain to audio sources accessible to playback device 102 or playback queues with which the playback device (or some other playback device) can be associated. In embodiments described below, memory 213 is configured to store a set of command data for selecting a particular VAS when processing a voice input. During operation, FIG. 1A One or more of the playback zones in the environment can each be playing different audio content. For example, a user can be grilling in the patio zone and listening to hip hop music being played by playback device 102c, while another user can be preparing food in the kitchen zone and listening to classical music being played by playback device 102i. In another example, a playback zone can be playing the same audio content in synchronization with another playback zone.

[0108] For example, a user can be in the office zone, where playback device 102n is playing the same music as playback device 102c in the patio zone, which is playing hip hop music. In this case, playback devices 102c and 102n can play the hip hop music in synchronization, such that the user can enjoy the audio content being played out-of-ear seamlessly (or at least substantially seamlessly) as the user moves between the different playback zones. As described previously in U.S. Patent No. 8,234,395, synchronization between playback zones can be achieved in a manner similar to the synchronization between playback devices.

[0109] As described above, the zone configuration of MPS 100 can be dynamically modified. Thus, MPS 100 can support a variety of configurations. For example, if a user physically moves one or more playback devices into or out of a zone, MPS 100 can be reconfigured to accommodate the change. For example, if a user physically moves playback device 102c from the patio zone to the office zone, the office zone can now include playback devices 102c and 102n. In some cases, the user can use, for example, one of controller devices 104 and / or voice input to pair or group the moved playback device 102c with the office zone and / or rename the player in the office zone. As another example, if one or more playback devices 102 are moved to a particular space in the home environment that is not yet a playback zone, the moved playback devices can be renamed or associated with the playback zone for the particular space.

[0110] Further, different playback zones of the MPS 100 can be dynamically combined into zone groups or divided into separate playback zones. For example, the dining room zone and the kitchen zone can be combined into a zone group for a dinner party, such that the playback devices 102i and 1021 can synchronously present audio content. As another example, the bound playback devices in the study zone can be divided into (i) a television zone and (ii) a separate listening zone. The television zone can include the front playback device 102b. The listening zone can include the right playback device 102a, the left playback device 102j, and the subwoofer (SUB) playback device 102k, which can be combined, paired, or merged as described above. Dividing the study zone in this way can allow one user to listen to music in the listening zone in one area of the living room space while another user watches television in another area of the living room space. In a related example, a user can control the study zone with either of the NMDs 103a or 103b FIG. 1B ) before dividing the study zone into the television zone and the listening zone. Once divided, the listening zone can be controlled, for example, by a user in proximity to the NMD 103a, and the television zone can be controlled, for example, by a user in proximity to the NMD 103b. However, as described above, any NMD 103 can be configured to control various playback devices and other devices of the MPS 100.

[0111] c. Example Controller Devices

[0112] FIG. 4 is a functional block diagram illustrating certain aspects of one of the selected controller devices 104 of the MPS 100 of FIG. 1A Such a controller device can also be referred to herein as a "control device" or a "controller." FIG. 4 The controller device shown in TM , iPad TM , or any other smartphone, tablet, or network device (e.g., a network computer (e.g., a PC or a Mac TM ) on which a media playback system controller application can be installed.

[0113] The memory 413 of the controller device 104 can be configured to store controller application software and other data associated with the MPS 100 and / or users of the system 100. The memory 413 can be loaded with instructions in the software 414 that are executable by the processor 412 to implement certain functionality, e.g., to facilitate user access, control, and / or configuration of the MPS 100. As noted above, the controller device 104 is configured to communicate with other network devices over the network interface 424, which can take the form of a wireless interface.

[0114] In one example, system information (e.g., state variables) can be communicated between the controller device 104 and other devices over the network interface 424. For example, the controller device 104 can receive playback zone and zone group configurations in the MPS 100 from a playback device, NMD, or another network device. Similarly, the controller device 104 can send such system information to a playback device or another network device over the network interface 424. In some cases, the other network device can be another controller device.

[0115] The controller device 104 can also communicate playback device control commands, e.g., volume control and audio playback control, to playback devices via the network interface 424. As noted above, configuration changes to the MPS 100 can also be performed by a user using the controller device 104. Configuration changes can include adding / removing one or more playback devices from a zone, adding / removing one or more zones from a zone group, forming a bound or merged player, detaching one or more playback devices from a bound or merged player, etc.

[0116] As shown in FIG. 4 , the controller device 104 also includes a user interface 440 that is generally configured to facilitate user access and control of the MPS 100. The user interface 440 can include a touchscreen display or other physical interface, e.g., the controller interfaces 540a and 540b shown in FIG. 5A and 5B , that is configured to provide various graphical controller interfaces. Referring to FIG. 5A and FIG. 5B together, the controller interfaces 540a and 540b include a playback control region 542, a playback zone region 543, a playback status region 544, a playback queue region 546, and a source region 548. The user interfaces shown are merely examples of the types of interfaces that can be presented on network devices (e.g., the controller device 104, playback devices, NMDs, etc.) in the MPS 100. FIG. 4One example of an interface provided on a controller device (e.g., the controller device shown) and accessed by a user to control a media playback system (e.g., the MPS 100) is shown. Alternatively, other user interfaces of varying formats, styles, and interaction sequences can be implemented on one or more network devices to provide similar control access to a media playback system.

[0117] The playback control region 542( FIG. 5A ) can include selectable icons (e.g., by touch or by using a cursor) that, when selected, cause the playback devices in the selected playback zone or zone group to play or pause, fast forward, rewind, skip to next, skip to previous, enter / exit a random play mode, enter / exit a repeat mode, enter / exit a crossfade mode, etc. The playback control region 542 can also include selectable icons that, when selected, modify equalization settings, playback volume, etc.

[0118] The playback zone region 543( FIG. 5B ) can include representations of playback zones within the MPS 100. As shown, the playback zone region 543 can also include representations of zone groups, e.g., the Dining Room + Kitchen zone group.

[0119] In some embodiments, the graphical representations of playback zones can be selectable to bring up additional selectable icons to manage or configure the playback zones in the MPS 100, e.g., creation of a bound zone, creation of a zone group, detachment of a zone group, and renaming of a zone group, etc.

[0120] For example, as shown, a "group" icon can be provided within each graphical representation of a playback zone. The "group" icon provided within the graphical representation of a particular zone can be selectable to bring up options for selecting one or more other zones in the MPS 100 to group with the particular zone. Once grouped, the playback devices in the zones that have been grouped with the particular zone will be configured to play audio content in synchronization with the playback devices in the particular zone. Similarly, a "group" icon can be provided within the graphical representation of a zone group. In this case, the "group" icon can be selectable to bring up options for deselecting one or more zones in the zone group to remove from the zone group. Other interactions and implementations of grouping and ungrouping zones through the user interface are also possible. The representations of playback zones in the playback zone region 543( FIG. 5B ) can be dynamically updated when playback zone or zone group configurations are modified.

[0121] The playback status region 544( FIG. 5A) can include graphical representations of audio content that is currently playing, previously played, or scheduled to play next in the selected playback zone or zone group. The selected playback zone or zone group can be visually distinguished on the user interface, for example, within the playback zone area 543 and / or the playback status area 544. The graphical representations can include track titles, artist names, album names, album years, track lengths, and / or other relevant information that can be useful to the user when controlling the MPS 100 through the controller interface.

[0122] The playback queue area 546 can include graphical representations of audio content in a playback queue associated with the selected playback zone or zone group. In some embodiments, each playback zone or zone group can be associated with a playback queue that includes information corresponding to zero or more audio items played back by the playback zone or zone group. For example, each audio item in the playback queue can include a uniform resource identifier (URI), a uniform resource locator (URL), or some other identifier that can be used by a playback device in the playback zone or zone group to look up and retrieve the audio item from a local audio content source or a network audio content source, which can then be played back by the playback device.

[0123] In one example, a playlist can be added to the playback queue, in which case information corresponding to each audio item in the playlist can be added to the playback queue. In another example, the audio items in the playback queue can be saved as a playlist. In another example, the playback queue can be empty or populated but "unused" when the playback zone or zone group is playing continuous streaming audio content (e.g., an internet radio, which can play continuously until stopped) rather than discrete audio items with playback durations. In alternative embodiments, the playback queue can include internet radio and / or other streaming audio content items and be "in use" when the playback zone or zone group is playing these items. Other examples are possible as well.

[0124] When a playback zone or zone group is "grouped" or "ungrouped," the playback queue associated with the affected playback zone or zone group can be cleared or reassociated. For example, if a first playback zone including a first playback queue is grouped with a second playback zone including a second playback queue, the resulting zone group can have an associated playback queue that is initially empty, contains audio items from the first playback queue (e.g., if the second playback zone is added to the first playback zone), or contains audio items from the second playback queue (e.g., if the first playback zone is added to the second playback zone), or contains a combination of audio items from both the first playback queue and the second playback queue. Subsequently, if the resulting zone group is ungrouped, the resulting first playback zone can be reassociated with the previous first playback queue, or associated with a new playback queue that is empty, or contains audio items from the playback queue associated with the resulting zone group prior to the resulting zone group being ungrouped. Similarly, the resulting second playback zone can be reassociated with the previous second playback queue, or associated with a new playback queue that is empty, or contains audio items from the playback queue associated with the resulting zone group prior to the resulting zone group being ungrouped. Other examples are possible.

[0125] Still referring to FIG. 5A and 5B , the graphical representation of the audio content in the playback queue area 646 FIG. 5A may include a track title, an artist name, a track length, and other relevant information associated with the audio content in the playback queue. In one example, the graphical representation of the audio content can be selectable to bring up additional selectable icons to manage and / or manipulate the playback queue and / or the audio content represented in the playback queue. For example, the represented audio content can be removed from the playback queue, moved to a different position within the playback queue, or selected to be played immediately, or played after any currently playing audio content, etc. The playback queue associated with a playback zone or zone group can be stored in memory on one or more playback devices in the playback zone or zone group, on playback devices not in the playback zone or zone group, and / or on some other designated device. Playback of such a playback queue can involve one or more playback devices playing back the media items in the queue in a sequential or random order.

[0126] The source area 548 can include graphical representations of selectable audio content sources and / or selectable voice assistants associated with corresponding VASs. VASs can be selectively assigned. In some examples, the same NMD can invoke multiple VASs, e.g., AMAZON’s Alexa, MICROSOFT’s Cortana, etc. In some embodiments, a user can assign a VAS exclusively to one or more NMDs. For example, a user can assign a first VAS to one or both of the NMDs 102a and 102b in the living room shown, and assign a second VAS to the NMD 103f in the kitchen. Other examples are possible. FIG. 1A

[0127] d. Example Audio Content Sources

[0128] The audio sources in the source area 548 can be audio content sources from which audio content can be retrieved and played back by a selected playback zone or zone group. One or more playback devices in a zone or zone group can be configured to retrieve playback audio content from various available audio content sources (e.g., according to a corresponding URI or URL of the audio content). In one example, a playback device can retrieve audio content directly from a corresponding audio content source (e.g., through a line-in connection). In another example, audio content can be provided to a playback device over a network, via one or more other playback devices or network devices. As described in more detail below, in some embodiments, audio content can be provided by one or more media content services.

[0129] Example audio content sources can include a memory of one or more playback devices in a media playback system (e.g., the MPS 100 of FIG. 1A ), a local music library on one or more network devices (e.g., a controller device, a network-enabled personal computer, or a network-attached storage (“NAS”)), a streaming audio service that provides audio content over the Internet (e.g., a cloud-based music service), or an audio source connected to the media playback system through a line-in connection on a playback device or network device, etc.

[0130] In some embodiments, audio content can be retrieved from a media playback system (e.g., the MPS 100 of FIG. 1A ​add or remove audio content sources in the MPS 100. In one example, indexing of audio items can be performed each time one or more audio content sources are added, removed, or updated. Indexing of audio items can include scanning all folders / directories shared on a network accessible by playback devices in the media playback system for identifiable audio items and generating or updating an audio content database that includes metadata (e.g., title, artist, album, track length, etc.) and other associated information (e.g., URI or URL for each identifiable audio item found). Other examples for managing and maintaining audio content sources are also possible.

[0131] FIG. 6 is a message flow diagram illustrating data exchange between devices of the MPS 100. At step 650a, the MPS 100 receives, via the control device 130a, an indication of selected media content (e.g., one or more songs, albums, playlists, podcasts, videos, radio stations). The selected media content can include, for example, media items stored locally on one or more devices connected to the media playback system (e.g., the audio source 105) and / or media items stored on one or more media service servers (e.g., one or more remote computing devices 106a-106c) accessible by the media playback system. In response to receiving the indication of the selected media content, the control device 130a sends a message 651a to the playback device 110a to add the selected media content to a playback queue on the playback device 110a. FIG. 1B FIG. 1A-1B

[0132] At step 650b, the playback device 110a receives the message 651a and adds the selected media content to the playback queue for playback.

[0133] At step 650c, the control device 130a receives input corresponding to a command to play back the selected media content. In response to receiving the input corresponding to the command to play back the selected media content, the control device 130a sends a message 651b to the playback device 110a to cause the playback device 110a to play back the selected media content. In response to receiving the message 651b, the playback device 110a sends a message 651c to the computing device 106a to request the selected media content. In response to receiving the message 651c, the computing device 106a sends a message 651d that includes data (e.g., audio data, video data, URL, URI) corresponding to the requested media content.

[0134] At step 650d, the playback device 110a receives the message 651d with data corresponding to the requested media content and plays back the associated media content.

[0135] ​​At step 650e, playback device 110a optionally causes one or more other devices to playback the selected media content. In one example, playback device 110a is one of a binding zone of two or more players. Playback device 110a can receive the selected media content and send all or a portion of the media content to the other devices in the binding zone. In another example, playback device 110a is a coordinator of a group and is configured to send and receive timing information from one or more other devices in the group. The other one or more devices in the group can receive the selected media content from computing device 106a and start playback of the selected media content in response to a message from playback device 110a so that all devices in the group playback the selected media content in synchronization.

[0136] III. Example Command Keyword Events

[0137] FIG. 7A And FIG. 7B are functional block diagrams illustrating aspects of NMD 703a and NMD 703b configured in accordance with embodiments of the present disclosure. NMD 703a and NMD 703b are collectively referred to as NMD 703. NMD 703 can generally be similar to NMD 103 and include similar components. As described in greater detail below, NMD 703a FIG. 7A is configured to locally process certain voice inputs without having to send data representative of the voice inputs to a voice assistant service. However, NMD 703a is also configured to process other voice inputs using a voice assistant service. NMD 703b FIG. 7B is configured to process voice inputs using a voice assistant service and can have limited or no local NLU or command keyword detection.

[0138] Referring to FIG. 7A , NMD 703 includes a voice capture component (“VCC”) 760, a VAS wake word engine 770a, and a speech extractor 773. VAS wake word engine 770a and speech extractor 773 are operatively coupled to VCC 760. NMD 703a also includes a command keyword engine 771a that is operatively coupled to VCC 760.

[0139] NMD 703 also includes microphone 720 and at least one network interface 724 described above, and can also include other components, e.g., audio amplifiers, speakers, user interfaces, etc., that are not shown in FIG. 7A for the sake of clarity. Microphone 720 of NMD 703a is configured to provide detected sounds S D to VCC 760 from an environment of NMD 703. Detected sounds S Dmay take the form of one or more analog or digital signals. In example implementations, the detected sound S D may be composed of a plurality of signals associated with respective channels 762 fed to the VCC 760.

[0140] Each channel 762 can correspond to a particular microphone 720. For example, an NMD having six microphones can have six corresponding channels. The detected sound S D Each channel of the detected sound S D may have certain similarities to other channels, but can differ in certain respects, which can be due to the position of the respective microphone of a given channel relative to the microphones of other channels. For example, one or more channels of the detected sound S D may have a greater signal-to-noise ratio (“SNR”) of speech to background noise than other channels.

[0141] As FIG. 7A further shown, the VCC 760 includes an AEC 763, a spatial processor 764, and one or more buffers 768. In operation, the AEC 763 receives the detected sound S D and filters or otherwise processes the sound to suppress echoes and / or improve the quality of the detected sound S D . The processed sound can then be passed to the spatial processor 764.

[0142] The spatial processor 764 is generally configured to analyze the detected sound S D and identify certain characteristics, e.g., the amplitude (e.g., decibel level), frequency spectrum, directionality, etc. of the sound. In one aspect, as noted above, the spatial processor 764 can help filter or suppress ambient noise in the detected sound S D from potential user speech based on similarities and differences of the constituent channels 762 of the detected sound S D . As one possibility, the spatial processor 764 can monitor a metric that distinguishes speech from other sounds. For example, such a metric can include the energy within a speech band relative to background noise and the entropy (a measure of spectral structure) within that speech band, which is typically lower than most common background noise. In some implementations, the spatial processor 764 can be configured to determine a speech presence probability, an example of which functionality is disclosed in U.S. Patent Application No. 15 / 984,073, filed May 18, 2018, entitled “Linear Filtering for Noise-Suppressed Speech Detection,” the entirety of which is incorporated by reference herein.

[0143] In operation, one or more buffers 768 (one or more of which can be part of or separate from memory 213 FIG. 2A ) capture data corresponding to detected sound S D . More specifically, one or more buffers 768 capture detected sound data processed by upstream AEC 764 and spatial processor 766.

[0144] Network interface 724 can then provide this information to a remote server that can be associated with MPS 100. In one aspect, the information stored in additional buffer 769 does not reveal the content of any voice, but rather indicates certain unique characteristics of the detected sound itself. In a related aspect, the information can be transmitted between computing devices, such as various computing devices of MPS 100, without necessarily implicating privacy concerns. In practice, MPS 100 can use this information to adjust and fine-tune speech processing algorithms, including sensitivity tuning as described below. In some implementations, the additional buffer can contain or include functionality similar to a lookback buffer as disclosed, for example, in U.S. Patent Application No. 15 / 989,715, filed May 25, 2018, entitled “Determining and Adapting to Changes in Microphone Performance of Playback Devices”; U.S. Patent Application No. 16 / 141,875, filed September 25, 2018, entitled “Voice Detection Optimization Based on Selected Voice Assistant Service”; and U.S. Patent Application No. 16 / 138,111, filed September 21, 2018, entitled “Voice Detection Optimization Using Sound Metadata”, which are incorporated by reference herein in their entireties.

[0145] In any event, the detected sound data forms a digital representation of the sound detected by microphone 720 (i.e., a sound data stream) S DS . In practice, sound data stream S DS may take a variety of forms. As one possibility, sound data stream S DS may be comprised of frames, each of which can include one or more sound samples. Frames can be streamed (i.e., read out) from one or more buffers 768 for further processing by downstream components (e.g., VAS wake-word engines 770a-770b and speech extractor 773 of NMD 703).

[0146] In some implementations, the at least one buffer 768 utilizes a sliding window approach to capture detected sound data, where a given amount (i.e., a given window) of the most recently captured detected sound data is retained in the at least one buffer 768, while older detected sound data falls outside the window, they are overwritten. For example, the at least one buffer 768 can temporarily retain a frame of 20 sound samples at a given time, discard the oldest frame after an expiration time, and then capture a new frame, which is added to the 19 previous frames of sound samples.

[0147] In practice, when the sound data stream S DS When composed of frames, the frames can take various forms with various properties. As one possibility, the frames can take the form of audio frames with a certain resolution (e.g., 16-bit resolution), which can be based on a sampling rate (e.g., 44, 100 Hz). Additionally or alternatively, the frames can include information corresponding to a given sound sample defined by the frames, such as metadata indicating a frequency response, a power input level, an SNR, a microphone channel identification, and / or other information for a given sound sample, among other examples. Thus, in some embodiments, a frame can include a portion of sound (e.g., one or more samples of a given sound sample) and metadata about the portion of sound. In other embodiments, a frame can include only a portion of sound (e.g., one or more samples of a given sound sample) or metadata about the portion of sound.

[0148] In any case, downstream components of the NMD 703 can process the sound data stream S DS . For example, the VAS wake-word engines 770a-770b can be configured to apply one or more recognition algorithms to the sound data stream S DS (e.g., the stream sound frames) to find potential wake-words in the detected sound S D . This process can be referred to as automatic speech recognition. The VAS wake-word engines 770a and the command keyword engine 771a apply different recognition algorithms corresponding to their respective wake-words and also generate different events based on detecting a wake-word in the detected sound S D .

[0149] Example wake-word detection algorithms take audio as input and provide an indication of whether a wake-word is present in the audio. Many first-party and third-party wake-word detection algorithms are known and commercially available. For example, a voice service operator can make its algorithm available to third-party devices. Alternatively, an algorithm can be trained to detect certain wake-words.

[0150] For example, when the VAS wake-word engine 770a detects a potential VAS wake- word, the VAS wake-word engine 770a provides an indication of a "VAS wake-word event" (also referred to as a "VAS wake-word trigger"). In the illustrated example of FIG. 7, the VAS wake-word engine 770a outputs a signal S FIG. 7A indicating that a VAS wake-word event has occurred to the speech extractor 773. VW .

[0151] In a multi-VAS implementation, the NMD 703 can include a VAS selector 774 (shown in dashed lines) that is generally configured to extract directly from the speech extractor 773, and transmit the sound data stream S DS to the appropriate VAS when a given wake-word is recognized by a particular wake-word engine (and corresponding wake-word trigger), e.g., the VAS wake-word engine 770a and at least one additional VAS wake-word engine 770b (shown in dashed lines).

[0152] Similar to the above discussion, each VAS wake-word engine 770a-770b can be configured to receive a sound data stream S DS as input from one or more buffers 768, and apply a recognition algorithm to cause a wake-word trigger for the appropriate VAS. Thus, as one example, the VAS wake-word engine 770a can be configured to recognize the wake-word "Alexa," and cause the NMD 703a to invoke the AMAZON VAS when "Alexa" is found. As another example, the wake-word engine 770b can be configured to recognize the wake-word "Ok, Google," and cause the NMD 703b to invoke the GOOGLE VAS when "Ok, Google" is found. In a single-VAS implementation, the VAS selector 774 can be omitted.

[0153] In response to a VAS wake-word event (e.g., in response to a signal S VW indicating a wake-word event), the speech extractor 773 is configured to receive and format (e.g., encapsulate) the sound data stream S DS . For example, the speech extractor 773 encapsulates frames of the sound data stream S DS into messages. The speech extractor 773 sends or streams these messages M V containing potentially real-time or near real-time speech input to the remote VAS via the network interface 724.

[0154] The VAS is configured to process the sound data stream S V contained in the messages MDS More specifically, the NMD 703a is configured to identify the voice input 780 based on the sound data stream S DS As described in connection with FIG. 2C the voice input 780 can include a keyword portion and an utterance portion. The keyword portion corresponds to the detected sound that caused the wake-word event, or a command keyword event when one or more particular conditions are met (e.g., particular playback conditions). For example, when the voice input 780 includes a VAS wake-word, the keyword portion corresponds to the detected sound that caused the wake-word engine 770a to output a wake-word event signal SVW to the voice extractor 773. In this case, the utterance portion corresponds to the detected sound that potentially includes a user request following the keyword portion.

[0155] When a VAS wake-word event occurs, the VAS can first process the keyword portion within the sound data stream S DS to verify the presence of the VAS wake-word. In some instances, the VAS can determine that the keyword portion includes a false wake-word (e.g., the word “Election” when the word “Alexa” is the target VAS wake-word). In this case, the VAS can send a response to the NMD 703a instructing the NMD 703a to stop extracting sound data, which causes the voice extractor 773 to stop further streaming of the detected sound data to the VAS. The wake-word engine 770a can resume or continue monitoring sound samples until it finds another potential VAS wake-word that causes another VAS wake-word event. In some implementations, the VAS does not process or receive the keyword portion, but only processes the utterance portion.

[0156] In any case, the VAS processes the utterance portion to identify the presence of any words in the detected sound data and determine potential intents from those words. The words can correspond to one or more commands, as well as certain keywords. Keywords can be, for example, words in the voice input that identify a particular device or grouping in the MPS 100. For example, in the illustrated example, the keywords can be one or more words that identify one or more zones (e.g., the living room and dining room FIG. 1A ) in which music is to be played.

[0157] To determine the intent of the words, the VAS typically communicates with one or more databases associated with the VAS (not shown) and / or one or more databases (not shown) of the MPS 100. Such databases can store a variety of user data, analytics, directories, and other information for natural language processing and / or other processing. In some implementations, such databases can be updated based on speech input processing for adaptive learning and feedback of neural networks. In some cases, the utterance portion can include additional information, such as detected pauses (e.g., periods of non-speech) between words spoken by the user, as shown in FIG. 2C The pauses can demarcate the location of individual commands, keywords, or other information spoken by the user within the utterance portion.

[0158] After processing the speech input, the VAS can send a response with instructions to the MPS 100 to perform one or more actions based on its determined intent from the speech input. For example, based on the speech input, the VAS can instruct the MPS 100 to initiate playback on one or more playback devices 102, control one or more of these playback devices 102 (e.g., raise / lower volume, group / un-group devices, etc.), or turn on / off certain smart devices, among other actions. As discussed above, upon receiving the response from the VAS, the wake-word engine 770a of the NMD 703 can resume or continue monitoring the sound data stream S DS1 until it finds another potential wake-word.

[0159] Generally, the one or more recognition algorithms applied by a particular VAS wake-word engine (e.g., the VAS wake-word engine 770a) are configured to analyze certain characteristics of the detected sound stream S DS and compare these characteristics to corresponding characteristics of one or more particular VAS wake-words of the particular VAS wake-word engine. For example, the wake-word engine 770a can apply one or more recognition algorithms to find spectral characteristics in the detected sound stream S DS that match spectral characteristics of one or more wake-words of the engine, thereby determining that the detected sound S D includes a speech input containing a particular VAS wake-word.

[0160] In some implementations, the one or more recognition algorithms can be third-party recognition algorithms (i.e., recognition algorithms developed by a company other than the company that provides the NMD 703a). For example, the operators of a voice service (e.g., AMAZON) can make their respective algorithms (e.g., recognition algorithms corresponding to AMAZON’s ALEXA) available to third-party devices (e.g., NMD 103), which are then trained to recognize one or more wake words for a particular voice assistant service. Additionally or alternatively, the one or more recognition algorithms can be first-party recognition algorithms that are developed and trained to recognize certain wake words that are not necessarily specific to a given voice service. Other possibilities exist as well.

[0161] As noted above, the NMD 703a also includes a command keyword engine 771a in parallel with the VAS wake word engine 770a. Like the VAS wake word engine 770a, the command keyword engine 771a can apply one or more recognition algorithms corresponding to one or more wake words. When a particular command keyword is recognized in the detected sound S D In contrast to random words that are typically used as VAS wake words, command keywords serve as both an activation word and as a command itself. For example, example command keywords can correspond to playback commands (e.g., “play,” “pause,” “skip,” etc.) as well as control commands (“turn on”), among others. Under the appropriate conditions, based on detecting one of these command keywords, the NMD 703a executes the corresponding command.

[0162] The command keyword engine 771a can use an automatic speech recognizer 772. The ASR 772 is configured to output phonetic or phenetic representations, e.g., text corresponding to words, based on sounds in the sound data stream S DS The ASR 772 can transcribe spoken words represented in the sound data stream S DS into one or more strings of text representing the voice input 780. The command keyword engine 771 can feed the ASR output (labeled S ASR ) to a local natural language unit (NLU) 779 that recognizes particular keywords as command keywords for invoking command keyword events, as described below.

[0163] As noted above, in some example implementations, the NMD 703a is configured to perform natural language processing, which can be performed using an on-board natural language processor, referred to herein as a natural language unit (NLU) 779. The local NLU 779 is configured to analyze the text output of the ASR 772 of the command keyword engine 771a to discover (i.e., detect or recognize) keywords in the voice input 780. In some implementations, the NLU 779 is configured to recognize command keywords that are not necessarily specific to a given voice service.FIG. 7A In particular, the output is shown as a signal S ASR The local NLU 779 includes a library of keywords (i.e., words and phrases) that correspond to respective commands and / or parameters.

[0164] In one aspect, the library of the local NLU 779 includes command keywords. When the local NLU 779 recognizes a command keyword in the signal S ASR , the command keyword engine 771a generates a command keyword event and executes a command corresponding to the command keyword in the signal S ASR , assuming that one or more conditions corresponding to the command keyword are satisfied.

[0165] In addition, the library of the local NLU 779 can also include keywords that correspond to parameters. The local NLU 779 can then determine a potential intent from matching keywords in the voice input 780. For example, if the local NLU matches the keywords “David Bowie” and “kitchen” in conjunction with a play command, the local NLU 779 can determine an intent to play David Bowie in the kitchen 101h on the playback device 102i. The local processing of the voice input 780 by the local NLU 779 can be relatively less complex compared to the processing of the voice input 780 by the cloud-based VAS, as the NLU 779 does not have access to the relatively greater processing power and larger voice database that the VAS typically has access to.

[0166] In some examples, the local NLU 779 can determine an intent with one or more slots that correspond to respective keywords. For example, returning to the play David Bowie in the kitchen example, when processing the voice input, the local NLU 779 can determine that the intent is to play music (e.g., intent = play music), while a first slot includes David Bowie as the target content (e.g., slot 1 = David Bowie) and a second slot includes the kitchen 101h as the target playback device (e.g., slot 2 = kitchen). Here, the intent (“play music”) is based on a command keyword, and the slots are parameters that modify the intent to specific target content and playback device.

[0167] In examples, the command keyword engine 771a outputs a signal S CW indicating the occurrence of a command keyword event to the local NLU 779. In response to the command keyword event (e.g., in response to the signal S CW indicating the command keyword event), the local NLU 779 is configured to receive and process a signal S ASR . In particular, the local NLU 779 looks at the words in the signal S ASR to find keywords that match keywords in the library of the local NLU 779.

[0168] Some errors are expected in performing local automatic speech recognition. In an example, the ASR 772 can generate a confidence score when transcribing a spoken word into text, the confidence score indicating how well the spoken word in the speech input 780 matches the sound pattern of the word. In some implementations, generating a command keyword event is based on the confidence score of a given command keyword. For example, the command keyword engine 771a can generate a command keyword event when the confidence score of a given sound exceeds a given threshold (e.g., 0.5 within a 0-1 scale, indicating that the given sound is more likely to be a command keyword). Conversely, the command keyword engine 771a does not generate a command keyword event when the confidence score of a given sound is equal to or below a given threshold.

[0169] Similarly, some errors are expected in performing keyword matching. In an example, the local NLU can generate a confidence score when determining an intent, the confidence score indicating how well the transcribed words in the signal S ASR are matched to corresponding keywords in the local NLU’s library. In some implementations, performing an operation according to the determined intent is based on the confidence score of the keywords matched in the signal S ASR . For example, the NMD 703 can perform an operation according to the determined intent when the confidence score of a given sound exceeds a given threshold (e.g., 0.5 within a 0-1 scale, indicating that the given sound is more likely to be a command keyword). Conversely, the NMD 703 does not perform an operation according to the determined intent when the confidence score of a given sound is equal to or below a given threshold.

[0170] As mentioned above, in some implementations, phrases can be used as command keywords, which provide additional syllables to match (or not match). For example, the phrase “give me some music” has more syllables than “play,” which provides additional sound patterns to match the word. Thus, command keywords that are phrases can generally be less susceptible to false wake-up words.

[0171] As mentioned above, the NMD 703a can only generate a command keyword event (and perform a command corresponding to the detected command keyword) if certain conditions corresponding to the detected command keyword are satisfied. These conditions are intended to reduce the incidence of false command keyword events. For example, after detecting the command keyword “skip,” the NMD 703a only generates a command keyword event (and skips to the next track) if certain playback conditions are satisfied that indicate that a skip should be performed. These playback conditions can include, for example, (i) a first condition that a media item is being played back; (ii) a second condition that the queue is active; and (iii) a third condition that the queue includes a media item after the media item being played back. If any of these conditions are not satisfied, then a command keyword event is not generated (and the skip is not performed).

[0172] The NMD 703a includes one or more state machines 775a to facilitate determining whether appropriate conditions are satisfied. The state machines 775a transition between a first state and a second state based on whether one or more conditions corresponding to a detected command keyword are satisfied. Specifically, for a given command keyword corresponding to a particular command that requires one or more particular conditions, the state machine 775a transitions to the first state when the one or more particular conditions are satisfied and transitions to the second state when at least one of the one or more particular conditions is not satisfied.

[0173] In example implementations, the command conditions are based on states indicated in state variables. As described above, the devices of the MPS 100 can store state variables that describe states of the respective devices. For example, the playback device 102 can store state variables that indicate states of the playback device 102, e.g., the audio content that is currently playing (or paused), the volume level, the network connection status, etc. These state variables are updated (e.g., periodically, or based on events (i.e., when a state in the state variable changes)) and the state variables can also be shared among the devices of the MPS 100, including the NMD 703.

[0174] Similarly, the NMD 703 can maintain these state variables (either by being implemented in the playback device or as a standalone NMD). The state machines 775a monitor the states indicated in these state variables and determine whether the states indicated in the appropriate state variables indicate that the command conditions are satisfied. Based on these determinations, the state machines 775a transition between the first state and the second state, as described above.

[0175] In some implementations, the command keyword engine 771 can be disabled unless certain conditions have been satisfied via the state machine. For example, the first state and the second state of the state machine 775a can operate as an on / off switch to the command keyword engine 771a corresponding to a particular command keyword. Specifically, when the state machine 775a corresponding to a particular command keyword is in the first state, the state machine 775a enables the command keyword engine 771a for the particular command keyword. Conversely, when the state machine 775a corresponding to a particular command keyword is in the second state, the state machine 775a disables the command keyword engine 771a for the particular command keyword. Thus, the disabled command keyword engine 771a stops analyzing the sound data stream S DSIn such cases where the at least one command condition is not satisfied, NMD 703a can suppress generation of a command keyword event when the command keyword engine 771a detects a command keyword. Suppressing generation can involve gating, blocking, or otherwise preventing output from the command keyword engine 771a from generating a command keyword event. Alternatively, suppressing generation can involve the NMD 703 stopping the feeding of the sound data stream S DS to the ASR 772. This suppression prevents the execution of a command corresponding to the detected command keyword when the at least one command condition is not satisfied. In such embodiments, the command keyword engine 771a can continue to analyze the sound data stream S DS but the command keyword event is disabled.

[0176] Other example conditions can be based on the output of a voice activity detector (“VAD”) 765. The VAD 765 is configured to detect the presence (or absence) of voice activity in the sound data stream S DS corresponding to the pre-roll portion of the voice input 780. In particular, the VAD 765 can analyze frames corresponding to the pre-roll portion of the voice input 780 with one or more voice detection algorithms to determine whether voice activity is present in the environment for some window of time prior to the keyword portion of the voice input 780. FIG. 2D

[0177] The VAD 765 can utilize any suitable voice activity detection algorithm. Example voice detection algorithms involve determining whether a given frame includes one or more features or qualities corresponding to voice activity, and further determining whether these features or qualities are separated from noise by a given degree (e.g., if a value exceeds a threshold for a given frame). Some example voice detection algorithms involve filtering or otherwise reducing noise in a frame prior to identifying the features or qualities.

[0178] In some examples, the VAD 765 can determine whether voice activity is present in the environment based on one or more metrics. For example, the VAD 765 can be configured to distinguish between frames that include voice activity and frames that do not include voice activity. The VAD-determined frames that have voice activity can be caused by speech, whether it is near-field or far-field. In this example and others, the VAD 765 can determine a count of frames that indicate voice activity in the pre-roll portion of the voice input 780. If this count exceeds a threshold percentage or number of frames, the VAD 765 can be configured to output a signal or set a state variable to indicate that voice activity is present in the environment. Other metrics can also be used in addition to, or as an alternative to, this count.

[0179] ​The presence of voice activity in the environment can indicate that voice input is directed to the NMD 73. Thus, when the VAD 765 indicates that there is no voice activity in the environment (as indicated by a state variable set by the VAD 765), this can be configured as one of the command conditions for a command keyword. When this condition is met (i.e., the VAD 765 indicates that there is voice activity in the environment), the state machine 775a will transition to a first state to enable execution of a command based on a command keyword, so long as any other conditions for the particular command keyword are met.

[0180] Further, in some implementations, the NMD 703 can include a noise classifier 766. The noise classifier 766 is configured to determine sound metadata (frequency response, signal level, etc.) and identify signatures in the sound metadata that correspond to various noise sources. The noise classifier 766 can include a neural network or other mathematical model configured to identify different types of noise in detected sound data or metadata. One classification of noise can be speech (e.g., far-field speech). Another classification can be a particular type of speech (e.g., background speech), and reference is made to FIG. 8 Examples of which are described in more detail. Background speech can be distinguished from other types of speech-like activity (e.g., more general voice activity (e.g., cadence, pauses, or other characteristics) of speech-like activity detected by the VAD 765).

[0181] For example, analyzing the sound metadata can include comparing one or more features of the sound metadata to known noise reference values, or comparing the sample population data to known noise. For example, any feature of the sound metadata (e.g., signal level, frequency response spectrum, etc.) can be compared to noise reference values or values collected and averaged over a sample population. In some examples, analyzing the sound metadata includes projecting the frequency response spectrum onto a feature space corresponding to aggregated frequency response spectra from a population of NMDs. Further, projecting the frequency response spectrum onto the feature space can be performed as a pre-processing step to facilitate downstream classification.

[0182] In various embodiments, any number of different techniques for classifying noise using sound metadata can be used, e.g., machine learning using decision trees, or Bayesian classifiers, neural networks, or any other classification technique. Alternatively or additionally, various clustering techniques can be used, e.g., K-Means clustering, Mean Shift clustering, Expectation-Maximization clustering, or any other suitable clustering technique. Techniques for classifying noise can include one or more techniques disclosed in U.S. Application No. 16 / 227,308, filed December 20, 2018, entitled “Optimization of Network Microphone Devices Using Noise Classification,” which is incorporated by reference herein in its entirety.

[0183] To illustrate, FIG. 8 First and second curves 882a and 882b are shown. The first and second curves 882a and 882b show analyzed sound metadata associated with background speech. These signatures shown in the curves are generated using Principal Component Analysis (PCA). The data collected from various NMDs provides an overall distribution of possible frequency response spectra. In general, principal component analysis can be used to find an orthonormal basis that describes the variance of all the field data. This feature space is reflected in the contours shown in the plots of FIG. 8 Each point in the curves represents a known noise value (e.g., a single frequency response spectrum from an NMD exposed to a specified noise source) projected onto the feature space. As FIG. 8 shown, these known noise values cluster together when projected onto the feature space. In this example, FIG. 8 The curves of

[0184] Referring back to FIG. 7A , in some implementations, an additional buffer 769 (shown in dashed lines) can store information (e.g., metadata, etc.) about the detected sound S D processed by the upstream AEC 763 and spatial processor 764. This additional buffer 769 can be referred to as a “sound metadata buffer.” Examples of such sound metadata include: (1) frequency response data, (2) echo return loss enhancement measurements, (3) speech direction measurements; (4) arbitration statistics; and / or (5) voice spectrum data. In example implementations, the noise classifier 766 can analyze the sound metadata in the buffer 769 to classify the noise in the detected sound S D .

[0185] As described above, one classification of sound can be background speech, e.g., speech indicative of far-field speech and / or speech indicative of a conversation not involving NMD 703. Noise classifier 766 can output a signal and / or set a state variable to indicate that background speech is present in the environment. The presence of speech activity (i.e., speech) in the pre-roll portion of speech input 780 indicates that speech input 780 can not be directed to NMD 703, but rather conversational speech within the environment. For example, a family member can say something like "Our kids should have a play date soon" without intending to direct the command keyword "play" to NMD 703.

[0186] Further, when noise classifier indicates that background speech is present in the environment, this condition can disable command keyword engine 771a. In some implementations, the condition that no background speech is present in the environment (possibly indicated by a state variable set by noise classifier 766) is configured as one of the command conditions for command keywords. Thus, when noise classifier 766 indicates that background speech is present in the environment, state machine 775a will not transition to the first state.

[0187] Further, noise classifier 766 can determine whether background speech is present in the environment based on one or more metrics. For example, noise classifier 766 can determine a count of frames in the pre-roll portion of speech input 780 that are indicative of background speech. If this count exceeds a threshold percentage or number of frames, noise classifier 766 can be configured to output a signal or set a state variable to indicate that background speech is present in the environment. In addition to, or as an alternative to, such a count, other metrics can also be used.

[0188] In example implementations, NMD 703a can support multiple command keywords. To facilitate such support, command keyword engine 771a can implement multiple recognition algorithms corresponding to respective command keywords. Alternatively, NMD 703a can implement an additional command keyword engine 771b configured to recognize respective command keywords. Further, the library of local NLU 779 can include multiple command keywords and be configured to search for text patterns corresponding to these command keywords in signal S ASR

[0189] ​Furthermore, command keywords may require different conditions. For example, the condition for "skip" may differ from the condition for "play," because "skip" may require the condition that a media item is being played, while play may require the opposite condition that no media item is being played. To facilitate these corresponding conditions, the NMD 703a may implement a corresponding state machine 775a for each command keyword. Alternatively, the NMD 703a may implement state machine 775a, which has a corresponding state for each command keyword. Other examples are also possible.

[0190] In some example implementations, the VAS wake word engine 770a generates a VAS wake word event when certain conditions are met. The NMD 703b includes a state machine 775b, which is similar to state machine 775a. State machine 775b transitions between a first state and a second state based on whether one or more conditions corresponding to the VAS wake word are met.

[0191] For example, in some examples, the VAS wake word engine 770a can generate a VAS wake word event only if there is no background speech in the environment before the VAS wake word event is detected. The indication of the presence of speech activity in the environment can come from the noise classifier 766. As described above, the noise classifier 766 can be configured to output a signal and / or set a state variable to indicate the presence of far-field speech in the environment. Furthermore, the VAS wake word engine 770a can generate a VAS wake word event only if there is speech activity in the environment. As described above, the VAD 765 can be configured to output a signal and / or set a state variable to indicate the presence of speech activity in the environment.

[0192] To illustrate, such as FIG. 7B As shown, the VAS wake word engine 770a is connected to the state machine 775b. The state machine 775b can remain in a first state when one or more conditions are met; these conditions may include the absence of voice activity in the environment. When the state machine 775b is in the first state, the VAS wake word engine 770a is enabled and a VAS wake word event is generated. If any of the one or more conditions are not met, the state machine 775b transitions to a second state, which disables the VAS wake word engine 770a.

[0193] Further, the NMD 703 can include one or more sensors that output signals indicative of whether one or more users are in proximity to the NMD 703. Example sensors include temperature sensors, infrared sensors, imaging sensors, and / or capacitive sensors, among others. The NMD 703 can use the output from these sensors to set one or more state variables to indicate whether one or more users are in proximity to the NMD 703. The state machine 775b can then use the presence or absence thereof as a condition for the state machine 775b. For example, when at least one user is in proximity to the NMD 703, the state machine 775b can enable the VAS wake word engine and / or the command keyword engine 771a.

[0194] To illustrate example state machine operations, FIG. 7C is a block diagram illustrating a state machine 775 for an example command keyword that requires one or more command conditions. At 777a, the state machine 775 remains in a first state 778a while all of the command conditions are satisfied. While the state machine 775 remains in the first state 778a (and all of the command conditions are satisfied), the NMD 703a will generate a command keyword event when the command keyword engine 771a detects a command keyword.

[0195] At 777b, the state machine 775 transitions to a second state 778b when any one of the command conditions is not satisfied. At 777c, the state machine 775 remains in the second state 778b when any one of the command conditions is not satisfied. While the state machine 775 remains in the second state 778b, the NMD 703a will not act on a command keyword event when the command keyword engine 771a detects a command keyword.

[0196] Referring back to FIG. 7A In some examples, one or more additional command keyword engines 771b can include a custom command keyword engine. A cloud service provider (e.g., a streaming audio service) can provide a custom keyword engine that is pre-configured with recognition algorithms configured to discover command keywords specific to the service. These service-specific command keywords can include commands to customize features of the service and / or custom names to access the service.

[0197] For example, the NMD 703a can include a specific streaming audio service (e.g., Apple Music) command keyword engine 771b. This specific command keyword engine 771b can be configured to detect command keywords specific to the specific streaming audio service and generate a streaming audio service wake word event. For example, one command keyword can be “Friends Mix,” which corresponds to a command to play back a custom playlist generated from the playback history of one or more “friends” within the specific streaming audio service.

[0198] Custom command keyword engines 771b can be relatively more susceptible to false wake words than VAS wake word engines 770a because custom command keyword engines 771b are typically less complex than VAS wake word engines 770a. To mitigate this, custom command keywords can need to satisfy one or more conditions before generating a custom command keyword event. Further, in some implementations, to reduce the incidence of false positives, multiple conditions can be imposed as a requirement for including a custom command keyword engine 771b in an NMD 703a.

[0199] These custom command keyword conditions can include service-specific conditions. For example, a command keyword corresponding to an advanced feature or a playlist can require a subscription as a condition. As another example, a custom command keyword corresponding to a particular streaming audio service can require a media item from that streaming audio service in the playback queue. Other conditions are also possible.

[0200] To gate the custom command keyword engine based on custom command keyword conditions, NMD 703a can include an additional state machine 775a corresponding to each custom command keyword. Alternatively, NMD 703a can implement a state machine 775a that has a respective state for each custom command keyword. Other examples are also possible. These custom command conditions can depend on state variables maintained by devices within MPS 100, and can also depend on state variables or other data structures representing the state of a user account for a cloud service (e.g., a streaming audio service).

[0201] FIG. 9A and FIG. 9B A table 985 is shown that illustrates example command keywords and corresponding conditions. As shown, example command keywords can include cognates that have similar intent and require similar conditions. For example, the “next” command keyword has cognates “skip” and “forward,” each of which invoke a skip command under the appropriate conditions. The conditions shown in table 985 are illustrative; different conditions can be used by various implementations.

[0202] Referring back to FIG. 7AIn example embodiments, the VAS wake-word engine 770a and the command keyword engine 771a can take a variety of forms. For example, the VAS wake-word engine 770a and the command keyword engine 771a can take the form of one or more modules stored in the memory (e.g., memory 112b) of the NMD 703a and / or the NMD 703b. As another example, the VAS wake-word engine 770a and the command keyword engine 771a can take the form of a general-purpose or special-purpose processor or module thereof. In this regard, the plurality of wake-word engines 770 and 771 can be part of the same component of the NMD 703a, or each wake-word engine 770 and 771 can take the form of a component that is specific to the particular wake-word engine. Other possibilities exist as well.

[0203] To further reduce false positives, the command keyword engine 771a can utilize a relatively low sensitivity as compared to the VAS wake-word engine 770a. In practice, the wake-word engines can include a modifiable sensitivity level setting. The sensitivity level can define a degree of similarity between the recognized words in the detected sound stream S DS1 and one or more particular wake-words of the wake-word engine that is considered a match (i.e., triggers a VAS wake-word or command keyword event). In other words, as one example, the sensitivity level defines to what degree the spectral characteristics in the detected sound stream S DS2 must match the spectral characteristics of one or more wake-words of the engine to be a wake-word trigger.

[0204] In this regard, the sensitivity level generally controls how many false positives the VAS wake-word engine 770a and the command keyword engine 771a identify. For example, if the VAS wake-word engine 770a is configured to recognize the wake-word "Alexa" with a relatively high sensitivity, then the false wake-words "Election" or "Lexus" can cause the wake-word engine 770a to flag the presence of the wake-word "Alexa." Conversely, if the command keyword engine 771a is configured to have a relatively low sensitivity, then the false wake-words "may" or "day" will not cause the command keyword engine 771a to flag the presence of the command keyword "play."

[0205] In practice, the sensitivity level can take a variety of forms. In example implementations, the sensitivity level takes the form of a confidence threshold that defines a minimum confidence (i.e., probability) level for the wake-word engine to use as a dividing line between triggering or not triggering a wake-word event when the wake-word engine is analyzing detected sounds for its particular wake-word. In this regard, a higher sensitivity level corresponds to a lower confidence threshold (and more false positives), while a lower sensitivity level corresponds to a higher confidence threshold (and fewer false positives). For example, lowering the confidence threshold of a wake-word engine configures it to trigger a wake-word event when it recognizes words that are less likely to be the actual particular wake-word, while raising the confidence threshold configures the engine to trigger a wake-word event when it recognizes words that are more likely to be the actual particular wake-word. In examples, the sensitivity level of the command keyword engine 771a can be based on more confidence scores, e.g., a confidence score when a command keyword is found and / or a confidence score when an intent is determined. Other examples of sensitivity levels are possible as well.

[0206] In example implementations, the sensitivity level parameters (e.g., sensitivity ranges) of a particular wake-word engine can be updated, which can occur in a variety of ways. As one possibility, the VAS or other third-party provider of a given wake-word engine can provide a wake-word engine update to the NMD 703 that modifies one or more sensitivity level parameters of the given VAS wake-word engine 770a. In contrast, the sensitivity level parameters of the command keyword engine 771a can be configured by the manufacturer of the NMD 703a or another cloud service (e.g., for customizing the wake-word engine 771b).

[0207] Notably, in certain examples, when processing the speech input 780 that includes a command keyword, the NMD 703a forgoes sending any data (e.g., the message M D ) representing the detected sound S V to the VAS. In implementations that include the local NLU 779, the NMD 703a can also process the speech utterance portion of the speech input 780 (in addition to the keyword portion) without having to send the speech utterance portion of the speech input 780 to the VAS. Thus, speaking the speech input 780 (with the command keyword) to the NMD 703 can provide increased privacy relative to other NMDs that use the VAS to process all speech inputs.

[0208] As mentioned above, the keywords in the library of the local NLU 779 correspond to parameters. These parameters can define performing a command corresponding to a detected command keyword. When a keyword is recognized in the speech input 780, the command corresponding to the detected command keyword is performed according to the parameter corresponding to the detected keyword.

[0209] For example, an example voice input 780 can be "play music at low volume," where "play" is a command keyword portion (corresponding to a playback command), and "music at low volume" is a voice utterance portion. When analyzing this voice input 780, the NLU 779 can identify that "low volume" is a keyword in its library that corresponds to a parameter that represents a certain (low) volume level. Accordingly, the NLU 779 can determine an intent to play at that low volume level. Then, when executing the playback command corresponding to "play," the command is executed according to the parameter that represents a certain volume level.

[0210] In a second example, another example voice input 780 can be "play my favorites in the kitchen," where "play" again is a command keyword portion (corresponding to a playback command), and "my favorites in the kitchen" as a voice utterance portion. When analyzing this voice input 780, the NLU 779 can identify that "favorites" and "kitchen" match keywords in its library. Specifically, "favorites" corresponds to a first parameter that represents specific audio content (i.e., a particular playlist that includes the user's favorite audio tracks), and "kitchen" corresponds to a second parameter that represents a target of the playback command (i.e., the kitchen 101h zone). Accordingly, the NLU 779 can determine an intent to play that particular playlist in the kitchen 101h zone.

[0211] In a third example, another example voice input 780 can be "turn up the volume," where "volume" is a command keyword portion (corresponding to a volume adjustment command), and "turn up" is a voice utterance portion. When analyzing this voice input 780, the NLU 779 can identify that "turn up" is a keyword in its library that corresponds to a parameter that represents a certain volume increase (an increase of 10 points on a 100-point volume scale). Accordingly, the NLU 779 can determine an intent to increase the volume. Then, when executing the volume adjustment command corresponding to "volume," the command is executed according to the parameter that represents a certain volume increase.

[0212] In examples, certain command keywords are functionally linked to a subset of keywords in the library of the local NLU 779, which can speed up analysis. For example, the command keyword "skip" can be functionally linked to the keywords "forward" and "backward" and their cognates. Thus, when the command keyword "skip" is detected in a given voice input 780, analyzing the speech utterance portion of that voice input 780 using the local NLU 779 can involve determining whether the voice input 780 includes any keywords that match the keywords linked to these functions (rather than determining whether the voice input 780 includes any keywords that match any of the keywords in the library of the local NLU 779). Because far fewer keywords are checked, this analysis is relatively faster than a full search of the library. In contrast, a random VAS wake word such as "Alexa" does not provide an indication of the range of accompanying voice inputs.

[0213] Some commands can require one or more parameters, so the individual command keyword does not provide enough information to execute the desired command. For example, the command keyword "volume" can require a parameter specifying whether the volume is to be increased or decreased, as the intent of "volume" alone is not clear. As another example, the command keyword "group" can require two or more parameters identifying the target devices to be grouped.

[0214] Thus, in some example implementations, when the command keyword engine 771a detects a given command keyword in a voice input 780, the local NLU 779 can determine whether the voice input 780 includes a keyword that matches a keyword in the library corresponding to a required parameter. If the voice input 780 does include a keyword that matches a required parameter, the NMD 703a proceeds to execute the command (corresponding to the given command keyword) according to the parameter specified by the keyword.

[0215] However, if the voice input 780 does not include a keyword that matches a required parameter for the command, the NMD 703a can prompt the user to provide the parameter. For example, in a first example, the NMD 703a can play an audible prompt such as "I heard the command, but I need more information" or "Can I help you with something?" Alternatively, the NMD 703a can send a prompt to the user's personal device via a control application (e.g., the software component 132c that controls the device 104).

[0216] In other examples, the NMD 703a can play a sound prompt that is tailored according to the detected command keyword. For example, after detecting a command keyword (e.g., “volume”) corresponding to a volume adjustment command, the sound prompt can include a more specific request, e.g., “Do you want to turn the volume up or down?” As another example, for a grouping command corresponding to the command keyword “group,” the sound prompt can be “Which devices do you want to group?” Supporting such specific sound prompts can be made feasible by supporting a relatively limited number of command keywords (e.g., fewer than 100), although other implementations can support more command keywords with the tradeoff of additional memory and processing power.

[0217] In additional examples, when the voice utterance portion does not include a keyword corresponding to one or more required parameters, the NMD 703a can execute the corresponding command according to one or more default parameters. For example, if a playback command does not include a keyword indicating a target playback device 102 for playback, the NMD 703a can default to playing back on the NMD 703a itself (e.g., if the NMD 703a is implemented within a playback device 102) or on one or more associated playback devices 102 (e.g., playback devices 102 in the same room or zone as the NMD 703a). Moreover, in some examples, a user can configure the default parameters using a graphical user interface (e.g., user interface 430) or a voice user interface. For example, if a grouping command does not specify playback devices 102 to group, the NMD 703a can default to instructing two or more preconfigured default playback devices 102 to form a synchronized group. The default parameters can be stored in a data storage device (e.g., memory 112b) and accessed by the NMD 703a when keywords exclude certain parameters. Other examples are possible as well.

[0218] In some cases, when the local NLU 779 is unable to process the voice input 780 (e.g., when the local NLU is unable to find a match for a keyword in a library, or when the local NLU 779 has a low confidence score regarding an intent), the NMD 703a sends the voice input 780 to the VAS. In examples, to trigger sending the voice input 780, the NMD 703a can generate a bridge event that causes the speech extractor 773 to process the sound data stream S D As described above. That is, the NMD 703a generates a bridge event to trigger the speech extractor 773 without the VAS wake word engine 770a detecting a VAS wake word (but based on a command keyword in the voice input 780 and the NLU 779 being unable to process the voice input 780).

[0219] Prior to sending the voice input 780 to the VAS (e.g., via message MV ), NMD 703a can obtain user consent from the user to send the voice input 780 to the VAS. For example, NMD 703a can play a sound prompt to send the voice input to a default or otherwise configured VAS, e.g., “Sorry, I didn’t understand that. Can I ask Alexa?” In another example, NMD 703a can play a sound prompt using a VAS voice (i.e., a voice associated with a particular VAS that most users know), e.g., “Can I help you with something?” In such an example, the generation of the bridge event (and the triggering of the voice extractor 773) depends on a second affirmative voice input 780 from the user.

[0220] In certain example implementations, the local NLU 779 can process the signal S ASR without having to generate a command keyword event by the command keyword engine 771a (i.e., directly). That is, the automatic speech recognition 772 can be configured to perform automatic speech recognition on the sound data stream S D , which the local NLU 779 processes to match keywords without the need for a command keyword event. If a keyword match in the voice input 780 is found that corresponds to a command (possibly with one or more keywords corresponding to one or more parameters), the NMD 703a executes the command according to the one or more parameters.

[0221] Further, in such examples, the local NLU 779 can directly process the signal S ASR only if certain conditions are met. Specifically, in some embodiments, the local NLU 779 processes the signal S ASR only if the state machine 775a is in a first state. The particular conditions can include a condition corresponding to an absence of background speech in the environment. An indication of whether background speech is present in the environment can come from the noise classifier 766. As described above, the noise classifier 766 can be configured to output a signal and / or set a state variable to indicate that far-field speech is present in the environment. Further, another condition can correspond to speech activity in the environment. The VAD 765 can be configured to output a signal and / or set a state variable to indicate that speech activity is present in the environment. Similarly, conditions determined by the state machine 775a can be used to reduce the incidence of false positive detections of commands using the direct processing method.

[0222] In some examples, the library of the local NLU 779 is partially customized for a single user. In a first aspect, the library can be customized for a household (e.g., the environment 101 FIG. 1AFor example, the library of the local NLU can include keywords corresponding to names of devices within the home, such as zone names for playback devices 102 in the MPS 100. In a second aspect, the library of the local NLU 779 can be customized for a user of the devices within the home. For example, the library of the local NLU 779 can include keywords corresponding to names or other identifiers of the user's preferred playlists, artists, albums, etc. The user can then refer to these names or identifiers when directing speech input to the command keyword engine 771a and the local NLU 779.

[0223] In example implementations, the NMD 703a can locally populate the library of the local NLU 779 within the network 111 FIG. 1B As described above, the NMD 703a can maintain or access state variables indicating respective states of devices (e.g., playback devices 104) connected to the network 111. These state variables can include names of various devices. For example, the kitchen 101h can include a playback device 101b assigned a zone name "Kitchen." The NMD 703a can read these names from the state variables and include them in the library of the local NLU 779 by training the local NLU 779 to recognize them as keywords. The keyword entry for a given name can then be associated with the corresponding device in the associated parameters (e.g., by an identifier of the device, such as a MAC address or IP address). The NMD 703a can then use these parameters to customize control commands and direct the commands to specific devices.

[0224] In other examples, the NMD 703a can populate the library by discovering devices connected to the network 111. For example, the NMD 703a can send a discovery request via the network 111 according to a protocol configured for device discovery (e.g., Universal Plug and Play (UPnP) or Zeroconf networking). Devices on the network 111 can then respond to the discovery request and exchange data representing device names, identifiers, addresses, etc. to facilitate communication and control via the network 111. The NMD 703a can read these names from the exchanged messages and include them in the library of the local NLU 779 by training the local NLU 779 to recognize them as keywords.

[0225] In other examples, the NMD 703a can use the cloud to populate the library. To illustrate, FIG. 10is a schematic diagram of the MPS 100 and the cloud network 902. The cloud network 902 includes cloud servers 906, identified as a media playback system control server 906a, a streaming audio service server 906b, and an IOT cloud server 906c, respectively. The streaming audio service server 906b can represent cloud servers of different streaming audio services. Similarly, the IOT cloud server 906c can represent cloud servers corresponding to different cloud services that support the smart devices 990 in the MPS 100.

[0226] One or more communication links 903a, 903b, and 903c (hereinafter referred to as "links 903") communicatively couple the MPS 100 and the cloud servers 906. The links 903 can include one or more wired networks and one or more wireless networks (e.g., the Internet). Further, similar to the network 111 FIG. 1B ), the network 911 communicatively couples the links 903 with at least a portion of the devices of the MPS 100 (e.g., one or more of the playback devices 102, the NMDs 103 and 703a, the control devices 104, and / or the smart devices 990).

[0227] In some implementations, the media playback system control server 906a facilitates populating the library of the local NLU 779 with the NMD 703a (representing one or more NMDs 703a within the MPS 100 FIG. 7A ). In an example, the media playback system control server 906a can receive data from the NMD 703a representing a request to populate the library of the local NLU 779. Based on the request, the media playback system control server 906a can communicate with the streaming audio service server 906b and / or the IOT cloud server 906c to obtain user-specific keywords.

[0228] In some examples, the media playback system control server 906a can utilize a user account and / or a user profile to obtain the user-specific keywords. As described above, a user of the MPS 100 can set up a user profile to define settings and other information within the MPS 100. The user profile can then be registered with user accounts of one or more streaming audio services in turn, to facilitate streaming of audio from these services to the playback devices 102 of the MPS 100.

[0229] By using these registered streaming audio services, the streaming audio service server 906b can collect data indicating user saved or preferred playlists, artists, albums, tracks, etc. via usage history or via user input (e.g., via user input designating media items as saved or favorites). This data can be stored in a database on the streaming audio service server 906b to facilitate providing certain features of the streaming audio service to the user, e.g., custom playlists, recommendations, and similar features. Under appropriate conditions (e.g., after receiving user permission), the streaming audio service server 906b can share this data with the media playback system control server 906a over link 903b.

[0230] Thus, in examples, the media playback system control server 906a can maintain or access data indicating user saved or preferred playlists, artists, albums, tracks, genres, etc. If a user has registered their user profile with multiple streaming audio services, the saved data can include saved playlists, artists, albums, tracks, etc. from two or more streaming audio services. Moreover, the media playback system control server 906a can have a more comprehensive view of the user's preferred playlists, artists, albums, tracks, etc. by aggregating data from two or more streaming audio services, as compared to a streaming audio service that can only access data generated using its own service.

[0231] Moreover, in some implementations, in addition to the data shared from the streaming audio service server 906b, the media playback system control server 906a can collect usage data from the MPS 100 over link 903a after receiving user permission. This can include data indicating user saved or preferred media items on a zone basis. Different types of music can be preferred in different rooms. For example, a user can prefer upbeat music in the kitchen 101h, and more mellow music in the office 101e to help focus.

[0232] Using data indicating the user's saved or preferred playlists, artists, albums, tracks, etc., the media playback system control server 906a can identify names of playlists, artists, albums, tracks, etc. that the user can reference when providing playback commands to the NMD 703a via voice input. Data representing these names can then be sent to the NMD 703a via the link 903a and the network 904, which are then added to the library of the local NLU 779 as keywords. For example, the media playback system control server 906a can send instructions to the NMD 703a to include certain names as keywords in the library of the local NLU 779. Alternatively, the NMD 703a (or another device of the MPS 100) can identify names of playlists, artists, albums, tracks, etc. that the user can reference when providing playback commands to the NMD 703a via voice input, and then include these names in the library of the local NLU 779.

[0233] As a result of this customization, similar voice inputs can result in different operations being performed when the local NLU 779 processes the voice inputs as compared to when the VAS processes the voice inputs. For example, a first voice input "Alexa, play my favorites in the office" can trigger a VAS wake word event because it includes the VAS wake word ("Alexa"). A second voice input "play my favorites in the office" can trigger a command keyword because it includes the command keyword ("play"). As a result, the first voice input is sent by the NMD 703a to the VAS, while the second voice input is processed by the local NLU 779.

[0234] While these voice inputs are nearly identical, they can result in different operations. Specifically, the VAS can do the best it can to determine a first audio track playlist to add to the queue of the playback device 102f in the office 101e. Similarly, the local NLU 779 can recognize the keywords "favorites" and "kitchen" in the second voice input. As a result, the NMD 703a executes the voice command "play" using the parameters <favorites playlist> and <kitchen 101h zone>, which causes a second audio track playlist to be added to the queue of the playback device 102f in the office 101e. However, the second audio track playlist can include a more complete and / or more accurate set of the user's favorite audio tracks because the second audio track playlist can utilize data indicating the user's saved or preferred playlists, artists, albums, and tracks from multiple streaming audio services, and / or usage data collected by the media playback system control server 906a. In contrast, the VAS can utilize its relatively limited concept of the user's saved or preferred playlists, artists, albums, and tracks when determining the first playlist.

[0235] To illustrate,FIG. 11 A table 1100 is shown that illustrates respective content of a first playlist and a second playlist that are determined based on similar voice inputs but are processed differently. Specifically, the first playlist is determined by the VAS while the second playlist is determined by the NMD 703a (possibly in conjunction with the media playback system control server 906a). As shown, while both playlists purport to include the user's favorites, the two playlists include audio content from different artists and genres. Specifically, the second playlist is configured from the use of the playback device 102f in the office 101e and the user's interactions with multiple streaming audio services, while the first playlist is based on multiple user interactions with the VAS. As a result, the second playlist is more suited to the type of music the user prefers to listen to in the office 101e (e.g., indie rock and folk), while the first playlist is more representative of interactions with the VAS as a whole.

[0236] A household can include multiple users. Two or more users can use the MPS 100 to configure their respective user profiles. Each user profile can have its own user account with one or more streaming audio services associated with the respective user profile. Further, the media playback system control server 906a can maintain or access data indicating each user's saved or preferred playlists, artists, albums, tracks, genres, etc., which can be associated with the user's user profile.

[0237] In various examples, names corresponding to user profiles can be populated into the library of the local NLU 779. This can facilitate referencing a particular user's saved or preferred playlists, artists, albums, tracks, or genres. For example, when the local NLU 779 processes the voice input "play Annie's favorites on the patio," the local NLU 779 can determine that "Annie" matches a stored keyword corresponding to a particular user. Then, when executing the playback command corresponding to the voice input, the NMD 703a adds a playlist of audio tracks that the particular user favors to the queue of the playback device 102c in the patio 101i.

[0238] In some cases, a voice input can not include a keyword corresponding to a particular user, but the MPS 100 is configured with multiple user profiles. In some cases, the NMD 703a can determine a user profile to use when executing a command using voice recognition. Alternatively, the NMD 703a can default to a certain user profile. Further, the NMD 703a can use preferences from multiple user profiles when executing a command corresponding to a voice input that does not identify a particular user profile. For example, the NMD 703a can determine a favorites playlist that includes preferred or saved audio tracks from each user profile registered with the MPS 100.

[0239] The IOT cloud server 906c can be configured to provide cloud services that support the smart devices 990. The smart devices 990 can include various "smart" internet-connected devices, e.g., lights, thermostats, cameras, security systems, appliances, etc. For example, the IOT cloud server 906c can provide cloud services that support a smart thermostat, which allow a user to control the smart thermostat over the internet via a smartphone app or website.

[0240] Accordingly, in examples, the IOT cloud server 906c can maintain or have access to data associated with the user's smart devices 990, e.g., device names, settings, and configurations. Under the appropriate conditions (e.g., after receiving user permission), the IOT cloud server 906c can share this data with the media playback system control server 906a and / or the NMD 703a via link 903c. For example, the IOT cloud server 906c that provides cloud services for a smart thermostat can provide the NMD 703a with data representing these keywords, which facilitates populating the local NLU 779's library with keywords that correspond to temperatures.

[0241] Further, in some cases, the IOT cloud server 906c can also provide keywords specific to controlling its corresponding smart devices 990. For example, the IOT cloud server 906c that provides cloud services that support a smart thermostat can provide a set of keywords that correspond to voice control of the thermostat, e.g., "temperature," "warmer," or "cooler," etc. Data representing these keywords can be sent from the IOT cloud server 906c to the NMD 703a over link 903 and network 904.

[0242] As noted above, some households can include multiple NMDs 703a. In example implementations, two or more NMDs 703a can synchronize or otherwise update their respective local NLU 779's libraries. For example, a first NMD 703a and a second NMD 703a can share data representing their respective local NLU 779's libraries, possibly using a network (e.g., network 904). This sharing can facilitate the NMDs 703a being able to respond similarly to voice inputs, among other possible benefits.

[0243] In some embodiments, one or more of the above-described components can operate in conjunction with the microphone 720 to detect and store a user's voice profile, which can be associated with a user account of the MPS 100. In some embodiments, the voice profile can be stored as and / or compared to variables stored in a set of command information or data tables. The voice profile can include tonal or frequency aspects of the user's voice and / or other unique aspects of the user, such as those described in previously referenced U.S. Patent Application No. 15 / 438,749.

[0244] In some embodiments, one or more of the above-described components can operate in conjunction with the microphone 720 to determine a user's location in the home environment and / or relative to one or more NMDs 103. Techniques for determining a user's location or proximity can include one or more of the techniques disclosed in previously referenced U.S. Patent Application No. 15 / 438,749, U.S. Patent No. 9,084,058, filed December 29, 2011, entitled "Sound Field Calibration Using Listener Localization," and U.S. Patent No. 8,965,033, filed August 31, 2012, entitled "Acoustic Optimization." Each of these applications is incorporated by reference herein in its entirety.

[0245] IV. Example Command Keyword Techniques

[0246] FIG. 12 is a flowchart illustrating an example method 1200 of executing a first playback command based on a command keyword event. The method 1200 can be performed by a networked microphone device such as the NMD 103s FIG. 1A ) can include features of the NMD 703a FIG. 7A ) in some implementations, the NMD is implemented within a playback device, as shown by the playback device 102r.

[0247] At block 1202, the method 1200 involves monitoring an input sound data stream for (i) a wake-word event and (ii) a first command keyword event. For example, the VAS wake-word engine 770a of the NMD 703a can apply one or more wake-word recognition algorithms to the sound data stream S DS ( FIG. 7A ). In addition, the command keyword engine 770 can monitor the sound data stream S DS , possibly using the ASR 772 and the local NLU 779, as described above in connection with FIG. 7A ​

[0248] At block 1204, the method 1200 involves detecting a wake-word event. Detecting the wake-word event can involve the VAS wake-word engine 770a of the NMD 703a detecting that the first sound detected via the microphone 720 includes a first speech input that includes a wake-word. The VAS wake-word engine 770a can use a recognition algorithm to detect such a wake-word in the first speech input.

[0249] At block 1206, the method 1200 involves streaming sound data corresponding to the first speech input to one or more remote servers of a voice assistant service. For example, the speech extractor 773 can extract at least a portion of the first speech input (e.g., the wake-word portion and / or the speech utterance portion) from the sound data stream S DS ( FIG. 7A ) to the one or more remote servers of the voice assistant service via the network interface 724.

[0250] At block 1208, the method 1200 involves detecting a first command keyword event. For example, after detecting the second sound, the command keyword engine 771a of the NMD 703a can detect a first command keyword in the sound data stream S D corresponding to the second speech input in the second sound. Other examples are also possible.

[0251] At block 1210, the method 1200 involves determining whether one or more playback conditions corresponding to the first command keyword are satisfied. Determining whether the one or more playback conditions corresponding to the first command keyword are satisfied can involve determining a state of a state machine. For example, the state machine 775 of the NMD 703a can transition to a first state when the one or more playback conditions corresponding to the first command keyword are satisfied and to a second state when at least one of the one or more playback conditions corresponding to the first command keyword are not satisfied FIG. 7C . Example playback conditions are shown in Table 985 FIG. 9A and FIG. 9B .

[0252] At block 1212, the method 1200 involves executing a first playback command corresponding to the first command keyword. For example, the NMD 703a can execute the first playback command based on detecting the first command keyword event and determining that the one or more playback conditions corresponding to the first command keyword are satisfied. In an example, executing the first playback command can involve generating one or more instructions to execute the command that cause a target playback device to execute the first playback command.

[0253] In the example, the target playback device 102 executing the first playback command can be defined explicitly or implicitly. For example, the target playback device 102 can be explicitly defined by referencing the name of one or more playback devices in the voice input 780 (e.g., by referencing a zone or zone group name). Alternatively, the voice input may not include any reference to the name of one or more playback devices, but may implicitly refer to the playback device 102 associated with NMD 703a. The playback device 102 associated with NMD 703a may include a playback device implementing NMD 703a (such as playback device 102d implementing NMD 103d). FIG. 1B (as shown) or configured as an associated playback device (e.g., where playback device 102 and NMD 703a are located in the same room or area).

[0254] In the example, performing the first playback operation may involve sending one or more instructions over one or more networks. For example, the NMD 703a can send instructions locally to one or more playback devices 102 via network 903 to perform actions such as transmission control. FIG. 10 Instructions like ) FIG. 6 The message exchange is shown. Furthermore, NMD 703a can send a request to streaming audio service 906b to stream one or more audio tracks to target playback device 102 via link 903 ( FIG. 10 Playback. Alternatively, instructions may be provided internally (e.g., via a local bus or other interconnect system) to one or more software or hardware components (e.g., electronics 112 of playback device 102).

[0255] Furthermore, transmission instructions can involve both local and cloud-based operations. For example, NMD 703a can send instructions locally to one or more playback devices 102 via network 903 to add one or more audio tracks to the playback queue via network 903. The one or more playback devices 102 can then send a request to the streaming audio service 906b to stream one or more audio tracks to the target playback device 102 for playback via link 903. Other examples are also possible.

[0256] FIG. 13 This is a flowchart illustrating an example method 1300 for executing a first playback command based on a command keyword event according to one or more parameters. Similar to method 1200, method 1300 can be implemented using, for example, NMD 120 (…). FIG. 1A This is performed by networked microphone devices such as the NMD 703a. FIG. 7A The characteristics of NMD. In some implementations, NMD is implemented within the playback device, as shown in playback device 102r.

[0257] At block 1302, the method 1300 involves monitoring an input sound data stream for (i) a wake-word event and (ii) a first command keyword event. For example, the VAS wake-word engine 770a of the NMD 703a can apply one or more wake-word recognition algorithms to the sound data stream S DS ( FIG. 7A ). In addition, the command keyword engine 770 can monitor the sound data stream S DS , possibly using the ASR 772 and the local NLU 779, as described above in connection with FIG. 7A .

[0258] At block 1304, the method 1300 involves detecting the wake-word event. Detecting the wake-word event can involve the VAS wake-word engine 770a of the NMD 703a detecting that the first sound detected via the microphone 720 includes a first speech input that includes a wake-word. The VAS wake-word engine 770a can use a recognition algorithm to detect such a wake-word in the first speech input.

[0259] At block 1306, the method 1300 involves streaming sound data corresponding to the first speech input to one or more remote servers of a voice assistant service. For example, the speech extractor 773 can extract at least a portion of the first speech input (e.g., the wake-word portion and / or the speech utterance portion) from the sound data stream S DS ( FIG. 7A ). The NMD 703 can then stream this extracted data to one or more remote servers of a voice assistant service via the network interface 724.

[0260] At block 1308, the method 1300 involves detecting the first command keyword event. For example, after detecting the second sound, the command keyword engine 771a of the NMD 703a can detect a first command keyword in the sound data stream S D corresponding to the second speech input in the second sound ( FIG. 7A ). In addition, the local NLU 779 can detect that the second speech input includes at least one keyword from a library of the local NLU 779. For example, the local NLU 779 can determine whether the speech input includes any keywords that match keywords in the library of the local NLU 779. The local NLU 779 is configured to analyze the signal S ASR to find (i.e., detect or recognize) keywords in the speech input.

[0261] At block 1310, the method 1300 involves determining an intent based on the at least one keyword. For example, the local NLU 779 can determine an intent from the one or more keywords in the second speech input. As described above, the keywords in the library of the local NLU 779 correspond to parameters. The keywords in the speech input can indicate an intent, such as playing a particular audio content in a particular zone.

[0262] At block 1312, the method 1300 involves executing the first playback command according to the determined intent. In an example, executing the first playback command can involve generating one or more instructions to execute the command according to the determined intent, the one or more instructions causing a target playback device to execute the first playback command customized by the parameter in the speech utterance portion of the speech input. As indicated above in connection with block 1212 FIG. 12 ) of the method 1200, the target playback device 102 to execute the first playback command can be defined explicitly or implicitly. Further, executing the first playback operation can involve sending the one or more instructions over one or more networks, or can involve providing the instructions internally (e.g., to a playback device component).

[0263] FIG. 14 is a flow diagram illustrating an example method 1400 of executing a first playback command according to one or more parameters based on a command keyword event corresponding to a command keyword. The command keyword event can be generated only if certain conditions are met. A first condition is that there is no background speech in the environment when the command keyword is detected.

[0264] Similar to the methods 1200 and 1300, the method 1400 can be performed by a networked microphone device, such as the NMD 120 FIG. 1A ) can include features of the NMD 703a FIG. 7A ) in some implementations, the NMD is implemented within a playback device, as illustrated by the playback device 102r.

[0265] At block 1402, the method 1400 involves detecting sound via one or more microphones. For example, the NMD 703 can detect sound via the microphone 720 FIG. 7A ) of the VCC 760.

[0266] At block 1404, the method 1400 involves determining (i) that the detected sound includes a speech input, (ii) that the detected sound does not include background speech, and (iii) that the speech input includes a command keyword.

[0267] For example, to determine whether the detected sound includes a voice input, the voice activity detector 765 can analyze the detected sound to determine the presence (or absence) of voice activity in the sound data stream S DS ( FIG. 7A ). Moreover, to determine whether the detected sound does not include background speech, the noise classifier 766 can analyze sound metadata corresponding to the detected sound and determine whether the sound metadata includes features corresponding to background speech.

[0268] Moreover, to determine whether the voice input includes a command keyword, the command keyword engine 771a can analyze the sound data stream S DS ( FIG. 7A ). Specifically, the ASR 772 can transcribe the sound data stream S DS into text (e.g., signal S ASR ), and the local NLU 779 can determine that a word matching a command keyword is in the transcribed text. In other examples, the command keyword engine 771a can use one or more keyword recognition algorithms on the sound data stream S DS . Other examples are possible.

[0269] At block 1406, the method 1400 involves performing a playback function corresponding to the command keyword. For example, the NMD can perform the playback function based on determining that (i) the detected sound includes a voice input, (ii) the detected sound does not include background speech, and (iii) the voice input includes a command keyword. FIG. 12 and FIG. 13 blocks 1212 and 1312, respectively, provide examples of performing a playback function.

[0270] V. ILLUSTRATIVE EXAMPLES

[0271] FIG. 15A 、 FIG. 15B 、 FIG. 15C and FIG. 15D illustrate example inputs and outputs of an example NMD configured in accordance with aspects of the disclosure.

[0272] FIG. 15A illustrates a first scenario in which a wake-word engine of an NMD is configured to detect three command keywords (“play,” “stop,” and “resume”). A local NLU is disabled. In this scenario, a user has spoken the voice input “play” to the NMD, which triggers a new recognition of one of the command keywords (e.g., a command keyword event corresponding to play).

[0273] In addition, the voice activity detector (VAD) and the noise classifier have analyzed the first 150 frames of the pre-roll portion of the voice input. As shown, the VAD has detected voice in 140 of the 150 pre-roll frames, which indicates that there can be voice input in the detected sound. In addition, the noise classifier has detected ambient noise in 11 frames, background speech in 127 frames, and fan noise in 12 frames. In this NMD, the noise classifier classifies the primary source of noise in each frame. This indicates that there is background speech. Accordingly, the NMD has determined not to trigger on the detected command keyword "play."

[0274] FIG. 15B A second scenario is shown in which the wake-word engine of the NMD is configured to detect a command keyword ("play") and two cognates of the command keyword ("play something" and "play a song for me"). The local NLU is disabled. In this second scenario, the user has spoken the voice input "play something" to the NMD, which triggers a new recognition of one of the command keywords (e.g., a command keyword event).

[0275] In addition, the voice activity detector (VAD) and the noise classifier have analyzed the first 150 frames of the pre-roll portion of the voice input. As shown, the VAD has detected voice in 87 of the 150 pre-roll frames, which indicates that there can be voice input in the detected sound. In addition, the noise classifier has detected ambient noise in 18 frames, background speech in 8 frames, and fan noise in 124 frames. This indicates that there is no background speech. Given the above, the NMD has determined to trigger on the detected command keyword "play."

[0276] FIG. 15C A third scenario is shown in which the wake-word engine of the NMD is configured to detect three command keywords ("play," "stop," and "resume"). The local NLU is enabled. In this third scenario, the user has spoken the voice input "play the Beatles in the kitchen" to the NMD, which triggers a new recognition of one of the command keywords (e.g., a command keyword event corresponding to play).

[0277] As shown, the ASR has transcribed the speech input as "play some music in the office." It is expected that some errors will occur when performing the ASR (e.g., "some music"). Here, the local NLU has matched the keyword "some music" to "some music" in the local NLU library, setting the content parameter of the play command to the music. Further, the local NLU has matched the keyword "office" to "office" in the local NLU library, setting the target parameter of the play command to the office area. The local NLU produces a confidence score of 0.63428231948273443 associated with the intent determination.

[0278] Here, the voice activity detector (VAD) and noise classifier have analyzed 150 frames of the pre-roll portion of the speech input. As shown, the noise classifier has detected ambient environmental noise in 142 frames, background speech in 8 frames, and fan noise in 0 frames. This indicates that there is no background speech. The VAD has detected speech in 112 frames of the 150 pre-roll frames, indicating that there can be speech input in the detected sound. Here, the NMD has determined to trigger on the detected command keyword "play."

[0279] Further, the voice activity detector (VAD) and noise classifier have analyzed 150 frames of the pre-roll portion of the speech input. As shown, the VAD has detected speech in 140 frames of the 150 pre-roll frames, indicating that there can be speech input in the detected sound. Further, the noise classifier has detected ambient environmental noise in 11 frames, background speech in 127 frames, and fan noise in 12 frames. This indicates that there is background speech. Accordingly, the NMD has determined not to trigger on the detected command keyword "play."

[0280] FIG. 15D A fourth scenario is shown in which the keyword engine of the NMD is not configured to find any command keywords. Rather, the keyword engine will perform ASR and pass the output of the ASR to the local NLU. The local NLU is enabled and configured to detect keywords corresponding to commands and parameters. In the fourth scenario, the user has spoken the speech input "play some music in the office" to the NMD.

[0281] As shown, the ASR has transcribed the speech input as "lay some music in the office." Here, the local NLU has matched the keyword "lay" to "play" in the local NLU library, thereby responding to a playback command. In addition, the local NLU has matched the keyword "office" to "office" in the local NLU library, thereby setting the office zone as a target parameter for the play command. The local NLU produces a confidence score of 0.14620494842529297 associated with the keyword match. In some examples, this low confidence score can cause the NMD to not accept the speech input (e.g., if the confidence score is below a threshold, e.g., 0.5).

[0282] CONCLUSION

[0283] The above description discloses, among other things, various example systems, methods, apparatus, and articles of manufacture, including, among other things, firmware and / or software stored on hardware. It is to be understood that the examples are merely illustrative, and that they should not be considered to be limiting in any way. For example, it is contemplated that any or all of the firmware, hardware, and / or software aspects or components can be embodied exclusively in hardware, exclusively in software, exclusively in firmware, or in any combination of hardware, software, and / or firmware. Accordingly, the examples provided are merely examples and are not intended to limit the scope of the systems, methods, apparatus, and / or articles of manufacture.

[0284] This specification presents aspects in terms of illustrative environments, systems, processes, steps, logic blocks, processing, and other symbolic representations of operations on data that are directly or indirectly coupled to a data processing apparatus, such as a network. These process descriptions and representations are typically used by those skilled in the art to most effectively convey the substance of their work to others skilled in the art. Various specific details are set forth in this description to provide a thorough understanding of the disclosure. Those skilled in the art, however, will understand that the disclosure can be practiced without particular, specific details, and that the disclosure can not include all of the details needed to implement the disclosure in every

[0285] When any of the appended claims are read to cover a purely software and / or firmware implementation, at least one element in at least one of the examples is hereby expressly defined to include a non-transitory, tangible medium such as a memory, DVD, CD, Blu-ray, etc. storing the software and / or firmware.

[0286] For example, the present technology is illustrated in accordance with various aspects described below. For convenience, the various examples of aspects of the present technology are described as numbered examples (1, 2, 3, etc.). These are provided merely as examples and do not limit the present technology. Note that any dependent example can be combined in any combination and placed in the corresponding independent example. Other examples can be presented in a similar manner.

[0287] Example 1 : A method performed by a playback device, the playback device comprising a network interface and at least one microphone configured to detect sound, the method comprising: monitoring an input sound data stream representing sound detected by the at least one microphone for (i) a wake-word event and (ii) a first command keyword event; detecting the wake-word event, wherein detecting the wake-word event comprises: after detecting a first sound via the one or more microphones, determining that the detected first sound comprises a first speech input, the first speech input comprising a wake-word; streaming, via the network interface, sound data corresponding to at least a portion of the first speech input to one or more remote servers of a voice assistant service; detecting the first command keyword event, wherein detecting the first command keyword event comprises: after detecting a second sound via the one or more microphones, determining that the detected second sound comprises a second speech input, the second speech input comprising a first command keyword, wherein the first command keyword is one of a plurality of command keywords supported by the playback device; determining that one or more playback conditions corresponding to the first command keyword are satisfied; and in response to detecting the first command keyword event and determining that the one or more playback conditions corresponding to the first command keyword are satisfied, performing a first playback command corresponding to the first command keyword.

[0288] Example 2: The method of example 1, further comprising: after detecting the first command keyword event, detecting a subsequent first command keyword event, wherein detecting the subsequent first command keyword event comprises: after detecting a third sound via the at least one microphone, determining that the third sound comprises a third speech input, the third speech input comprising the first command keyword; determining that at least one of the one or more playback conditions corresponding to the first command keyword are not satisfied; and in response to determining that the at least one playback condition is not satisfied, forgoing performing the first playback command corresponding to the first command keyword.

[0289] Example 3: The method of any of examples 1 and 2, further comprising: detecting a second command keyword event, wherein detecting the second command keyword event comprises: after detecting a third sound via the at least one microphone, determining that the third sound includes a third voice input, the third voice input including a second command keyword in the detected third sound; determining that one or more playback conditions corresponding to the second command keyword are satisfied; and in response to detecting the second command keyword, and determining that the one or more playback conditions corresponding to the second command keyword are satisfied, executing a second playback command corresponding to the second command keyword.

[0290] Example 4: The method of example 3, wherein at least one of the one or more playback conditions corresponding to the second command keyword is not a playback condition of the one or more playback conditions corresponding to the first command keyword.

[0291] Example 5: The method of any of examples 1-4, wherein the first command keyword is a skip command, wherein the one or more playback conditions corresponding to the first command keyword include: (i) a first condition that a media item is being played back on the playback device, (ii) a second condition that a queue is active on the playback device, and (iii) a third condition that the queue includes a media item after the media item being played back on the playback device, and wherein executing the first playback command corresponding to the first command keyword comprises skipping forward in the queue to play the media item after the media item being played back on the playback device.

[0292] Example 6: The method of any of examples 1-4, wherein the first command keyword is a pause command, wherein the one or more playback conditions corresponding to the first command keyword include a condition that audio content is being played back on the playback device, and wherein executing the first playback command corresponding to the first command keyword comprises pausing playback of the audio content on the playback device.

[0293] Example 7: The method of any of examples 1-4, wherein the first command keyword is a volume up command, wherein the one or more playback conditions corresponding to the first command keyword include: a first condition that audio content is being played back on the playback device; and a second condition that a volume level on the playback device is not at a maximum volume level, wherein executing the first playback command corresponding to the first command keyword comprises increasing the volume level on the playback device.

[0294] Example 8: The method of any of examples 1-8, wherein the one or more playback conditions corresponding to the first command keyword include a first condition that no background speech is absent from the detected first sound.

[0295] Example 9: A tangible, non-transitory computer-readable medium having instructions stored thereon, the instructions executable by one or more processors to cause a playback device to perform the method of any of examples 1-8.

[0296] Example 10: A playback device comprising: a speaker; a network interface; one or more microphones configured to detect sound; one or more processors; and a tangible, non-transitory computer-readable medium having instructions stored thereon, the instructions executable by the one or more processors to cause the playback device to perform the method of any of examples 1-8.

[0297] Example 11: A method performed by a playback device, the playback device comprising a network interface and at least one microphone configured to detect sound, the method comprising: monitoring an input sound data stream representing sound detected by the at least one microphone for (i) a wake-word event and (ii) a first command-keyword event; detecting the wake-word event, wherein detecting the wake-word event comprises, after detecting a first sound via the one or more microphones, determining that the detected first sound includes a first speech input, the first speech input including a wake-word; streaming, via the network interface, sound data corresponding to at least a portion of the first speech input to one or more remote servers of a voice assistant service; detecting the first command-keyword event, wherein detecting the first command-keyword event comprises, after detecting a second sound via the one or more microphones, determining that the detected second sound includes a second speech input, the second speech input including a first command keyword and at least one keyword, wherein the first command keyword is one of a plurality of command keywords supported by the playback device, and wherein the first command keyword corresponds to a first playback command; determining, via a local natural language unit (NLU), an intent based on the at least one keyword, wherein the NLU includes a predetermined keyword library, the predetermined keyword library including the at least one keyword; and after (a) detecting the first command-keyword event and (b) determining the intent, performing the first playback command in accordance with the determined intent.

[0298] Example 12: The method of example 11, wherein the method further comprises detecting a second command keyword event, wherein detecting the second command keyword event comprises: after detecting a third sound via the at least one microphone, determining that the third sound includes a third voice input, the third voice input including the second command keyword; determining that the third voice input including the second command keyword does not include at least one other keyword from the predetermined library of keywords; and after determining that the third voice input including the second command keyword does not include the at least one keyword from the predetermined library of keywords, streaming sound data representing at least a portion of the voice input including the second command keyword to one or more servers of the voice assistant service for processing by the one or more remote servers of the voice assistant service.

[0299] Example 13: The method of example 12, wherein the method further comprises: playing back a sound prompt to request confirmation to invoke the voice assistant service to process the second command keyword; and after playing back the sound prompt, receiving data representing a confirmation to invoke the voice assistant service to process the second command keyword, wherein streaming sound data representing at least a portion of the voice input including the second command keyword to one or more servers of the voice assistant service only occurs after receiving data representing a confirmation to invoke the voice assistant service.

[0300] Example 14: The method of any of examples 10-13, further comprising detecting a second command keyword event, wherein detecting the second command keyword event comprises: after detecting a third sound via the at least one microphone, determining that the third sound includes a third voice input, the third voice input including the second command keyword; determining that the third voice input including the second command keyword does not include at least one other keyword from the predetermined library of keywords; and after determining that the third voice input including the second command keyword does not include the at least one keyword in the predetermined library of keywords, executing the first playback command according to one or more default parameters.

[0301] Example 15: The method of any of examples 10-14, wherein the first keyword of the at least one keyword in the detected second sound represents a zone name corresponding to a first zone of a media playback system, wherein executing the first playback command according to the determined intent comprises sending one or more instructions to execute the first playback command in the first zone, and wherein the media playback system includes the playback device.

[0302] Example 16: The method of any of examples 10-15, further comprising populating the predetermined keyword library with zone names corresponding to zones within the media playback system, wherein each zone includes one or more respective playback devices, and wherein the predetermined keyword library is populated with a zone name corresponding to the first zone of the media playback system.

[0303] Example 17: The method of any of examples 10-16, further comprising discovering, via the network interface, smart home devices connected to a local area network; and populating the predetermined keyword library with names corresponding to respective smart home devices discovered on the local area network.

[0304] Example 18: The method of any of examples 10-17, wherein the media playback system includes the playback device, wherein the media playback system is registered to one or more user profiles, and wherein the functionality further comprises populating the predetermined keyword library with names corresponding to playlists designated as favorites by the one or more user profiles.

[0305] Example 19: The method of example 18, wherein a first user profile of the one or more user profiles is associated with a user account of a first streaming audio service and a user account of a second streaming audio service, and wherein the playlists include: a first playlist of the first streaming audio service designated as a favorite by the user account of the first streaming audio service; and a second playlist including audio tracks from the first streaming audio service and the second streaming audio service.

[0306] Example 20: The method of any of examples 10-16, wherein detecting the first command keyword event further comprises determining that one or more playback conditions corresponding to the first command keyword are satisfied.

[0307] Example 21 : The method of example 20, wherein the one or more playback conditions corresponding to the first command keyword include a first condition that no background speech is present in the detected first sound.

[0308] Example 22: A tangible, non-transitory computer-readable medium having instructions stored thereon, the instructions executable by one or more processors to cause a playback device to perform the method of any of examples 10-21.

[0309] Example 23: A playback device comprising: a speaker; a network interface; one or more microphones configured to detect sound; one or more processors; and a tangible, non-transitory computer-readable medium having stored thereon instructions that are executable by the one or more processors to cause the playback device to perform the method of any one of examples 10-21.

[0310] Example 24: A method performed by a playback device comprising a network interface and at least one microphone configured to detect sound, the method comprising: detecting sound via the one or more microphones; determining (i) that the detected sound includes speech input, (ii) that the detected sound does not include background speech, and (iii) that the speech input includes a command keyword; and in response to determining (i) that the detected sound includes speech input, (ii) that the detected sound does not include background speech, and (iii) that the speech input includes a command keyword, performing a playback function corresponding to the command keyword.

[0311] Example 25: The method of example 24, wherein the detected sound is a first detected sound, and wherein the method further comprises: detecting a second sound via the at least one microphone; determining that the detected second sound includes a wake word; and after determining that the detected second sound includes the wake word, streaming speech input in the detected second sound to one or more remote servers of a voice assistant service via a network interface of the playback device.

[0312] Example 26: The method of any one of examples 24 and 25, wherein determining that there is no background speech in the detected second sound comprises: determining sound metadata corresponding to the detected sound; and analyzing the sound metadata to classify the detected sound according to one or more particular signatures selected from a plurality of signatures, wherein each signature of the plurality of signatures is associated with a source of noise, and wherein at least one signature of the plurality of signatures is a background speech signature indicative of background speech.

[0313] Example 27: The method of example 26, wherein analyzing the sound metadata comprises: classifying frames associated with the detected sound as having a particular speech signature other than the background speech signature; and comparing a number of frames classified with the background speech signature, if any, to a number of frames classified with signatures other than the background speech signature.

[0314] Example 28: The method of any one of examples 24 and 25, wherein determining that there is speech input in the detected sound comprises detecting speech activity in the detected sound.

[0315] Example 29: The method of example 28, wherein detecting speech activity in the detected sound comprises: determining a plurality of first frames associated with the detected sound as containing speech; and comparing the plurality of first frames to a plurality of second frames that are (a) associated with the detected sound and (b) do not indicate speech.

[0316] Example 30: The method of example 29, wherein the first frames comprise: one or more frames generated in response to near-field speech activity; and one or more frames generated in response to far-field speech activity.

[0317] Example 31 : A tangible, non-transitory computer-readable medium having stored thereon instructions executable by one or more processors to cause a playback device to perform the method of any of examples 10-21.

[0318] Example 32: A playback device comprising: one or more microphones configured to detect sound; one or more processors; and a tangible, non-transitory computer-readable medium having stored thereon instructions executable by the one or more processors to cause the playback device to perform the method of any of examples 24-30.

Claims

1. A method performed by a playback device, the playback device comprising: At least one microphone is configured to detect sound; And a first wake-up word engine and a second wake-up word engine, configured to receive input sound data representing the sound detected by the at least one microphone, The first wake-up word engine is configured to: when the first wake-up word engine detects a Voice Assistant Service (VAS) wake-up word in the input sound data, generate a Voice Assistant Service (VAS) wake-up word event, wherein the VAS wake-up word event causes the playback device to stream sound data representing the sound detected by the at least one microphone to one or more servers of the Voice Assistant Service, the method comprising: The second wake word engine detects the first command keyword among a plurality of command keywords in the sound data, and each of the plurality of command keywords corresponds to a corresponding playback command; If it is determined that one or more playback conditions corresponding to the detected first command keyword are met, a command keyword event corresponding to the detected first command keyword is generated via the second wake word engine; and In response to the command keyword event, execute the first playback command corresponding to the first command keyword.

2. The method according to claim 1, wherein, The playback device further includes a state machine configured to: transition to a first state when one or more playback conditions corresponding to the first command keyword are met, and transition to a second state when at least one of the one or more playback conditions corresponding to the first command keyword is not met, wherein determining that the one or more playback conditions corresponding to the first command keyword are met includes determining that the state machine is in the first state.

3. The method according to claim 2, wherein, The playback device further includes an additional state machine corresponding to each of the plurality of command keywords, wherein each additional state machine is configured to: transition to the first state when one or more playback conditions corresponding to the corresponding command keyword are met, and transition to the second state when at least one of the one or more playback conditions corresponding to the corresponding command keyword is not met.

4. The method according to claim 1, further comprising: If it is determined that at least one of the playback conditions corresponding to the detected first command keyword is not met, then the generation of the command keyword event corresponding to the detected first command keyword is suppressed.

5. The method according to claim 1, further comprising: The buffer stores sound data representing the sound detected by the at least one microphone; When the first wake word engine detects the first VAS wake word, it generates a VAS wake word event corresponding to the detected first VAS wake word. as well as In response to the command keyword event corresponding to the first command keyword, a portion of buffered audio data is streamed to one or more servers of the voice assistant service via a network interface, including: (i) buffered audio data for a predetermined duration prior to the first VAS wake word; (ii) buffered audio data representing the speech sound following the first VAS wake word.

6. The method according to claim 1, wherein: The playback device further includes a third wake word engine, which is configured to receive input sound data representing the sound detected by the at least one microphone; and The method further includes: The third wake-up word engine detects a specific streaming audio service wake-up word among multiple audio service wake-up words supported by the third wake-up word engine, and each of the specific streaming audio service wake-up words corresponds to a corresponding streaming audio service command; When it is determined that one or more streaming audio service conditions corresponding to the specific streaming audio service wake word are met: The third wake-up word engine generates a streaming audio service wake-up word event corresponding to the specific streaming audio service wake-up word; as well as In response to the streaming audio service wake word event, execute the specific streaming audio service command corresponding to the specific streaming audio service wake word.

7. The method according to claim 1, wherein: The sound data is a first voice input, which includes the first command keyword and the first voice utterance; and The method further includes: In response to the command keyword event, the local natural language unit (NLU) determines whether the first speech utterance includes at least one keyword from a predetermined keyword library; and When the first voice utterance includes one or more specific keywords, executing the first playback command includes: executing the first playback command according to one or more parameters corresponding to the one or more specific keywords in the first voice utterance.

8. The method according to claim 7, further comprising: When the first spoken utterance does not include at least one keyword from the predetermined keyword library, it includes at least one of the following: Streaming sound data representing the first voice input to one or more servers of the voice assistant service for processing by the one or more servers of the voice assistant service; as well as Play back the audio prompt of the second voice input to the voice assistant service; Streaming sound data representing the second voice input to one or more servers of the voice assistant service for processing by the one or more servers of the voice assistant service; as well as The first playback command is executed based on one or more default parameters.

9. The method according to claim 8, wherein: The first keyword in the one or more specific keywords represents the zone name corresponding to the first zone of the media playback system, and Executing the first playback command according to one or more parameters includes sending one or more instructions to execute the first playback command in the first zone.

10. The method of claim 9, further comprising: The predetermined keyword library is populated using zone names corresponding to the corresponding zones within the media playback system. Each zone includes one or more corresponding playback devices. The predetermined keyword library is filled with zone names corresponding to the first zone of the media playback system.

11. The method of claim 7, further comprising: The system discovers smart home devices connected to the local area network via the network interface. as well as The predetermined keyword library is populated with names corresponding to the smart home devices found on the local area network.

12. The method of claim 7, further comprising: The predetermined keyword library is populated with names corresponding to favorite playlists specified in one or more user profiles registered with the media playback system.

13. The method of claim 7, further comprising: It is determined that the first playback command requires parameters; Playback of audio prompts to provide a second voice input including keywords corresponding to the given parameters, the second voice input including a second voice utterance; The local natural language unit (NLU) determines whether the second speech utterance includes keywords corresponding to the given parameters. as well as When the second speech contains a keyword corresponding to the given parameter, the first playback command is executed according to the given parameter.

14. A non-transitory computer-readable medium having instructions stored thereon, the instructions being executable by one or more processors to cause a playback device to perform the method according to any one of the preceding claims.

15. A playback device, comprising: Network interface; One or more processors; At least one microphone is configured to detect sound; At least one speaker; First wake-up word engine and second wake-up word engine; as well as A data storage device storing instructions that can be executed by the one or more processors to cause the playback device to perform the method according to any one of claims 1-13.

Images