Simultaneous acoustic event detection across multiple assistant devices

Abstract

To provide simultaneous acoustic event detection across multiple assistant devices.SOLUTION: An implementation embodiment detects audio data that capture acoustic events at various assistant devices in an ecosystem, processes the audio data locally at each assistant device to generate a measure associated with the acoustic event using respective event detection models, processes the respective measures to determine whether the detected acoustic event is an actual acoustic event, and causes actions associated with the actual acoustic event to be performed in response to determining that the detected acoustic event is an actual acoustic event. In some implementation embodiments, the assistant device that has detected the audio data is anticipated to detect audio data that capture the actual acoustic event based on previous acoustic events being detected in each of the assistant devices.SELECTED DRAWING: Figure 4

Description

[Background technology]

[0001] A person can engage in a person-to-computer interaction with an interactive software application referred to herein as an "automation assistant" (also known as a "chatbot," "two-way personal assistant," "intelligent personal assistant," "personal voice assistant," or "conversational agent"). For example, a person (who may be called a "user" when interacting with an automation assistant) may provide input to the automation assistant (e.g., commands, queries, and / or requests) which can cause the automation assistant to generate and provide response outputs, control one or more Internet of Things (IoT) devices, and / or perform one or more other functions. The input provided by the user may be spoken natural language input (i.e., utterances) which may, in some cases, be converted into text (or other semantic representations) and then further processed, and / or typed natural language input.

[0002] In some cases, an automated assistant may include an automated assistant client that runs locally on an assistant device and involves direct user interaction, as well as a cloud-based counterpart that leverages virtually unlimited cloud resources to help the automated assistant respond to user input. For example, the automated assistant may provide the cloud-based counterpart with audio recordings (or their textual transcriptions) of user utterances and, optionally, data indicating user identification information (e.g., certificates). The cloud-based counterpart may perform various operations on queries and return the results to the automated assistant client, which may then provide the corresponding output to the user.

[0003] Many users may interact with automation assistants using multiple assistant devices. For example, some users may have a coordinated “ecosystem” of assistant devices that can receive user input directed to the automation assistant and / or be controlled by the automation assistant, among other things, one or more smartphones, one or more tablet computers, one or more vehicle computing systems, one or more wearable computing devices, one or more smart televisions, one or more interactive standalone speakers, and / or one or more IoT devices. Users may engage in person-computer interaction with the automation assistant using any of these assistant devices (assuming the automation assistant client is installed and the assistant device is capable of receiving input). In some cases, these assistant devices may be scattered around the user’s main residence, second homes, workplace, and / or other structures. For example, mobile assistant devices such as smartphones, tablets, and smartwatches may be worn by the user and / or wherever the user last put them. Other assistant devices such as traditional desktop computers, smart televisions, interactive standalone speakers, and IoT devices may be more stationary, but still may be located in various places (e.g., rooms) in the user’s home or workplace. [Overview of the project] [Problems that the invention aims to solve]

[0004] Techniques exist to alert users (e.g., a single user, multiple users in a family, colleagues, roommates, etc.) when an acoustic event is detected in a given assistant device within the assistant device ecosystem. However, such techniques can lead to the detection of false positive events. In other words, a given assistant device may determine that an acoustic event has occurred when it did not actually occur. As a result, an automated assistant client of a given assistant device may cause one or more actions to be performed when the automated assistant client should not have performed any action. [Means for solving the problem]

[0005] The implementation described herein relates to the simultaneous detection of audio data capturing acoustic events in various assistant devices within an ecosystem comprising multiple assistant devices. Each audio data detected in each of the various assistant devices capturing acoustic events may be processed using an event detection model to generate a corresponding measure related to the acoustic event. Each of the corresponding measures may be processed to determine whether the acoustic event is an actual acoustic event. In response to determining that the acoustic event is an actual acoustic event, an action related to the acoustic event may be performed. Each audio data may be processed locally in the assistant device using the event detection model, processed remotely in a remote system using the event detection model, and / or processed in a given assistant device in the ecosystem (e.g., one that may or may not have detected audio data capturing an acoustic event) using the event detection model.

[0006] In some implementations, acoustic events can correspond to hotword events. In these implementations, an acoustic event detection model can correspond to a hotword detection model, which is trained to detect specific words or phrases that, when detected, cause one or more components and / or functions of an automation assistant to be activated in one or more assistant devices in the ecosystem. For example, suppose a given assistant device with a microphone is located in the home of a user related to the ecosystem. We further assume that the given assistant device detects audio data via its microphone, which captures acoustic events, and that the audio data is processed using a hotword detection model to generate probabilities indicating whether the audio data contains a specific word or phrase. Furthermore, suppose at least one additional assistant device with an additional microphone is located geographically close to the given assistant device in the ecosystem. We further assume that at least one additional assistant device detects additional audio data via its additional microphone, which similarly captures acoustic events, and that the additional audio data is processed using a hotword detection model to generate additional probabilities indicating whether the additional audio data contains a specific word or phrase. In this example, this probability and additional probabilities may be used to determine whether the acoustic event actually corresponds to the occurrence of that particular word or phrase.

[0007] For example, suppose the probability is 0.70 and the additional probability is 0.65. In this case, if both the probability and the additional probability meet the threshold (e.g., 0.60), the hotword event can be validated as a real hotword event, and one or more components and / or features of the automation assistant can be activated even if neither probability is very high. Alternatively, suppose the probability is 0.90 and the additional probability is 0.55. In this case, the additional probability may not meet the threshold, but since the indicated probability is relatively high (and optionally the additional probability is within the threshold range (e.g., 0.10)), the hotword event can still be validated as a real hotword event, and one or more components and / or features of the automation assistant can be activated.

[0008] In several additional or alternative implementations, acoustic events can correspond to sound events. In these implementations, the acoustic event detection model can correspond to a sound detection model, which is trained to detect one or more specific sounds that, when detected, cause notifications to be rendered visually and / or audibly on the user's client devices related to the ecosystem. Specific sounds may include, for example, glass breaking, a dog barking, a cat meowing, a doorbell ringing, a fire alarm going off, a carbon monoxide detector going off, a baby crying, a door knocking, and / or any other sounds that may occur in the ecosystem. In some versions of these implementations, a variety of sound detection models may be trained to detect each specific sound, while in other implementations, a single sound detection model may be trained to detect a variety of sounds. Audio data may be processed in the same or similar manner as described above with respect to hotword events, but additionally or alternatively, this may include processing the audio data using a sound detection model to verify whether an acoustic event is an actual sound event.

[0009] In various implementations, at least one additional assistant device in the ecosystem may be identified in response to a given assistant device detecting audio data capturing an acoustic event. If the acoustic event detected by a given assistant device is indeed a genuine acoustic event, then at least one additional assistant device identified in the ecosystem should have detected temporally corresponding audio data capturing the same acoustic event. Audio data captured by an assistant device may be considered temporally corresponding audio data based, for example, on the respective timestamps associated with the audio data captured by the assistant device. For example, audio data may be considered temporally corresponding audio data when the timestamps match or when the timestamps are within a threshold time period relative to each other (e.g., a few milliseconds, a few seconds, or any other appropriate length of time). In some versions of these implementations, at least one additional assistant device may be identified based, for example, on the fact that a given assistant device and at least one additional assistant device have previously detected audio data capturing the same acoustic event. In some additional or alternative versions of these implementations, at least one additional assistant device may be identified, for example, based on the fact that a given assistant device and at least one additional assistant device belong to the same group of assistant devices in the ecosystem's device topology representation.

[0010] By using the techniques described herein to detect and verify the occurrence of acoustic events using various assistant devices within the ecosystem, the amount of false positives for acoustic events can be reduced. As a result, both computational and network resources can be saved. For example, by using the techniques described herein to detect and verify the occurrence of hotword events within the ecosystem, automation assistant components and / or functions can be kept dormant when they might have been activated without these techniques, even though they were not intended to be activated. As another example, by using the techniques described herein to detect and verify the occurrence of specific sounds within the ecosystem, notifications presented to users related to the ecosystem can be withheld when they might have been presented without these techniques, even though no acoustic event actually occurred. Furthermore, by using the techniques described herein to predict which assistant devices within the ecosystem should detect the temporally corresponding audio data that captures an acoustic event, the number of assistant devices processing audio data can be reduced. As a result, both computational and network resources can be saved. For example, by using the techniques described herein when predicting which assistant device in the ecosystem should have detected an acoustic event, other assistant devices that may have captured temporally corresponding audio data that does not correspond to the acoustic event may not need to be considered when determining whether the acoustic event is an actual acoustic event.

[0011] The above description is given as an overview of only some of the implementations of this disclosure. Further descriptions of those implementations and other implementations are provided more in detail herein. As one example, various implementations are described more in detail in the claims included herein.

[0012] In addition, some implementations include one or more processors in one or more computing devices, the one or more processors being operable to execute instructions stored in associated memory, and the instructions being configured to cause the execution of any of the methods described herein. Some implementations also include one or more non-temporary computer-readable storage media storing computer instructions that can be executed by one or more processors to perform any of the methods described herein.

[0013] It should be understood that all combinations of the aforementioned concepts and any additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. [Brief explanation of the drawing]

[0014] [Figure 1] This is a block diagram of an exemplary environment in which the implementations disclosed herein may be implemented. [Figure 2A] This figure shows examples related to simultaneous acoustic event detection of hotword events in multiple assistant devices within an ecosystem, using various implementation forms. [Figure 2B] This figure shows examples related to simultaneous acoustic event detection of hotword events in multiple assistant devices within an ecosystem, using various implementation forms. [Figure 3] This figure shows examples related to the simultaneous detection of specific acoustic events in multiple assistant devices within an ecosystem, using various implementation forms. [Figure 4] This flowchart illustrates an exemplary method for determining whether an acoustic event is an actual acoustic event, based on audio data captured by multiple assistant devices within an ecosystem, using various implementation methods. [Figure 5] A flowchart showing an exemplary method of identifying which assistant device should have detected an acoustic event in an ecosystem based on the detection of the acoustic event by a given assistant device in various implementation forms. [Figure 6] A diagram showing an exemplary architecture of a computing device in various implementation forms.

DETAILED DESCRIPTION OF THE INVENTION

[0015] Smart multi-sensing network-connected devices (also referred to herein as assistant devices), such as smartphones, tablet computers, vehicle computing systems, wearable computing devices, smart televisions, two-way standalone speakers (with or without a display), sound speakers, home alarms, door locks, cameras, lighting systems, treadmills, thermostats, weighing scales, smart beds, sprinkler systems, garage door openers, household appliances, baby monitors, fire alarms, moisture meters, etc., are increasing rapidly. Often, multiple assistant devices are located within the boundaries of a structure such as a home, or within various related structures such as the user's main residence and the user's vacation home, the user's vehicle, and / or the user's workplace.

[0016] Furthermore, there has been a surge in assistant devices (also referred to herein as assistant input devices) that each contain an automated assistant client capable of forming a logical instance of an automated assistant. These assistant input devices may be dedicated solely to assistant functionality (e.g., a two-way standalone speaker and / or standalone audio / visual device that includes only the assistant client and associated interface and is dedicated solely to assistant functionality), or they may perform assistant functionality in addition to other functions (e.g., a mobile phone or tablet that includes the assistant client as one of its various applications). Moreover, some IoT devices can also be assistant input devices. For example, some IoT devices may include an automated assistant client and at least a speaker and / or microphone that (at least partially) serve as a user interface output and / or input device for the automated assistant client's assistant interface. Some assistant devices may not implement an automated assistant client or have means for interfacing with a user (e.g., a speaker and / or microphone), but they can still be controlled by an automated assistant (also referred to herein as non-assistant input devices). For example, a smart light bulb may not include an automation assistant client, speaker, and / or microphone, but commands and / or requests can be sent to the smart light bulb via the automation assistant to control its functions (e.g., turn the light on / off, dim it, change the color, etc.).

[0017] Various techniques have been proposed for adding assistant devices (including both assistant input devices and non-assistant input devices) to an assistant device ecosystem and / or for grouping assistant devices within the ecosystem. For example, when a new assistant device is added to the ecosystem, a user associated with the ecosystem can manually add the new assistant device to a group of assistant devices in the ecosystem via a software application (e.g., via an automated assistant application, a software application associated with the ecosystem, a software application associated with the new assistant device, etc.) in the device topology representation of the ecosystem. Further, when an assistant device moves within the ecosystem, the user can manually change the group to which the assistant device is assigned via a software application. Otherwise, the group to which the assistant device is assigned may not accurately reflect the location of the assistant device within the ecosystem. For example, a smart speaker labeled "Living Room Speaker" is located in the living room of the user's residence and is associated with the "Living Room" group of assistant devices, but if the smart speaker moves to the kitchen of the user's residence, the smart speaker may still be labeled "Living Room Speaker" and included in the "Living Room" group of assistant devices unless the user manually changes the label and group in the device topology representation for the ecosystem of the user's residence, even if the label and group do not represent the location of the assistant device.

[0018] A device topology representation may include a label (or unique identifier) ​​associated with each assistant device. Furthermore, a device topology representation can specify a label (or unique identifier) ​​associated with each assistant device. Device attributes for a given assistant device may, for example, indicate one or more input and / or output modes supported by each assistant device. For example, device attributes for an assistant client device that is a standalone speaker only may indicate that it can provide an auditory output but not a visual output. Additional or alternative device attributes for a given assistant device may, for example, identify one or more controllable states of a given assistant device, identify the party (e.g., first-party (1P) or third-party (3P)) that manufactures, distributes, and / or creates the assistant device's firmware, and / or identify a unique identifier for a given assistant device, such as an immutable identifier provided by the 1P or 3P, or a label assigned to a given assistant device by the user. According to the various implementations disclosed herein, the device topology representation can optionally further specify which smart devices can be locally controlled by which assistant devices, the local addresses of the locally controllable assistant devices (or the local addresses of the hubs that can directly locally control those assistant devices), the local signal strength, and / or other preferred indicators between each assistant device. Furthermore, according to the various implementations disclosed herein, the device topology representation (or a variation thereof) can be stored locally in each of the multiple assistant devices for use when locally controlling the assistant devices and / or locally assigning indicators to them. Moreover, the device topology representation can specify groups associated with each assistant device, which may be defined at various levels of granularity.For example, multiple smart lights in the living room of a user's home could be considered to belong to the "Living Room Lighting" group. Furthermore, if the living room also includes a smart speaker, all assistant devices located in the living room could be considered to belong to the "Living Room Assistant Devices" group.

[0019] An automated assistant can detect various events occurring in the ecosystem based on one or more signals generated by one or more assistant devices. For example, an automated assistant can process one or more signals to detect these events using an event detection model or event detection rules. Furthermore, an automated assistant can be configured to perform one or more actions based on outputs generated based on one or more signals for events occurring in the ecosystem. In some implementations, the events to be detected may be acoustic events captured through the microphones of one or more assistant devices. The automated assistant can be configured to process the audio data capturing the acoustic events using an acoustic event model. Acoustic events detected by an acoustic event model may include, for example, using a hotword detection model to detect hotwords that invoke an automated assistant in an utterance; using an ambient noise detection model to detect ambient noise in the ecosystem (and optionally, while utterance acceptance is active on one of the assistant devices); using a sound detection model to detect specific sounds in the ecosystem (e.g., glass breaking, a dog barking, a cat meowing, a doorbell ringing, a fire alarm going off, a carbon monoxide detector going off, a baby crying, a door knocking, and / or other acoustic events); and / or other acoustic-related events that may be detected using the respective acoustic event detection models. For example, suppose audio data is detected through each microphone of at least one of the assistant devices. In this example, the automated assistant may have the audio data processed by at least one hotword detection model on the assistant device to determine whether the audio data captures a hotword to invoke the automated assistant.Furthermore, the automated assistant may, as an additional or alternative, have the audio data processed by at least one ambient noise detection model of the assistant device to classify any ambient (or background) noise captured in the audio data into one or more heterogeneous semantic categories of ambient noise (e.g., movie or television sounds, cooking sounds, and / or other heterogeneous sound categories). In addition, the automated assistant may, as an additional or alternative, have the audio data processed by at least one sound detection model of the assistant device to determine whether any particular sound is captured in the audio data.

[0020] The implementations described herein relate to determining whether an acoustic event has actually occurred in an ecosystem based on the fact that temporally corresponding audio data is captured by the respective microphones of various assistant devices. These implementations further relate to processing the audio data locally in the assistant devices to generate a measure indicating whether an acoustic event has actually occurred. Furthermore, these implementations relate to identifying which assistant device in the ecosystem should have detected the temporally corresponding audio data based on a device topology representation of the ecosystem, and ensuring that when various assistant devices detect the temporally corresponding audio data, the temporally corresponding audio data is processed by their respective event detection models.

[0021] Figure 1 shows an exemplary environment in which the techniques disclosed herein may be implemented. The exemplary environment includes multiple assistant input devices 106 1-N (Also referred to herein simply as “Assistant Input Device 106”), one or more cloud-based automation assistant components 119, one or more assistant non-input systems 180, one or more assistant non-input devices 185 1-NThis includes the Assistant Input Device 106 (also referred to herein simply as “Assistant Non-Input Device 185”), the Device Activity Database 191, the Machine Learning ("ML") Model Database, and the Device Topology Database 193. The Assistant Input Device 106 and Assistant Non-Input Device 185 in Figure 1 may also be collectively referred to herein as “Assistant Devices.”

[0022] One or more (for example, all) of the assistant input devices 106 connect to their respective automation assistant clients 118 1-N Each instance of can be executed. However, in some implementations, one or more of the assistant input devices 106 are optional, and each automation assistant client 118 1-N It may lack an instance of and still include engines and hardware components (e.g., a microphone, speaker, speech recognition engine, natural language processing engine, speech synthesis engine, etc.) for receiving and processing user input directed to the automated assistant. Automated Assistant Client 118 1-N An instance of may be a separate application from the operating system of each assistant input device 106 (for example, installed "on top of" the operating system), or alternatively, may be implemented directly by the operating system of each assistant input device 106. As further described below, the automated assistant client 118 1-N Each instance is optional and corresponds to one of the respective user interface components 107 of the respective assistant input device 106. 1-N In responding to various requests provided by, it can interact with one or more cloud-based automation assistant components 119. Furthermore, as also described below, other engines of the assistant input device 106 can optionally interact with one or more cloud-based automation assistant components 119.

[0023] One or more cloud-based automation assistant components 119 may be implemented on one or more computing systems (for example, servers collectively referred to as "cloud" or "remote" computing systems) that are communicatively coupled to each assistant input device 106 via one or more local area networks (including "LANs," such as Wi-Fi LANs, Bluetooth networks, short-range communication networks, and mesh networks) and / or wide area networks (including "WANs," such as the Internet). The communicative coupling of the cloud-based automation assistant components 119 with the assistant input devices 106 is generally shown by 1101 in Figure 1. In some embodiments, the assistant input devices 106 may also be communicatively coupled to each other via one or more networks (for example, LANs and / or WANs), as generally shown by 1102 in Figure 1.

[0024] One or more cloud-based automated assistant components 119 may also be communicatively coupled to one or more assistant non-input systems 180 via one or more networks (e.g., LAN and / or WAN). The communicative coupling of the cloud-based automated assistant component 119 to the assistant non-input system 180 is generally shown by 1103 in FIG. 1. Further, each assistant non-input system 180 may be communicatively coupled to one or more (e.g., a group) of assistant non-input devices 185 via one or more networks (e.g., LAN and / or WAN). For example, the first assistant non-input system 180 may be communicatively coupled to and receive data from one or more first groups of assistant non-input devices 185, the second assistant non-input system 180 may be communicatively coupled to and receive data from one or more second groups of assistant non-input devices 185, and so on. The communicative coupling of the assistant non-input system 180 to the assistant non-input device(s)185 is generally shown by 1104 in FIG.

[0025] An instance of the automated assistant client 118 may, through its interaction with one or more of the cloud-based automated assistant components , form what appears to the user to be a logical instance of an automated assistant 120 that the user can interact with in a human-computer interaction scenario Two instances of such automated assistants are shown in FIG. 1 The first automated assistant 120A, enclosed by a dashed line, includes the automated assistant client 1181 of the assistant input device 1061, and one or more cloud-based automated assistant components . The second automated assistant 120B, enclosed by a dash-dash-dot line, is for the automated assistant client 118 of the assistant input device 106 N of the automated assistant client 118 Nand one or more cloud-based automation assistant components 119. Therefore, it should be understood that each user interacting with an automation assistant client 118 running on one or more of the assistant input devices 106 may, in effect, interact with a user-specific logical instance of the automation assistant 120 (or a logical instance of the automation assistant 120 shared among a household or other groups of users). For brevity and simplicity, the term “automation assistant” as used herein refers to a combination of an automation assistant client 118 running on one of each of the assistant input devices 106 and one or more cloud-based automation assistant components 119 (which may be shared among various automation assistant clients 118). Although only multiple assistant input devices 106 are shown in Figure 1, it should be understood that the cloud-based automation assistant components 119 are additional and can serve many additional groups of assistant input devices.

[0026] The assistant input device 106 may include, for example, one or more IoT devices capable of receiving user input directed to the automation assistant 120, such as a desktop computing device, laptop computing device, tablet computing device, mobile phone computing device, computing device in the user's vehicle (e.g., in-car communication system, in-car entertainment system, in-car navigation system), two-way standalone speaker (e.g., with or without a display), smart appliance such as a smart television, the user's wearable device including a computing device (e.g., the user's wristwatch with a computing device, the user's glasses with a computing device, a virtual reality or augmented reality computing device), and / or any IoT device. Additional and / or alternative assistant input devices may be provided. The assistant non-input device 185 may include many of the same devices as the assistant input device 106, but is not capable of receiving user input directed to the automation assistant 120 (e.g., does not include a user interface input component). The assistant non-input device 185 does not receive user input directed to the automation assistant 120, but the assistant non-input device 185 can still be controlled by the automation assistant 120.

[0027] In some implementations, multiple assistant input devices 106 and assistant non-input devices 185 can be associated with each other in various ways to facilitate the execution of the techniques described herein. For example, in some implementations, multiple assistant input devices 106 and assistant non-input devices 185 can be associated with each other by being communicatively coupled via one or more networks (for example, via network 110 in Figure 1). This may be the case, for example, when multiple assistant input devices 106 and assistant non-input devices 185 are deployed across a specific area or environment, such as a home or building. Additionally or alternatively, in some implementations, multiple assistant input devices 106 and assistant non-input devices 185 can be associated with each other by being members of a coordinated ecosystem that is at least selectively accessible by one or more users (for example, an individual, family, an employee of an organization, or other predetermined group). In some of those implementations, the ecosystem of multiple assistant input devices 106 and assistant non-input devices 185 can be associated with each other manually and / or automatically in a device topology representation of the ecosystem stored in a device topology database 193.

[0028] The assistant non-input system 180 may include one or more first-party (1P) systems and / or one or more third-party (3P) systems. A 1P system refers to a system controlled by the same party that controls the automated assistant 120 as referred herein. A 3P system, as used herein, refers to a system controlled by a party other than the party that controls the automated assistant 120 as referred herein.

[0029] The Assistant Non-Input System 180 can receive data from the Assistant Non-Input Device 185 and / or one or more cloud-based automation assistant components 119 that are communicably coupled to it (for example, via the network 110 in Figure 1), and can selectively transmit data (e.g., status, status changes, and / or other data) to the Assistant Non-Input Device 185 and / or one or more cloud-based automation assistant components 119. For example, suppose the Assistant Non-Input Device 1851 is a smart doorbell IoT device. In response to an individual pressing a button on the doorbell IoT device, the doorbell IoT device can transmit data corresponding to one of the Assistant Non-Input Systems 180 (for example, one of the Assistant Non-Input Systems managed by the doorbell manufacturer, which may be a 1P system or a 3P system). One of the Assistant Non-Input Systems 180 can determine a status change of the doorbell IoT device based on such data. For example, one of the assistant non-input systems 180 can determine a change in the doorbell state from an inactive state (e.g., no recent button press) to an active state (recent button press), and the change in doorbell state can be transmitted (e.g., via the network 110 in Figure 1) to one or more cloud-based automation assistant components 119 and / or assistant input devices 106. In particular, user input is received at the assistant non-input device 1851 (e.g., a button press on the doorbell), but the user input is not directed to the automation assistant 120 (hence the term "assistant non-input device"). As another example, suppose the assistant non-input device 1851 is a smart thermostat IoT device with a microphone, but the smart thermostat does not include an automation assistant client 118. An individual can operate the smart thermostat (e.g., using touch input or spoken input) to change the temperature, set specific values ​​as setpoints for controlling the HVAC system via the smart thermostat, etc.However, unless the smart thermostat includes an automated assistant client 118, individuals cannot communicate directly with the automated assistant 120 through the smart thermostat.

[0030] In various implementations, one or more cloud-based automation assistant components 119 may further include various engines. For example, as shown in Figure 1, one or more cloud-based automation assistant components 119 may further include an event detection engine 130, a device identification engine 140, an event processing engine 150, a semantic indicator engine 160, and a query / command processing engine 170. These various engines are shown as one or more cloud-based automation assistant components 119 in Figure 1, but it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, an assistant input device 106 and / or an assistant non-input device 185 may include one or more of these various engines. As another example, these various engines may be distributed across the assistant input devices 106, and an assistant non-input device 185 may include one or more of these various engines and / or one or more cloud-based automation assistant components 119.

[0031] In some implementations, the event detection engine 130 can detect various events occurring in the ecosystem. In some versions of these implementations, the event detection engine 130 can detect when a given assistant input device 106 and / or a given assistant non-input device 185 (for example, a given assistant device) is newly added to the ecosystem or moves within the ecosystem. For example, the event detection engine 130 can determine when a given assistant device is newly added to the ecosystem based on one or more wireless signals detected via the network 110 and via the device identification engine 140. For example, when a given assistant device is newly connected to one or more of the networks 110, the given assistant device may broadcast a signal indicating that it is newly added to the network 110. As another example, the event detection engine 130 can determine when a given assistant device has moved within the ecosystem based on one or more wireless signals detected via the network 110. In these examples, the device identification engine 140 can process signals to determine that one of the given assistant devices is newly added to the network 110 and / or that one of the given assistant devices has moved within the ecosystem. One or more wireless signals detected by the device identification engine 140 may be, for example, network signals and / or acoustic signals that are imperceptible to humans and optionally include the unique identifiers of one of the given assistant devices and / or other assistant devices that are geographically close to one of the given assistant devices. For example, when one of the given assistant devices moves within the ecosystem, the device identification engine 140 may detect one or more wireless signals transmitted by other assistant devices that are geographically close to one of the given assistant devices.By processing these signals, it is possible to determine that one or more other assistant devices that are spatially close to one given assistant device are different from one or more assistant devices that were previously spatially close to one given assistant device.

[0032] In some further versions of these implementations, the automation assistant 120 may enable one given assistant device that is newly added to or moved within the ecosystem to be assigned to a group of assistant devices (for example, in the device topology representation of the ecosystem stored in the device topology database 193). For example, in an implementation where one given assistant device is newly added to the ecosystem, the given assistant device may be added to an existing group of assistant devices, or a new group of assistant devices may be created that includes the given assistant device. For example, if one given assistant device is geographically close to several assistant devices belonging to the "Kitchen" group (e.g., a smart oven, a smart coffee maker, a two-way standalone speaker associated with a unique identifier or sign indicating its location in the kitchen, and / or other assistant devices), the given assistant device may be added to the "Kitchen" group, or a new group may be created. As another example, in an implementation where one given assistant device moves within the ecosystem, the given assistant device may be added to an existing group of assistant devices, or a new group of assistant devices may be created that includes the given assistant device. For example, if one given assistant device was geographically close to several assistant devices belonging to the aforementioned "kitchen" group, but is now geographically close to several assistant devices belonging to the "garage" group (e.g., smart garage door, smart door lock, and / or other assistant devices), then one given assistant device may be removed from the "kitchen" group and added to the "garage" group.

[0033] In some additional or alternative versions of these implementations, the event detection engine 130 can detect the occurrence of an acoustic event. The occurrence of an acoustic event may be detected based on audio data received in one or more of the assistant input devices 106 and / or one or more of the assistant non-input devices 185 (for example, one or more of the assistant devices). The audio data received in one or more of the assistant devices may be processed by an event detection model stored in the ML model database 192. In these implementations, each of the one or more assistant devices that detect the occurrence of an acoustic event includes its own microphone.

[0034] In some further versions of these implementations, the occurrence of an acoustic event may include ambient noise captured in audio data in one or more assistant devices (and optionally, only the occurrence of ambient noise detected when speech acceptance is active in one or more assistant devices). Ambient noise detected in each of the one or more assistant devices may be stored in the device activity database 191. In these implementations, the event processing engine 150 may process ambient noise detected in one or more assistant devices using an ambient noise detection model that is trained to classify ambient noise into one or more heterogeneous semantic categories based on a scale generated when processing ambient noise using the ambient noise detection model. The heterogeneous categories may include, for example, a category of movie or television sounds, a category of cooking sounds, a category of music sounds, a category of garage or workshop sounds, a category of courtyard sounds, and / or other heterogeneous sound categories that have semantic meaning. For example, if the event processing engine 150 determines that ambient noise processed using the ambient noise detection model includes sounds corresponding to a microwave oven humming, food being cooked in a frying pan, or a food processor processing food, the event processing engine 150 can classify the ambient noise into the cooking sounds category. As another example, if the event processing engine 150 determines that ambient noise processed using the ambient noise detection model includes sounds corresponding to the operation of a circular saw or hammering, the event processing engine 150 can classify the ambient noise into the garage or workshop category. The classification of ambient noise detected in a particular device can also be used as a device-specific signal used when inferring semantic markers for assistant devices (for example, as described with respect to the semantic marker engine 160).

[0035] In some additional or alternative versions of these further implementations, the occurrence of an acoustic event may include a hotword or specific sound detected in one or more assistant devices. In these implementations, the event processing engine 150 may process audio data detected in one or more assistant devices using a hotword detection model that is trained to determine whether the audio data contains a specific word or phrase that invokes the automated assistant 120, based on a scale generated when processing the audio data using the hotword detection model. For example, the event processing engine 150 may process the audio data to determine whether the audio data captures a user utterance containing “assistant,” “hey assistant,” “OK assistant,” and / or any other word or phrase that invokes the automated assistant. Furthermore, the scale generated using the hotword detection model may include a confidence level or probability indicating whether the audio data contains a word or phrase that invokes the automated assistant 120. In some versions of these implementations, the event processing engine 150 may determine that the audio data captures a word or phrase if the scale meets a threshold. For example, if the event processing engine 150 generates a metric of 0.70 associated with audio data that captures a word or phrase that invokes the automation assistant 120, and the threshold is 0.65, the event processing engine 150 may determine that the audio data captures a word or phrase that invokes the automation assistant 120. The hotword detection model may be trained using, for example, supervised learning techniques. For example, multiple training instances may be obtained. Each training instance may include a training instance input containing audio data (or features of the audio data, such as Mel frequency cepstrum coefficients, audio waveforms, Mel bank features, and / or other acoustic features), and a corresponding training instance output containing a marking indicating whether the training instance input contains a particular word or phrase that invokes the automation assistant 120.For example, if the training instance input contains audio data that captures a specific word or phrase, the corresponding training instance output may be assigned a label (e.g., "yes") or a value (e.g., "1") indicating that the training instance input contains the specific word or phrase, while if the training instance input does not contain the specific word or phrase, the corresponding training instance output may be assigned a different label (e.g., "no") or a value (e.g., "0").

[0036] In these implementations, the event processing engine 150 may, additionally or alternatively, process audio data detected in one or more assistant devices using a sound detection model that is trained to determine whether the audio data contains a particular sound based on a scale generated when processing the audio data using the sound detection model. Specific sounds may include, for example, glass breaking, a dog barking, a cat meowing, a doorbell ringing, a fire alarm sounding, a carbon monoxide detector sounding, a baby crying, or a door knocking. For example, the event processing engine 150 may process the audio data to determine whether the audio data captures any of these specific sounds. In this example, a single sound detection model may be trained to determine whether multiple specific sounds are captured in the audio data, or multiple sound detection models may be trained to determine whether a given specific sound is captured in the audio data. Furthermore, the scale generated using the sound detection model may include a confidence level or probability indicating whether the audio data contains a particular sound. In some versions of these implementations, the event processing engine 150 can determine that audio data captures a particular sound if the scale meets a threshold. For example, if the event processing engine 150 generates a scale of 0.70 associated with audio data that captures the sound of glass breaking, and the threshold is 0.65, the event processing engine 150 may determine that the audio data captures the sound of glass breaking.

[0037] In various implementations, the occurrence of an acoustic event can be detected by various assistant devices within the ecosystem. For example, various assistant devices in the environment can detect temporally corresponding audio data (for example, temporally corresponding in that the respective audio data is detected by the various assistant devices at the same time or within a time period of a threshold length). In these implementations, in response to a given assistant device detecting audio data in the ecosystem, the device identification engine 140 can identify one or more additional assistant devices that should have similarly detected temporally corresponding audio data that similarly captures an acoustic event. For example, based on the fact that one or more additional assistant devices have previously detected temporally corresponding audio data that similarly captures an acoustic event, the device identification engine 140 can identify one or more additional assistant devices that should have similarly detected temporally corresponding audio data that similarly captures an acoustic event. In other words, the device identification engine 140 can predict that, since a given assistant device and one or more additional assistant devices have previously captured temporally corresponding audio data containing the same acoustic event (as described with respect to, for example, Figures 2A, 2B, and 3), one or more additional assistant devices should similarly capture audio data containing the same acoustic event.

[0038] In various implementations, one or more device-specific signals generated or detected by each assistant device may be stored in the device activity database 191. In some implementations, the device activity database 191 may correspond to a portion of memory dedicated to the device activity of that particular assistant device. In some additional or alternative implementations, the device activity database 191 may correspond to the memory of a remote system communicating with the assistant device (for example, via the network 110 in Figure 1). This device activity may be used when generating semantic label candidates for one of the assistant devices (for example, as described with respect to the semantic label engine 160). Device activity may include, for example, queries or requests received by each assistant device (and / or semantic categories associated with each of multiple queries or requests), commands executed by each assistant device (and / or semantic categories associated with each of multiple commands), ambient noise detected by each assistant device (and / or semantic categories associated with various instances of ambient noise), unique identifiers or labels of any assistant devices geographically close to a given assistant device (for example, identified via the event detection engine 140), and / or any other data received, generated, and / or executed by each assistant device.

[0039] In some implementations, the semantic labeling engine 160 can process one or more device-specific signals to generate semantic labeling candidates for one of the given assistant devices (e.g., one of the given assistant input devices 106 and / or one of the given assistant non-input devices 185) based on one or more device-specific signals. The semantic labeling candidates may be generated using one or more rules (optionally heuristically defined) or a machine learning model (e.g., stored in the ML model database 192). For example, one or more heuristically defined rules may indicate that semantic labeling candidates should be generated for each of the semantic categories to which one or more device-specific signals are classified. For example, suppose the device-specific signals are classified into the "kitchen" category, the "cooking" category, the "bedroom" category, and the "living room" category. In this example, the semantic label candidates may include a first semantic label candidate, “kitchen assistant device,” a second semantic label candidate, “cooking assistant device,” a third semantic label candidate, “bedroom assistant device,” and a fourth semantic label, “living room assistant device.” As another example, one or more device-specific signals (or one or more corresponding semantic categories) may be processed using a machine learning model trained to generate semantic label candidates. For example, the machine learning model may be trained based on multiple training instances. Each training instance may include a training instance input and a corresponding training instance output. The training instance input may include, for example, one or more device-specific signals and / or one or more semantic categories, and the corresponding training instance output may include, for example, a ground truth output corresponding to the semantic label to be assigned based on the training instance input.

[0040] In some versions of these implementations, a given assistant device can be identified for which a semantic marker candidate is generated in response to a decision that the given assistant device is newly added to the ecosystem and / or moves within the ecosystem. In some additional or alternative versions of these implementations, a given assistant device for which a semantic marker candidate is generated may be identified periodically (e.g., once a month, once every six months, once a year, etc.). In some additional or alternative versions of these implementations, a given assistant device for which a semantic marker candidate is generated may be identified in response to a decision that the purpose of the part of the ecosystem in which the given assistant device is located has been changed (e.g., a room in the main house of the ecosystem has been changed in purpose from a study to a bedroom). In these implementations, a given assistant device can be identified using the event detection engine 130. Identifying a given assistant device in these and other ways is described with reference to Figures 2A and 2B.

[0041] In some implementations, the semantic labeling engine 160 can select a given semantic label for a given assistant device from among semantic labeling candidates based on one or more device-specific signals. In implementations where semantic labeling candidates for a given assistant device are generated based on queries, requests, and / or commands stored in the device activity database 191 (or corresponding text), the queries, requests, and / or commands may be processed using a semantic classifier (for example, stored in the ML model database 192) to index the device activity for a given assistant device into one or more different semantic categories corresponding to heterogeneous queries, requests, and / or commands. Semantic labeling candidates may also be generated based on the semantic categories to which queries, commands, and / or requests are classified, and a given semantic label selected for a given assistant device may be selected based on the number of queries, requests, and / or commands classified in a given semantic category. For example, suppose a given assistant device has previously received nine queries regarding retrieving cooking recipes and two commands regarding controlling smart lighting in the ecosystem. In this example, candidate semantic labels could include, for example, a first semantic label "kitchen device" and a second semantic label "smart lighting control device". Furthermore, the semantic labeling engine 160 may select the first semantic label "kitchen device" as the given semantic label for the given assistant device because past use of the given assistant device indicates that it is primarily used for cooking-related activities.

[0042] In some implementations, the semantic classifier stored in the ML model database 192 may be a natural language understanding engine (for example, implemented by the NLP module 122 described below). Intentions determined based on the processing of queries, commands, and / or requests previously received by the assistant device may be mapped to one or more semantic categories. In particular, the multiple heterogeneous semantic categories described herein may be defined at various levels of granularity. For example, a semantic category may be associated with a subcategory of smart device commands, and / or a type category of such subcategory class, such as a category of smart lighting commands, a category of smart thermostat commands, and / or a category of smart camera commands. In other words, each category may have its own set of intentions associated with each category, determined by the semantic classifier, but some intentions of a category may also be associated with additional categories. In some additional or alternative implementations, a semantic classifier stored in the ML model database 192 may be used to generate text embeddings (e.g., low-dimensional representations such as word2vec representations) corresponding to the text of queries, commands, and / or requests. These embeddings may be points in the embedding space such that semantically similar words or phrases are associated with the same or similar parts of the embedding space. Furthermore, these parts of the embedding space may be associated with one or more heterogeneous semantic categories, and a given embedding may be classified into one of the given semantic categories if the distance measure between the given embedding and one or more parts of the embedding space satisfies a distance threshold. For example, a word or phrase related to cooking may be associated with a first part of the embedding space associated with the semantic marker “cooking,” a word or phrase related to weather may be associated with a second part of the embedding space associated with the semantic marker “weather,” and so on.

[0043] In implementations where one or more device-specific signals include ambient noise activity as an addition or substitution, instances of ambient noise may be processed using an ambient noise detection model (e.g., stored in the ML model database 192) to index device activity for a given assistant device into one or more different semantic categories corresponding to heterogeneous ambient noise. Candidate semantic labels may be generated based on the semantic categories to which instances of ambient noise are classified, and a given semantic label selected for a given assistant device may be selected based on the amount of instances of ambient noise classified in a given semantic category. For example, suppose the ambient noise detected in a given assistant device (and optionally only when speech recognition is active) mainly includes ambient noise that is classified as cooking sounds. In this example, the semantic labeling engine 160 may select the semantic label "kitchen device" as a given semantic label for a given assistant device because the ambient noise captured in the audio data indicates that the device is located near cooking-related activity.

[0044] In some implementations, ambient noise detection models stored in the ML model database 192 can be trained to detect specific sounds, and based on the outputs generated across the ambient noise detection models, it can be determined whether an instance of ambient noise contains a specific sound. The ambient noise detection models can be trained, for example, using supervised learning techniques. For example, multiple training instances can be obtained. Each training instance may include a training instance input containing ambient noise and a corresponding training instance output that includes a marking indicating whether the training instance input contains a specific sound that the ambient noise detection model is trained to detect. For example, if an ambient noise detection model is trained to detect the sound of breaking glass, training instances containing the sound of breaking glass may be assigned a mark (e.g., "yes") or a value (e.g., "1"), while training instances that do not contain the sound of breaking glass may be assigned a different mark (e.g., "no") or a value (e.g., "0"). In some additional or alternative implementations, ambient noise detection models stored in the ML model database 192 may be used to generate audio embeddings (e.g., low-dimensional representations of instances of ambient noise) based on instances of ambient noise (or their acoustic features, such as Mel-frequency cepstrum coefficients, raw audio waveforms, and / or other acoustic features). These embeddings may be points in the embedding space such that similar sounds (or acoustic features that capture sounds) are associated with the same or similar parts of the embedding space. Furthermore, these parts of the embedding space may be associated with one or more of several heterogeneous semantic categories, and a given embedding may be classified into one of the given semantic categories if the measure of distance between a given embedding and one or more parts of the embedding space satisfies a distance threshold. For example, an instance of glass breaking may be associated with a first part of the embedding space associated with the sound of "glass breaking," an instance of a doorbell ringing may be associated with a second part of the embedding space associated with the sound of "doorbell," and so on.

[0045] In an implementation where one or more device-specific signals include, as an addition or alternative, unique identifiers or labels of additional assistant devices located near a given assistant device, semantic label candidates may be generated based on those unique identifiers or labels, and a given semantic label selected for a given assistant device may be selected based on one or more of the unique identifiers or labels of the additional assistant devices. For example, suppose a first label, “smart oven,” is associated with a first assistant device located near a given assistant device, and a second label, “smart coffee maker,” is associated with a second assistant device located near a given assistant device. In this example, the semantic label engine 160 can select the semantic label “kitchen device” as a given semantic label for a given assistant device because the labels associated with the additional assistant devices located near a given assistant device are related to cooking. Unique identifiers or labels may be processed using semantic classifiers stored in the ML model database 192 in the same or similar manner as described above in relation to processing queries, commands, and / or requests.

[0046] In some implementations, the semantic marking engine 160 can automatically assign a given semantic marking to a given assistant device in the device topology representation of the ecosystem (stored, for example, in the device topology database 193). In some additional or alternative implementations, the semantic marking engine 160 can cause the automation assistant 120 to generate a prompt containing semantic marking candidates. The prompt can ask a user associated with the ecosystem to select one of the marking candidates as a given semantic marking. Furthermore, the prompt can be rendered visually and / or audibly on one of the given assistant devices (which may or may not be the given assistant device to which the given semantic marking is assigned) and / or on the user's client device (e.g., a mobile device). In response to receiving a selection of one of the marking candidates as a given semantic marking, the selected given semantic marking can be assigned to a given assistant device in the device topology representation of the ecosystem (stored, for example, in the device topology database 193). In some versions of these implementations, a given semantic marker assigned to a given assistant device may be added to the list of semantic markers for that given assistant device. In other words, multiple semantic markers may be associated with a given assistant device. In other versions of these implementations, a given semantic marker assigned to a given assistant device may supersede any other semantic marker for that given assistant device. In other words, only a single semantic marker may be associated with a given assistant device.

[0047] In some implementations, the query / command processing engine 170 can process queries, requests, or commands directed to the automation assistant 120 and received via one or more assistant input devices 106. The query / command processing engine 170 can process the queries, requests, or commands and select one or more assistant devices to satisfy the query or command. In particular, one or more assistant devices selected to satisfy the query or command may differ from one or more assistant input devices 106 that received the query or command. The query / command processing engine 170 can select one or more assistant devices to satisfy the utterance based on one or more criteria. One or more criteria may include, for example, the proximity of one or more devices to the user who provided the utterance (determined, for example, using the presence sensor 105 described below), the capabilities of one or more devices in the ecosystem, semantic markers assigned to one or more assistant devices, and / or other criteria for selecting an assistant device to satisfy the utterance.

[0048] For example, suppose a display device is required to satisfy a speech utterance. In this example, the candidate assistant devices considered when selecting a given assistant device to satisfy the speech utterance may be limited to those that include a display device. If multiple assistant devices in the ecosystem include a display device, the given assistant device that includes a display device and is closest to the user may be selected to satisfy the speech utterance. In contrast, in an implementation where only a speaker is required to satisfy a speech utterance (for example, no display device is required to satisfy a speech utterance), the candidate assistant devices considered when selecting a given assistant device to satisfy the speech utterance may include those that have a speaker, regardless of whether they include a display device.

[0049] As another example, suppose an utterance contains semantic characteristics that match semantic markers assigned to a given assistant device. The query / command processing engine 170 can determine that the semantic characteristics of an utterance match the semantic markers assigned to a given assistant device by generating first embeddings corresponding to one or more words of the utterance (or the corresponding text) and second embeddings corresponding to one or more words of the semantic markers assigned to the given assistant device, comparing those embeddings, and determining whether a distance measure between those embeddings satisfies a distance threshold indicating that the embeddings match (whether it is an exact match or a rough match, for example). In this example, the query / command processing engine 170 can select a given assistant device to satisfy the utterance based on the fact that the utterance matches the semantic markers (optionally, in addition to or instead of the proximity of the user who provided the utterance to the given assistant device). In this way, the selection of an assistant device to satisfy speech can be biased toward semantic markers assigned to assistant devices as described herein.

[0050] In various implementations, one or more assistant input devices 106 are configured to provide a signal indicating a detected presence, particularly the presence of a person, upon approval from the corresponding user, with one or more presence sensors 105 each. 1-NThis may include (also referred to herein simply as “presence sensor 105”). In some of their implementations, the automated assistant 120 may identify one or more of the assistant input devices 106, at least in part on the presence of a user, in order to satisfy an utterance from a user related to the ecosystem. The utterance may be satisfied by rendering (e.g., auditorily and / or visually) content to respond in one or more of the assistant input devices 106, thereby causing one or more of the assistant input devices 106 to be controlled on the basis of the utterance, and / or by causing one or more of the assistant input devices 106 to perform any other action to satisfy the utterance. As described herein, the automated assistant 120 may leverage data determined based on each presence sensor 105 when determining those assistant input devices 106 based on where the user is or has recently been, and provide corresponding commands only to those assistant input devices 106. In some additional or alternative implementations, the automation assistant 120 can leverage data determined based on each presence sensor 105 when determining whether a user (any user or a specific user) is currently near any of the assistant input devices 106, and can optionally suppress the provision of commands based on the determination that no user (any user or a specific user) is near any of the assistant input devices 106.

[0051] Each presence sensor 105 can take various forms. Some assistant input devices 106 may be equipped with one or more digital cameras configured to capture and provide signals indicating motion detected in their field of view. Additional or alternative, some assistant input devices 106 may be equipped with other types of light-based presence sensors 105, such as passive infrared ("PIR") sensors that measure infrared ("IR") light emitted from objects in their field of view. Additional or alternative, some assistant input devices 106 may be equipped with presence sensors 105 that detect sound waves (or pressure waves), such as one or more microphones. Furthermore, in addition to the assistant input devices 106, one or more assistant non-input devices 185 may, additionally or alternatively, include each of the presence sensors 105 described herein, and signals from such sensors may be additionally utilized by the automated assistant 120 when determining whether and / or how to satisfy a speech utterance according to the implementations described herein.

[0052] As an addition or alternative, in some implementations, the presence sensor 105 may be configured to detect other phenomena related to the presence of a person or device in the ecosystem. For example, in some embodiments, a given one of the assistant devices may be equipped with a presence sensor 105 that detects various types of wireless signals (e.g., radio waves, ultrasonic waves, electromagnetic waves, etc.) emitted by other assistant devices carried / operated by a particular user (e.g., mobile devices, wearable computing devices, etc.) and / or other assistant devices in the ecosystem (e.g., described with respect to the event detection engine 130). For example, some of the assistant devices may be configured to emit waves that are imperceptible to humans, such as ultrasonic or infrared waves, which can be detected by one or more of the assistant input devices 106 (e.g., via an ultrasonic / infrared receiver such as an ultrasonic-enabled microphone).

[0053] As an addition or alternative, various assistant devices may emit other types of waves that are imperceptible to humans, such as radio waves (e.g., Wi-Fi, Bluetooth, cellular, etc.), which can be detected by other assistant devices (e.g., mobile devices, wearable computing devices, etc.) carried / operated by a particular user and used to determine the specific location of the user operating them. In some implementations, Wi-Fi triangulation may be used, for example, to detect a person's location based on Wi-Fi signals to / from an assistant device. In other implementations, other wireless signal characteristics such as time-of-flight and signal strength may be used, individually or collectively, by various assistant devices to determine the location of a particular person based on signals emitted by other assistant devices carried / operated by a particular user.

[0054] As an addition or alternative, in some implementations, one or more of the assistant input devices 106 may perform speech recognition to recognize the user from the user's voice. For example, some instances of the automated assistant 120 may be configured to match the voice to a user profile, for example, for the purpose of providing / restricting access to various resources. In some implementations, the speaker's movement may then be determined, for example, by the presence sensor 105 of the assistant device. In some implementations, the user's location may be predicted based on such detected movement, and this location may be considered the user's location when some content is rendered on those assistant devices based at least in part on the proximity of the assistant devices to the user's location. In some implementations, the user may simply be considered as being at the last location where the user interacted with the automated assistant 120, especially if not much time has passed since that last interaction.

[0055] Each of the assistant input devices 106 further comprises its respective user interface component 107 1-N These include (also referred to herein simply as "user interface components 107"), each of which may include one or more user interface input devices (e.g., a microphone, a touchscreen, a keyboard) and / or one or more user interface output devices (e.g., a display, a speaker, a projector). For example, the user interface component 1071 of the assistant input device 1061 may include only a speaker and a microphone, but the assistant input device 106 N User interface component 107 N This may include a speaker, touchscreen, and microphone. Additionally or alternatively, in some implementations, the assistant non-input device 185 may include one or more user interface input devices and / or one or more user interface output devices of the user interface component 107, but the user input devices (if any) for the assistant non-input device 185 may not allow the user to directly interact with the automation assistant 120.

[0056] Each of the assistant input devices 106 and / or any other computing devices on which one or more of the cloud-based automation assistant components 119 operate may include one or more memories for storing data and software applications, one or more processors for accessing data and running applications, and other components to facilitate communication over a network. The operations performed by one or more of the assistant input devices 106 and / or the automation assistant 120 may be distributed across multiple computer systems. The automation assistant 120 may be implemented, for example, as a computer program running on one or more computers located in one or more locations connected to one another via a network (for example, one of the networks 110 in Figure 1).

[0057] As described above, in various implementations, each of the assistant input devices 106 can operate its own automated assistant client 118. In various implementations, each automated assistant client 118 operates its own speech capture / text-to-speech (TTS) / speech-to-text (STT) module 114 1-N This may include (also referred to simply as "Speech Capture / TTS / STT Module 114" in this specification). In other implementations, one or more aspects of each Speech Capture / TTS / STT Module 114 may be implemented separately from each automated assistant client 118.

[0058] Each speech capture / TTS / STT module 114 may be configured to perform one or more functions, including, for example, capturing user speech (speech capture, e.g., via each microphone (which may include a presence sensor 105 in some cases)), converting the captured audio into text and / or other representations or embeddings (STT) using speech recognition models stored in the ML model database 192, and / or converting text into speech (TTS) using speech synthesis models stored in the ML model database 192. Instances of these models may be stored locally in each of the assistant input devices 106 and / or accessible by the assistant input devices (e.g., via the network 110 in Figure 1). In some implementations, since one or more of the assistant input devices 106 may have relatively limited computing resources (e.g., processor cycles, memory, battery, etc.), each speech capture / TTS / STT module 114 located locally in each of the assistant input devices 106 may be configured to use a speech recognition model to convert a finite number of different spoken phrases into text (or other forms such as low-dimensional embeddings). Other speech inputs may be sent to one or more cloud-based automated assistant components 119, which may include a cloud-based TTS module 116 and / or a cloud-based STT module 117.

[0059] The cloud-based STT module 117 may be configured to leverage virtually unlimited cloud resources to convert audio data captured by the speech capture / TTS / STT module 114 into text (which can then be provided to the natural language processor 122) using speech recognition models stored in the ML model database 192. The cloud-based TTS module 116 may be configured to leverage virtually unlimited cloud resources to convert text data (e.g., text compiled by the automation assistant 120) into computer-generated speech output using speech synthesis models stored in the ML model database 192. In some implementations, the cloud-based TTS module 116 may provide the computer-generated speech output to one or more assistant devices, for example, so that it is output directly using the respective speakers of each assistant device. In other implementations, text data generated by the automation assistant 120 using a cloud-based TTS module 116 (for example, client device notifications included in commands) may be provided to the speech capture / TTS / STT module 114 of each assistant device, which then locally converts the text data into computer-generated speech using a speech synthesis model, so that the computer-generated speech is rendered through the local speaker of each assistant device.

[0060] The automation assistant 120 (and in particular, one or more cloud-based automation assistant components 119) may include a natural language processing (NLP) module 122, the aforementioned cloud-based TTS module 116, the aforementioned cloud-based STT module 117, and other components, some of which are described in more detail below. In some implementations, one or more of the engine and / or modules of the automation assistant 120 may be omitted, combined, and / or implemented in components separate from the automation assistant 120. An instance of the NLP module 122 may be implemented locally in the assistant input device 106, either as an addition or as an alternative.

[0061] In some implementations, the automated assistant 120 generates response content in response to various inputs generated by one user of the assistant input device 106 during a human-computer interaction session with the automated assistant 120. The automated assistant 120 may provide response content for presentation to the user (for example, via one or more of the network 110 in Figure 1 when separate from the assistant devices) as part of an interaction session via the assistant input device 106 and / or the assistant non-input device 185. For example, the automated assistant 120 may generate response content in response to free-form natural language input provided via one of the assistant input devices 106. As used herein, free-form input is user-organized input that is not constrained to a group of choices presented for user selection.

[0062] The NLP module 122 of the automated assistant 120 processes natural language input generated by the user via the assistant input device 106 and can produce annotated output for use by one or more other components of the automated assistant 120, the assistant input device 106, and / or the assistant non-input device 185. For example, the NLP module 122 can process free-form natural language input generated by the user via one or more user interface input devices of the assistant input device 106. The annotated output produced based on processing the free-form natural language input may include one or more annotations of the natural language input and, optionally, one or more (e.g., all) of the words in the natural language input.

[0063] In some implementations, the NLP module 122 is configured to identify and annotate various types of grammatical information in natural language input. For example, the NLP module 122 may include a portion of an utterance tagger configured to annotate words by their grammatical role. In some implementations, the NLP module 122 may, as an addition and / or alternative, include an entity tagger (not shown) configured to annotate references to entities in one or more segments, such as references to people (e.g., including literary figures, celebrities, public figures, etc.), organizations, locations (real and imaginary), etc. In some implementations, data about entities may be stored in one or more databases, such as a knowledge graph (not shown). In some implementations, the knowledge graph may include nodes representing known entities (and possibly entity attributes), as well as edges connecting the nodes to represent relationships between entities.

[0064] The entity tagger in NLP module 122 can annotate references to an entity at a high level of granularity (for example, to allow identification of all references to a certain entity class, such as people) and / or at a low level of granularity (for example, to allow identification of all references to a specific entity, such as a particular person). The entity tagger may rely on the content of natural language input to resolve about a particular entity, and / or may optionally communicate with a knowledge graph or other entity database to resolve about a particular entity.

[0065] In some implementations, the NLP module 122 may include, as an addition and / or alternative, a coreference resolver (not shown) configured to group, or "cluster," references into the same entity based on one or more contextual cues. For example, the coreference resolver may be used to resolve the word "it" in the natural language input "lock it" to "front door lock" based on the fact that "front door lock" is mentioned in a client device notification rendered immediately before receiving the natural language input "lock it".

[0066] In some implementations, one or more components of NLP module 122 may rely on annotations from one or more other components of NLP module 122. For example, in some implementations, a given entity tagger may rely on annotations from a coreference resolver and / or dependency analyzer when annotating all references to a particular entity. Also, for example, in some implementations, a coreference resolver may rely on annotations from a dependency analyzer when clustering references to the same entity. In some implementations, when processing a particular natural language input, one or more components of NLP module 122 may use relevant data outside of the particular natural language input, such as an assistant input device notification rendered immediately before the assistant input device notification receives the natural language input on which it is based, to determine one or more annotations.

[0067] Figure 1 illustrates a particular configuration of components implemented by an assistant device and / or server, and depicts an assistant device and / or server communicating over a particular network, but it should be understood that this is for illustrative purposes only and is not intended to be limiting. For example, an assistant input device 106 and an assistant non-input device may be directly coupled to communicate with each other directly over one or more networks (not shown). As another example, the operation of one or more cloud-based automation assistant components 119 may be implemented locally in one or more of the assistant input devices 106 and / or one or more of the assistant non-input devices. As yet another example, instances of various ML models stored in the ML model database 192 may be stored locally in the assistant device, and / or instances of device topology representations of the ecosystem stored in the device topology database 193 may be stored locally in the assistant input device. Furthermore, in implementations in which data (e.g., device activity, corresponding audio data or recognized text, device topology representation, and / or any other data as described herein) is transmitted over one or more networks 110 in Figure 1, the data may be encrypted, filtered, or otherwise protected in any manner to ensure user privacy.

[0068] By using the techniques described herein to detect and verify the occurrence of acoustic events using various assistant devices within the ecosystem, the amount of false positives for acoustic events can be reduced. As a result, both computational and network resources can be saved. For example, by using the techniques described herein to detect and verify the occurrence of hotword events within the ecosystem, automation assistant components and / or functions can be kept dormant when they might have been activated without these techniques, even though they were not intended to be activated. As another example, by using the techniques described herein to detect and verify the occurrence of specific sounds within the ecosystem, notifications presented to users related to the ecosystem can be withheld when they might have been presented without these techniques, even though no acoustic event actually occurred. Furthermore, by using the techniques described herein to predict which assistant devices within the ecosystem should detect the temporally corresponding audio data that captures an acoustic event, the number of assistant devices processing audio data can be reduced. As a result, both computational and network resources can be saved. For example, by using the techniques described herein when predicting which assistant device in the ecosystem should have detected an acoustic event, other assistant devices that may have captured temporally corresponding audio data that is unlikely to correspond to the acoustic event may not be considered when determining whether the acoustic event is an actual acoustic event.

[0069] An explanation of the various components of Figure 1 is given here with reference to Figures 2A, 2B, and 3. A floor plan of a house is illustrated in Figures 2A, 2B, and 3. The illustrated floor plan includes multiple rooms 250-262. Multiple assistant input devices 106 1-5However, it is deployed across at least several rooms. Assistant input device 106 1-5 Each of these may implement an instance of the automated assistant client 118 configured using a selected aspect of the present disclosure, and may include one or more input devices, such as a microphone, capable of capturing speech spoken by a nearby person. For example, a first assistant input device 1061, in the form of a two-way standalone speaker and display device (e.g., a display screen, projector, etc.), is deployed in room 250, which is the kitchen, in this example. A second assistant input device 1062, in the form of a so-called "smart" television (e.g., a networked television with one or more processors implementing each instance of the automated assistant client 118), is deployed in room 252, which is the study, in this example. A third assistant input device 1063, in the form of a two-way standalone speaker without a display, is deployed in room 254, which is the bedroom, in this example. A fourth assistant input device 1064, in the form of another two-way standalone speaker, is deployed in room 256, which is the living room, in this example. A fifth assistant input device 1065, which similarly takes the form of a smart television, is also deployed in room 250, which is the kitchen, in this example.

[0070] Although not shown in Figures 2A, 2B, and 3, there are multiple assistant input devices 106. 1-4These may be coupled together via one or more wired or wireless WANs and / or LANs (for example, via network 110 in Figure 1) in a manner that enables communication with each other and / or other resources (e.g., the Internet). In addition, other assistant input devices, in particular certain mobile devices such as smartphones, tablets, laptops, and wearable devices, may also be present, for example, carried by one or more people in a house, and may or may not be connected to the same WAN and / or LAN. The configuration of assistant input devices illustrated in Figures 2A, 2B, and 3 is just one example, and more or fewer, and / or different assistant input devices 106 may be deployed across any number of other rooms and / or areas of a house, and / or in locations other than a residence (e.g., offices, hotels, public places, airports, vehicles, and / or other locations or spaces).

[0071] Multiple Assistant Non-Input Devices 185 1-5 However, this is further illustrated in Figures 2A, 2B, and 3. For example, a first assistant non-input device 1851, in the form of a smart doorbell, is deployed outside the house near the front door. A second assistant non-input device 1852, in the form of a smart lock, is deployed outside the house adjacent to the front door. A third assistant non-input device 1853, in the form of a smart washing machine, is deployed in room 262, which is the laundry room in this example. A fourth assistant non-input device 1854, in the form of a door open / close sensor, is deployed near the back door of room 262 to detect whether the back door is open or closed. A fifth assistant non-input device 1855, in the form of a smart thermostat, is deployed in room 252, which is the study in this example.

[0072] Each of the assistant non-input devices 185 can communicate with its respective assistant non-input system 180 (shown in Figure 1) (for example, via the network 110 in Figure 1) to provide data to its respective assistant non-input system 180, and optionally the data can be controlled based on commands provided by its respective assistant non-input system 180. Additionally or alternatively, one or more of the assistant non-input devices 185 can communicate directly with one or more of the assistant input devices 106 (for example, via the network 110 in Figure 1) to provide data to one or more of the assistant input devices 106, and optionally the data can be controlled based on commands provided by one or more of the assistant input devices 106. The configuration of the assistant non-input device 185 shown in Figures 2A, 2B, and 3 is merely an example, and more or fewer, and / or different, assistant non-input devices 185 may be deployed across any number of other rooms and / or areas of the house, and / or in locations other than the residence (e.g., offices, hotels, public places, airports, vehicles, and / or other locations or spaces).

[0073] In various implementations, acoustic events may be detected by various assistant devices juxtaposed within the ecosystem (e.g., either assistant input device 106 and / or assistant non-input devices 185, each containing a microphone). For example, an acoustic event may be captured in audio data detected via the microphone of a given assistant device in the ecosystem, or in audio data detected via the microphone of at least one additional assistant in the ecosystem. The audio data captured by the assistant devices may be processed using an event detection model, stored locally in each assistant device and / or remotely on a server and accessible by each assistant device, to generate respective metrics related to the acoustic event. Furthermore, each of these metrics may be processed to determine whether the acoustic event corresponds to an actual acoustic event that occurred in the ecosystem. If the acoustic event is an actual acoustic event, an action can be taken based on the occurrence of an actual acoustic event in the ecosystem. Otherwise, the audio data can be discarded.

[0074] In some implementations, one or more assistant devices (e.g., a given assistant device and at least one additional assistant device) can process their respective audio data locally at each of these assistant devices, using their respective event detection models, in order to generate their respective metrics indicating whether each audio data detected at each of these assistant devices captures an acoustic event. In some additional or alternative implementations, one or more assistant devices can each send audio data to a given assistant device in the ecosystem (e.g., one of the assistant devices that detected audio data or a separate assistant device that did not detect any audio data) and process the audio data locally at the given assistant device, using its event detection model, in order to generate their respective metrics indicating whether each audio data detected at each of these assistant devices captures an acoustic event. In some additional or alternative implementations, one or more assistant devices can each send audio data to a remote system separate from the ecosystem and process the audio data remotely at the remote system, using its event detection model, in order to generate their respective metrics indicating whether each audio data detected at each of these assistant devices captures an acoustic event.

[0075] In some implementations, at least one additional assistant device in the ecosystem may be identified in response to a given assistant device detecting audio data capturing an acoustic event. If the acoustic event detected by a given assistant device is indeed a genuine acoustic event, then at least one additional assistant device identified in the ecosystem should have similarly detected temporally corresponding audio data capturing the acoustic event. Audio data captured by an assistant device may be considered temporally corresponding audio data based, for example, on the timestamps associated with the audio data captured by the assistant device. For example, audio data may be considered temporally corresponding audio data when their timestamps match or when they are within a threshold time period of each other (e.g., a few milliseconds, a few seconds, or any other appropriate length of time). In some versions of those implementations, at least one additional assistant device may be identified based, for example, on the fact that a given assistant device and at least one additional assistant device have previously detected audio data capturing the same acoustic event. In some additional or alternative versions of these implementations, at least one additional assistant device may be identified, for example, on the basis that a given assistant device and at least one additional assistant device belong to the same group of assistant devices in the ecosystem's device topology representation.

[0076] In some implementations, the acoustic event is a hotword event, and the event detection model corresponds to the hotword detection model. A hotword event may correspond to detecting a specific word or phrase that, when detected, causes one or more components or functions of the automation assistant to be activated. For example, referring in particular to Figures 2A and 2B, suppose user 101 associated with the ecosystem shown in Figures 2A and 2B provides the utterance “Hey Assistant…” which is detected by various assistant devices in the ecosystem. For example, as shown in Figure 2A, user 101 may be located in room 250, which in this example is the kitchen, when the utterance is given. In this example, suppose the audio data corresponding to the utterance “Hey Assistant…” is detected in the first assistant input device 1061 (for example, as indicated by the dashed line from the speech bubble corresponding to the utterance to the first assistant input device 1061). Furthermore, we further assume that a fifth assistant input device 1065 and a fifth assistant non-input device 1855 (assuming the smart thermostat includes a microphone) are also expected to detect audio data capturing hotwords (for example, as again indicated by dashed lines). In this example, the fifth assistant input device 1065 may be expected to detect audio data based on the fact that the first assistant input device 1061 and the fifth assistant input device 1065 belong to the same group of devices (e.g., the "kitchen" group), and / or based on the fact that the first assistant input device 1061 and the fifth assistant input device 1065 have previously detected temporally corresponding audio data capturing the same utterance.Furthermore, the fifth non-input assistant device 1855 may be expected to detect audio data based on the fact that the first input assistant device 1061 and the fifth non-input assistant device 1855 have previously detected temporally corresponding audio data that captures the same utterance (for example, assuming that the first input assistant device 1061 belongs to the "kitchen" group and the fifth non-input assistant device 1855 belongs to the "bedroom" group).

[0077] Furthermore, we assume that a scale generated based on audio data detected by the first assistant input device 1061 indicates that the audio data captures a hotword, a scale generated based on audio data detected by the fifth assistant input device 1065 indicates that the audio data captures a hotword, but a scale generated based on audio data detected by the fifth assistant non-input device 1855 indicates that the audio data does not capture a hotword. In this example, a hotword detected in the ecosystem may be determined to be a hotword, and one or more components or functions of the automated assistant can be activated even if the fifth assistant non-input device 1855 indicates that the audio data does not capture a hotword (or even if the fifth assistant non-input device 1855 does not detect any audio data at all) because other assistant devices indicate that the hotword was captured in the utterance. In contrast, assume that a scale generated based on audio data detected by the first assistant input device 1061 indicates that the audio data captures a hotword, while a scale generated based on audio data detected by the fifth assistant input device 1065 and the fifth assistant non-input device 1855 indicates that the audio data does not capture a hotword. In this example, it may be determined that a hotword detected in the ecosystem is not actually a hotword, and even if the first assistant input device 1061 indicates that the audio data captures a hotword, one or more components or functions of the automated assistant may remain dormant because other assistant devices indicate that the hotword was not captured in the utterance.

[0078] As another example, as shown in Figure 2B, user 101 may be located in room 252, which in this example is the study, where the utterance is given. In this example, we assume that audio data corresponding to the utterance "Hey Assistant..." is detected in the first Assistant Input Device 1061 (for example, as indicated by the dashed line from the speech bubble corresponding to the utterance to the first Assistant Input Device 1061). We further assume that a fifth Assistant Non-Input Device 1855 (assuming the smart thermostat includes a microphone) is also expected to detect audio data capturing the hotword (for example, also indicated by the dashed line). In this example, the fifth Assistant Non-Input Device 1855 may be expected to detect the audio data based on the fact that the second Assistant Input Device 1062 and the fifth Assistant Non-Input Device 1855 belong to the same group of devices (for example, the "Study" group), and / or based on the fact that the second Assistant Input Device 1065 and the fifth Assistant Non-Input Device 1855 have previously detected temporally corresponding audio data capturing the same utterance.

[0079] Furthermore, assume that a scale generated based on audio data detected by the second assistant input device 1062 indicates that the audio data captures a hotword, and that a scale generated based on audio data detected by the fifth assistant non-input device 1855 indicates that the audio data captures a hotword. In this example, a hotword detected in the ecosystem may be determined to be a hotword, which can activate one or more components or functions of the automation assistant. In contrast, assume that a scale generated based on audio data detected by the second assistant input device 1062 indicates that the audio data captures a hotword, but a scale generated based on audio data detected by the fifth assistant non-input device 1855 indicates that the audio data does not capture a hotword. In this example, whether a hotword detected in the ecosystem is determined to be a hotword may depend on whether a scale generated based on audio data detected by the fifth assistant non-input device 1855 falls within a threshold that indicates the scale represents a hotword. For example, assume that the scale associated with a hotword event is a probability. We further assume that the scale generated based on audio data detected by the second assistant input device 1062 is 0.7, the scale generated based on audio data detected by the fifth assistant non-input device 1855 is 0.6, and the threshold for determining that a hotword is an actual hotword is 0.65. As a result, an acoustic event may not be considered a hotword because the scale associated with the second assistant input device 1062 indicates the occurrence of a hotword, but the scale associated with the fifth assistant non-input device 1062 does not.However, we further assume that an acoustic event does not have to be considered a hotword if the scale generated based on the audio data detected by the fifth non-assistant input device 1855 (e.g., 0.6) is within 0.1 of the threshold (e.g., 0.65). Thus, an acoustic event may be considered a hotword occurrence even if neither device is certain that a hotword has been detected in the ecosystem. In contrast, suppose the scale generated based on the audio data detected by the fifth non-assistant input device 1855 is 0.1. In this example, even if the scale generated based on the audio data detected by the second assistant input device 1062 is 0.7, indicating a hotword, the acoustic event is not considered a hotword occurrence because the scale generated based on the audio data detected by the fifth non-assistant input device 1855 does not meet the threshold (e.g., 0.65) and is not within the threshold range. In these examples, additional or alternative processing of scales may be used to determine whether a hotword event is an actual hotword event. For example, one or more of the mean, median, highest, lowest, or percentile of a scale generated based on each audio data detected in the assistant device may be used to determine whether a hotword event is an actual hotword event. One or more of the scales are optional and may be weighted in determining one or more of these statistics based on, for example, proximity to the user 101 (determined using, for example, the presence sensor 105 in Figure 1), the type of components in the assistant device (e.g., an assistant device with multiple microphones vs. an assistant device with a single microphone), and / or other factors.

[0080] Figures 2A and 2B illustrate herein that a hotword event activating one or more automated assistant components or functions is an acoustic event, but it should be understood that this is for illustrative purposes only and is not intended to be limiting. Additionally or alternatively, hotword-free events may also be detected by various assistant devices within the ecosystem. A hotword-free event may include, for example, detecting a gaze directed towards one of the given assistant devices (and optionally combined with the mouth movements of user 101) such that, when detected, one or more automated assistant components or functions are activated. For example, each image data may be captured by multiple assistant devices (including, for example, vision components described in relation to presence sensors), each image data may be processed using a hotword-free model (for example, stored in the ML model database 192) to generate each scale, and each scale may be processed to determine whether a hotword-free event is an actual hotword-free event. These techniques may also be used to detect other visual events that may occur within the ecosystem.

[0081] Furthermore, while Figures 2A and 2B are described herein in relation to acoustic events being hotword events, it should be understood that these are illustrative and not intended to be limiting. As another non-limiting example, referring particularly to Figure 3, any audio data captured in the environment may, additionally or alternatively, be processed using a sound detection model trained to detect the occurrence of a particular sound in the ecosystem. For example, suppose (rather than user 101 providing an utterance) a window in room 250 is broken, and further assume that the audio data captured by the first assistant input device 1061 is processed using respective sound detection models trained to generate measures related to detecting the occurrence of broken glass (and optionally additional measures related to other sounds, such as a doorbell ringing, a fire alarm sounding, and / or other sounds that may occur in the ecosystem). As described above with respect to Figure 2A, additional assistant devices in the ecosystem (e.g., a fifth assistant input device 1065 and a fifth assistant non-input device 1855) may be identified as expected to detect temporally corresponding audio data, process the temporally corresponding audio data to generate an additional scale, and determine, based on the scale and the additional scale, whether a sound corresponding to glass breaking was actually captured in the ecosystem. As a result, a notification indicating that a sound of glass breaking was detected in the ecosystem may be rendered on user 101's client device (e.g., a mobile device and / or one or more assistant devices in the ecosystem), which may optionally include an indication of the assistant device that detected the glass breaking, and / or an indication of the room in the ecosystem where the sound of glass breaking was detected.

[0082] Figures 2A, 2B, and 3 illustrate acoustic (and visual) events as described herein, but it should be understood that these are illustrative and not intended to be limiting. The techniques described herein can be used to verify the occurrence of any event that may be detected by various assistant devices in the ecosystem. For example, suppose the first assistant device is a mobile device carried by user 101, which includes an accelerometer, and the second assistant device is a smartwatch worn by user 101, which also includes an accelerometer. In this example, the accelerometer data generated by the mobile device and the smartwatch, respectively, may be processed in the same or similar manner as described above to determine whether the detected accelerometer event is an actual accelerometer event (e.g., user 101 walking, jogging, running, and / or any other movement that may be detected by the accelerometer).

[0083] Figure 4 illustrates a flowchart illustrating an exemplary method 400 for determining whether an acoustic event is an actual acoustic event based on audio data captured by multiple assistant devices in the ecosystem. For convenience, the operation of method 400 is described in relation to the system that performs the operation. The system of method 400 includes one or more processors and / or other components of a computing device. For example, the system of method 400 may be implemented by the assistant input device 106 in Figure 1, Figure 2A, Figure 2B, or Figure 3, the assistant non-input device 185 in Figure 1, Figure 2A, Figure 2B, or Figure 3, the computing device 610 in Figure 6, one or more servers, other computing devices, and / or any combination thereof. Furthermore, the operation of method 400 is shown in a particular order, but this is not intended to be limiting. One or more operations may be rearranged, omitted, and / or added.

[0084] In block 452, the system detects audio data capturing acoustic events via the microphones of assistant devices located within an ecosystem that includes multiple assistant devices. The detected acoustic events may be, for example, specific words or phrases provided by users associated with the ecosystem (e.g., hotwords), specific sounds in the ecosystem that do not correspond to human speech, and / or other acoustic events that may occur in the ecosystem.

[0085] In block 454, the system processes audio data capturing acoustic events using an event detection model stored locally in the assistant device and generates a metric associated with the acoustic events. In some implementations, audio data capturing acoustic events may be processed using a hotword detection model stored locally in the assistant device to generate a metric indicating whether the audio data detected in the assistant device contains a specific word or phrase (for example, to activate one or more components of the automation assistant). In some additional or alternative implementations, audio data capturing acoustic events may be processed using a sound detection model stored locally in the assistant device to generate a metric indicating whether the audio data detected in the assistant device contains a specific sound (for example, glass breaking, a dog barking, a cat meowing, a doorbell ringing, a fire alarm sounding, a carbon monoxide detector sounding, a baby crying, a door knocking, and / or any other sound that may occur in the ecosystem).

[0086] In block 456, the system detects additional audio data that similarly captures acoustic events through additional microphones of additional assistant devices located within the ecosystem. The additional audio data detected by the additional assistant devices may be temporally corresponding to the audio data detected by the assistant devices. The system can determine that the additional audio data is temporally corresponding to the audio data based on the respective timestamps corresponding to when the audio data and the additional audio data were received, respectively. Furthermore, the additional audio data may be considered temporally corresponding to the audio data if the timestamps match or if the timestamps are within a threshold time period relative to each other (for example, within a few milliseconds, a few seconds, or any other appropriate length of time).

[0087] In block 458, the system processes additional audio data capturing acoustic events using additional event detection models stored locally in the additional assistant device to generate additional metrics related to the acoustic events. In some implementations, the additional audio data capturing acoustic events may be processed using additional hotword detection models stored locally in the additional assistant device to generate additional metrics indicating whether the additional audio data detected in the additional assistant device contains a specific word or phrase (for example, to activate one or more components of the automation assistant). In some additional or alternative implementations, the additional audio data capturing acoustic events may be processed using additional sound detection models stored locally in the additional assistant device to generate additional metrics indicating whether the additional audio data detected in the additional assistant device contains a specific sound (for example, glass breaking, a dog barking, a cat meowing, a doorbell ringing, a fire alarm sounding, a carbon monoxide detector sounding, and / or any other sound that may occur in the ecosystem).

[0088] In block 460, the system processes both a scale and an additional scale to determine whether an acoustic event detected by at least both the assistant device and the additional assistant device is an actual acoustic event. The scale and additional scale may be, for example, a binary value indicating whether the audio data captures an acoustic event (e.g., "1" for yes, "0" for no), a probability related to whether an acoustic event occurred (e.g., 0.7 for audio data capturing an acoustic event vs. 0.3 for audio data not capturing an acoustic event), and / or other scales related to the acoustic event.

[0089] In block 462, the system determines whether an acoustic event is a real acoustic event, based on the scale and additional scales it processed in block 460. If, in an iteration of block 462, the system determines that the acoustic event is not a real acoustic event, it proceeds to block 464 and discards the audio data and additional audio data without further processing. The system then returns to block 452. For example, suppose the scale and additional scale correspond to the probabilities that the audio data and additional audio data capture a hotword, respectively. For example, suppose the scale is assumed to be a 0.7 probability that the audio data captures a hotword, and the additional scale is assumed to be a 0.3 probability that the additional audio data captures a hotword. In this example, the system might determine that the acoustic event is a real acoustic event (e.g., the user provided an utterance containing a hotword) by assuming that the probability of 0.7, based solely on the scale, satisfies a threshold probability level. However, the system might determine that the acoustic event is not a real acoustic event (e.g., the user did not provide an utterance containing a hotword) by assuming that the probability of 0.3, based solely on the additional scale, does not satisfy a threshold probability level. As a result, the system may determine that an acoustic event is not an actual acoustic event even if the scale indicates that it was an actual acoustic event, because additional scales must also indicate that the acoustic event was an actual acoustic event, and the audio data and additional audio data may be discarded. Alternatively, the system may determine whether an acoustic event is an actual acoustic event based on statistics related to the scale, such as the mean, median, highest, lowest, or percentile of the scale, which are generated based on each audio data detected in an assistant device that can be used to determine whether an acoustic event is an actual acoustic event.

[0090] If the system determines in the iteration of block 462 that the acoustic event is a real acoustic event, the system proceeds to block 466. For example, suppose the scale and the additional scale correspond to the probabilities that the audio data and the additional audio data each capture a hotword. For example, suppose the scale is assumed to be a 0.7 probability that the audio data captures a hotword, and the additional scale is assumed to be a 0.8 probability that the additional audio data captures a hotword. In this example, the system may determine that the acoustic event is a real acoustic event (e.g., the user provided an utterance containing a hotword) by assuming that the probabilities of 0.7 and 0.8 (or one or more statistics based on them) satisfy a threshold probability level based on the scale and the additional scale, respectively. As a result, the system may determine that the acoustic event is a real acoustic event because both the scale and the additional scale indicate that the acoustic event was a real acoustic event.

[0091] In block 466, the system ensures that actions related to actual acoustic events are performed. The system then returns to block 452. The actions performed by the system may be based on actual acoustic events captured in the audio data and additional audio data. For example, in an implementation where an actual acoustic event corresponds to the occurrence of a hotword, the system may activate one or more components or functions of the automation assistant (e.g., speech processing, natural language processing, and / or other components or functions of the automation assistant). In another example, in an implementation where an actual acoustic event corresponds to the occurrence of a specific sound, the system may generate a notification indicating that a specific sound has been detected in the ecosystem (and optionally identify the assistant device that detected the specific sound), and the notification may be rendered on the user's client devices related to the ecosystem (e.g., the user's mobile device, one or more assistant devices, and / or any other client devices related to the user).

[0092] Figure 5 illustrates a flowchart illustrating an exemplary method 500 for identifying which assistant device in the ecosystem should have detected an acoustic event, based on the fact that a given assistant device detected an acoustic event. For convenience, the operation of method 500 is described in relation to the system that performs the operation. The system of method 500 includes one or more processors and / or other components of a computing device. For example, the system of method 500 may be implemented by the assistant input device 106 in Figure 1, Figure 2A, Figure 2B, or Figure 3, the assistant non-input device 185 in Figure 1, Figure 2A, Figure 2B, or Figure 3, the computing device 610 in Figure 6, one or more servers, other computing devices, and / or any combination thereof. Furthermore, the operation of method 500 is shown in a particular order, but this is not intended to be limiting. One or more operations may be rearranged, omitted, and / or added.

[0093] In block 552, the system detects audio data capturing acoustic events via the microphones of assistant devices located within an ecosystem that includes multiple assistant devices. The detected acoustic events may be, for example, specific words or phrases provided by users associated with the ecosystem (e.g., hotwords), specific sounds in the ecosystem that do not correspond to human speech, and / or other acoustic events that may occur in the ecosystem.

[0094] In block 554, the system identifies at least one additional assistant device that should have detected additional audio data, which is temporally corresponding to the audio data detected by the assistant device, through its respective microphone, based on the location of the assistant device within the ecosystem. The system can identify at least one additional assistant device that should have detected the additional audio data, based on the fact that the assistant device and at least one additional assistant device have previously detected the temporally corresponding audio data. Furthermore, the system can determine that the additional audio data is temporally corresponding to the audio data, based on the respective timestamps corresponding to when the audio data and the additional audio data were received, respectively. In addition, the additional audio data may be considered temporally corresponding to the audio data if the timestamps match or if the timestamps are within a threshold time length relative to each other (e.g., within a few milliseconds, a few seconds, or any other suitable length of time).

[0095] In block 556, the system determines whether an additional assistant device has detected additional audio data that is temporally corresponding to the audio data detected by the assistant device. If, in an iteration of block 556, the system determines that there is no additional audio data that is temporally corresponding to the audio data detected by the assistant device, the system proceeds to block 566 and discards the audio data and any additional audio data without further processing. If, in an iteration of block 556, the system determines that there is additional audio data that is temporally corresponding to the audio data detected by the assistant device, the system proceeds to block 558. In other words, if any other assistant devices that should have detected audio data capturing an acoustic event have not captured any temporally corresponding audio data, the system may determine that the acoustic event is not a real acoustic event without processing the audio data using the event detection model.

[0096] In block 558, the system processes audio data capturing acoustic events using an event detection model stored locally in the assistant device to generate a measure associated with the acoustic events. In some implementations, the audio data capturing acoustic events may be processed using a hotword detection model stored locally in the assistant device to generate a measure indicating whether the audio data detected in the assistant device contains a particular word or phrase (for example, to activate one or more components of the automation assistant). In some additional or alternative implementations, the audio data capturing acoustic events may be processed using a sound detection model stored locally in the assistant device to generate a measure indicating whether the audio data detected in the assistant device contains a particular sound (for example, glass breaking, a dog barking, a cat meowing, a doorbell ringing, a fire alarm sounding, a carbon monoxide detector sounding, a baby crying, a door knocking, and / or any other sound that may occur in the ecosystem).

[0097] In block 560, the system processes additional audio data capturing acoustic events using additional event detection models stored locally in the additional assistant device to generate additional metrics related to the acoustic events. In some implementations, the additional audio data capturing acoustic events may be processed using additional hotword detection models stored locally in the additional assistant device to generate additional metrics indicating whether the additional audio data detected in the additional assistant device contains a specific word or phrase (e.g., for activating one or more components of the automation assistant). In some additional or alternative implementations, the additional audio data capturing acoustic events may be processed using additional sound detection models stored locally in the additional assistant device to generate additional metrics indicating whether the additional audio data detected in the additional assistant device contains a specific sound (e.g., glass breaking, a dog barking, a cat meowing, a doorbell ringing, a fire alarm sounding, a carbon monoxide detector sounding, and / or any other sound that may occur in the ecosystem).

[0098] In block 562, the system processes both a scale and an additional scale to determine whether an acoustic event detected by at least both the assistant device and the additional assistant device is an actual acoustic event. The scale and additional scale may be, for example, a binary value indicating whether the audio data captures an acoustic event (e.g., "1" for yes, "0" for no), a probability related to whether an acoustic event occurred (e.g., 0.7 for the audio data captures an acoustic event vs. 0.3 for the audio data does not capture an acoustic event), and / or other scales related to the acoustic event.

[0099] In block 564, the system determines whether an acoustic event is a real acoustic event, based on the scale and additional scales it processed in block 562. If, in an iteration of block 564, the system determines that the acoustic event is not a real acoustic event, the system proceeds to block 566 and discards the audio data and additional audio data without further processing. The system then returns to block 552. For example, suppose the scale and additional scale correspond to the probabilities that the audio data and additional audio data capture a hotword, respectively. For example, suppose the scale is assumed to be a 0.7 probability that the audio data captures a hotword, and the additional scale is assumed to be a 0.3 probability that the additional audio data captures a hotword. In this example, the system might determine that the acoustic event is a real acoustic event (e.g., the user provided an utterance containing a hotword) by assuming that the probability of 0.7, based solely on the scale, satisfies a threshold probability level. However, the system might determine that the acoustic event is not a real acoustic event (e.g., the user did not provide an utterance containing a hotword) by assuming that the probability of 0.3, based solely on the additional scale, does not satisfy a threshold probability level. As a result, the system may determine that an acoustic event is not an actual acoustic event even if the scale indicates that it was an actual acoustic event, because additional scales must also indicate that the acoustic event was an actual acoustic event, and the audio data and additional audio data may be discarded. Alternatively, the system may determine whether an acoustic event is an actual acoustic event based on statistics related to the scale, such as the mean, median, highest, lowest, or percentile of the scale, which are generated based on each audio data detected in an assistant device that can be used to determine whether an acoustic event is an actual acoustic event.

[0100] If the system determines in an iteration of block 564 that the acoustic event is a real acoustic event, the system proceeds to block 568. For example, suppose the scale and additional scale correspond to the probabilities that the audio data and the additional audio data each capture a hotword. For example, suppose the scale is assumed to be a 0.7 probability that the audio data captures a hotword, and the additional scale is assumed to be a 0.8 probability that the additional audio data captures a hotword. In this example, the system may determine that the acoustic event is a real acoustic event (e.g., the user provided an utterance containing a hotword) by assuming that the probabilities of 0.7 and 0.8 (or one or more statistics based on them) satisfy a threshold probability level based on the scale and the additional scale, respectively. As a result, the system may determine that the acoustic event is a real acoustic event because both the scale and the additional scale indicate that the acoustic event was a real acoustic event.

[0101] In block 568, the system ensures that actions related to actual acoustic events are performed. The system then returns to block 552. The actions performed by the system may be based on actual acoustic events captured in the audio data and additional audio data. For example, in an implementation where an actual acoustic event corresponds to the occurrence of a hotword, the system may activate one or more components or functions of the automation assistant (e.g., speech processing, natural language processing, and / or other components or functions of the automation assistant). In another example, in an implementation where an actual acoustic event corresponds to the occurrence of a specific sound, the system may generate a notification indicating that a specific sound has been detected in the ecosystem (and optionally identify the assistant device that detected the specific sound), and the notification may be rendered on the user's client devices related to the ecosystem (e.g., the user's mobile device, one or more assistant devices, and / or any other client devices related to the user).

[0102] Figure 6 is a block diagram of an exemplary computing device 610 that may be optionally used to perform one or more embodiments of the techniques described herein. In some implementations, one or more assistant input devices, one or more cloud-based automation assistant components, one or more assistant non-input systems, one or more assistant non-input devices, and / or other components may comprise one or more components of the exemplary computing device 610.

[0103] The computing device 610 typically includes at least one processor 614 that communicates with several peripheral devices via a bus subsystem 612. These peripheral devices may include, for example, a storage subsystem 624, which includes a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices enable user interaction with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to a corresponding interface device in other computing devices.

[0104] The user interface input device 622 may include pointing devices such as keyboards, mice, trackballs, touchpads, or graphics tablets, scanners, touchscreens integrated into displays, speech recognition systems, microphones, and / or other types of input devices. In general, the use of the term “input device” is intended to include all possible types of devices and methods for inputting information into the computing device 610 or a communication network.

[0105] The user interface output device 620 may include a non-visual display such as a display subsystem, printer, fax machine, or audio output device. The display subsystem may include a flat panel device such as a cathode ray tube (CRT), liquid crystal display (LCD), projection device, or any other mechanism for creating a visible image. The display subsystem may also provide a non-visual display via an audio output device, etc. In general, the use of the term “output device” is intended to include all possible types of devices and methods for outputting information from the computing device 610 to a user or another machine or computing device.

[0106] The storage subsystem 624 stores programming and data constructs that provide some or all of the functionality of the modules described herein. For example, the storage subsystem 624 may include logic for performing selected embodiments of the methods described herein, as well as for implementing the various components shown in Figure 1.

[0107] These software modules are generally executed by processor 614, either alone or in combination with other processors. The memory 625 used by the storage subsystem 624 may include several memories, including main random access memory (RAM) 630 for storing instructions and data during program execution, and read-only memory (ROM) 632 for storing fixed instructions. The file storage subsystem 626 can provide persistent storage for program and data files and may include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical drives, or removable media cartridges. Modules implementing several functional implementations may be stored in the storage subsystem 624, by the file storage subsystem 626, or on other machines accessible by processor 614.

[0108] The bus subsystem 612 provides a mechanism for various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple buses.

[0109] The computing device 610 can be of various types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, the description of the computing device 610 shown in Figure 6 is intended to be only a specific example illustrating several possible implementations. Many other configurations of the computing device 610 may have more or fewer components than the computing device shown in Figure 6.

[0110] Where some of the implementations discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about a user's social networks, a user's location, a user's time, a user's biometric information, as well as user activity and attribute information, relationships between users, etc.), the user is given one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use a user's personal information only if they have received explicit consent from the user concerned to do so.

[0111] For example, a user may be given control over whether a program or feature collects user information about that particular user or other users associated with that program or feature. Each user from whom personal information will be collected may be presented with one or more options to enable control over information collection related to that user, in order to provide permission or approval regarding whether information will be collected and which parts of the information should be collected. For example, a user may be given one or more such control options via a communication network. In addition, certain data may be handled in one or more ways before it is stored or used so that personally identifiable information is removed. As an example, a user's identification information may be handled in such a way that personally identifiable information cannot be determined. As another example, a user's geographical location may be generalized to a broader area so that the user's specific location cannot be determined.

[0112] In some implementations, a method is provided which is carried out by one or more processors and includes the steps of: detecting audio data capturing an acoustic event via one or more microphones of an assistant device located in an ecosystem of multiple assistant devices; processing the audio data capturing the acoustic event using an event detection model stored locally in the assistant device to generate a measure related to the acoustic event; detecting additional audio data similarly capturing an acoustic event via one or more microphones of an additional assistant device located in the ecosystem, the additional assistant device being added to the assistant device and being in the same location as the assistant device in the ecosystem; processing the additional audio data capturing an acoustic event using an additional event detection model stored locally in the additional assistant device to generate an additional measure related to the acoustic event; processing both the measure and the additional measure to determine whether the acoustic event detected by at least both the assistant device and the additional assistant device is an actual acoustic event; and, in response to the determination that the acoustic event is an actual acoustic event, causing an action related to the actual acoustic event to be performed.

[0113] These and other implementations of the technologies disclosed herein may include one or more of the following features:

[0114] In some implementations, the acoustic events may include hotword detection events, the event detection model stored locally in the assistant device may be a hotword detection model trained to detect whether a particular word or phrase is captured in the audio data, and the additional event detection models stored locally in additional assistant devices may be additional hotword detection models trained to detect whether a particular word or phrase is captured in additional audio data.

[0115] In some versions of these implementations, the metric associated with an acoustic event may be a confidence level corresponding to whether the audio data captures a particular word or phrase, and an additional metric associated with an acoustic event may be an additional confidence level corresponding to whether the additional audio data captures a particular word or phrase. Some further versions of these implementations that determine an acoustic event is an actual acoustic event may include determining, based on the confidence level and the additional confidence level, that a particular word or phrase is captured in both the audio data and the additional audio data. Yet another version of these implementations that causes an action to be taken related to an actual acoustic event may include activating one or more components of the automation assistant in response to the determination that the acoustic event data indicates that the audio data or the additional audio data captures a particular word or phrase.

[0116] In some additional or alternative versions of these implementations, the hotword detection model stored locally in the assistant device may be a separate hotword model from any additional hotword detection models stored locally in additional assistant devices.

[0117] In some implementations, acoustic events may be sound detection events, event detection models stored locally in the assistant device may be sound detection models trained to detect whether a particular sound is captured in audio data, and additional event detection models stored locally in additional assistant devices may be additional sound detection models trained to detect whether a particular sound being captured is additional audio data.

[0118] In some versions of these implementations, the metric associated with an acoustic event may be a confidence level corresponding to whether audio data captures a particular sound, and an additional metric associated with an acoustic event may be an additional confidence level corresponding to whether additional audio data captures a particular sound. In some further versions of these implementations, determining that an acoustic event is an actual acoustic event may include determining, based on the confidence level and the additional confidence level, that a particular sound is captured in both the audio data and the additional audio data. In yet another version of these implementations, ensuring that an action is taken in relation to an actual acoustic event may include generating a notification indicating the occurrence of an actual acoustic event and ensuring that the notification is presented to users related to the ecosystem via the user's computing device.

[0119] In some additional or alternative versions of these implementations, the specific sounds may include one or more of the following: glass breaking, a dog barking, a cat meowing, a doorbell ringing, a fire alarm going off, a carbon monoxide detector going off, a baby crying, or a door being knocked on.

[0120] In some implementations, it is the responsibility of a given assistant device to process both the scale and the additional scale to determine whether an acoustic event detected by both the assistant device and the additional assistant device is an actual acoustic event, and the given assistant device may be one or more of the assistant device, the additional assistant device, or further additional assistant devices that are in the same position in the ecosystem as the assistant device and the additional assistant device.

[0121] In some implementations, the method may further include the steps of sending audio data to a remote system by an assistant device and sending additional audio data to the remote system by an additional assistant device. In these implementations, it may be the remote system that processes both the scale and the additional scale to determine whether the acoustic events detected by both the assistant device and the additional assistant device are actual acoustic events.

[0122] In some implementations, audio data temporarily corresponds to additional audio data. In some versions of those implementations, processing both the scale and the additional scale to determine whether an acoustic event detected by both the assistant device and the additional assistant device is an actual acoustic event may be in response to determining that the timestamp associated with the audio data corresponds temporally to the additional timestamp associated with the additional audio data.

[0123] In some implementations, the method may further include the step of anticipating the detection of additional audio data via one or more additional microphones of an additional assistant device, based on the fact that multiple past acoustic events have been detected in both the assistant device and the additional assistant device, in response to the detection of audio data via one or more microphones of the assistant device.

[0124] In some implementations, the method further includes the steps of: detecting subsequent audio data capturing subsequent acoustic events via one or more microphones of an assistant device; processing the subsequent audio data capturing subsequent acoustic events using an event detection model to generate subsequent measures related to the subsequent acoustic events; and detecting additional subsequent audio data similarly capturing subsequent acoustic events via one or more additional microphones of further additional assistant devices located in the ecosystem, wherein the further additional assistant devices add to the assistant device, and the further additional assistant devices are assistant The process may include the steps of: processing additional subsequent audio data that captures an acoustic event using additional event detection models stored locally in the assistant device and further additional assistant devices, which are located in the same position in the assistant device and ecosystem, to generate additional subsequent measures related to the acoustic event; processing both the measures and the additional measures to determine whether the subsequent acoustic event detected by both the assistant device and further additional assistant devices is an actual subsequent acoustic event; and, in response to the determination that the subsequent acoustic event is an actual subsequent acoustic event, causing subsequent actions related to the actual subsequent acoustic event to be performed.

[0125] In some additional versions of these implementations, the method may further include the steps of anticipating the detection of additional subsequent audio data through one or more additional microphones of further additional assistant devices in response to the detection of subsequent audio data through one or more microphones of an assistant device, and not anticipating the detection of any audio data through one or more additional microphones of additional assistant devices based on the fact that multiple additional past acoustic events have been detected in both the assistant device and the further additional assistant devices.

[0126] In some implementations, a method is provided which is carried out by one or more processors and includes the steps of: detecting audio data capturing an acoustic event via one or more microphones of an assistant device located in an ecosystem of multiple assistant devices; identifying at least one additional assistant device, based on the location of the assistant devices in the ecosystem, that should have detected additional audio data that is temporally corresponding to the audio data via one or more microphones of at least one additional assistant device in the ecosystem; processing the audio data capturing an acoustic event using an event detection model stored locally in the assistant device to generate a measure related to the acoustic event, in response to the determination that at least one additional assistant device has detected additional audio data that is temporally corresponding to the audio data; processing the additional audio data capturing an acoustic event using each event detection model stored locally in at least one additional assistant device to generate an additional measure related to the acoustic event; determining, based on both the measure and the additional measure, whether the acoustic event detected by both the assistant device and the additional assistant device is an actual acoustic event; and causing an action related to the actual acoustic event to be performed in response to the determination that the acoustic event is an actual acoustic event.

[0127] These and other implementations of the technologies disclosed herein may include one or more of the following features:

[0128] In some implementations, the method may further include a step of discarding audio data in response to at least one additional assistant device determining that it has not detected any audio data that is temporally corresponding to the audio data.

[0129] In addition, some implementations include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU and / or a tensor processing unit (TPU))) of one or more computing devices, where one or more processors are operable to execute instructions stored in associated memory, and the instructions are configured to cause the execution of one of the methods described above. Some implementations also include one or more non-temporary computer-readable storage media that store computer instructions that can be executed by one or more processors to perform one of the methods described above. Some implementations also include a computer program product that includes instructions that can be executed by one or more processors to perform one of the methods described above.

[0130] It should be understood that all combinations of the aforementioned concepts and additional concepts, which are described in more detail herein, are considered to be part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. [Explanation of Symbols]

[0131] 105 Presence sensor 106 Assistant Input Devices 107 User Interface Components 110 Network 114 Speech Capture / TTS / STT Module 116 Cloud-based TTS module 117 Cloud-based STT module 118 Automation Assistant Client 119 Cloud-based automation assistant components 120 Automation Assistants 122 NLP Modules 130 Event Detection Engines 140 Device Identification Engines 150 Event Processing Engine 160 Semantic Signage Engine 170 Query / Command Processing Engine 180 Assistant Non-Input System 185 Assistant Non-Input Devices 191 Device Activity Database 192 ML Model Database 193 Device Topology Database 250 rooms 252 rooms 254 rooms 256 rooms 258 rooms 260 rooms 262 rooms 610 Computing Devices 612 Bus Subsystem 614 Processors 616 Network Interfaces 620 User Interface Output Devices 622 User Interface Input Devices 624 Storage Subsystems 625 Memory subsystem 626 File Storage Subsystem 630 RAM 632 ROM

Claims

1. A method carried out by one or more processors, The steps include detecting audio data that captures an acoustic event via one or more microphones of an assistant device located within an ecosystem that includes multiple assistant devices, The steps include: processing the audio data that captures the acoustic event using an event detection model locally stored in the assistant device to generate a scale related to the acoustic event; A step of receiving additional audio data that similarly captures the acoustic event from an additional assistant device located in the same position as the assistant device in the ecosystem, wherein the additional assistant device is added to the assistant device, the additional assistant device is identified based on the fact that the assistant device and the additional assistant device belong to the same group of assistant devices in the device topology representation of the ecosystem, the assistant device and the additional assistant device have previously detected temporally corresponding audio data that captures the same acoustic event, and the additional audio data is generated via one or more additional microphones of the additional assistant device. The steps include: using the event detection model stored locally in the assistant device, processing the additional audio data that captures the acoustic event to generate an additional measure related to the acoustic event; A step of processing both the aforementioned scale and the additional scale to determine whether the acoustic event detected by at least both the assistant device and the additional assistant device is an actual acoustic event, A method comprising the step of causing an action related to the actual acoustic event to be performed in response to the determination that the acoustic event is the actual acoustic event.

2. The method according to claim 1, wherein the acoustic event comprises a hotword detection event, and the event detection model stored locally in the assistant device comprises a hotword detection model trained to detect whether a particular word or phrase is captured in the audio data and the additional audio data.

3. The method according to claim 2, wherein the scale relating to the acoustic event comprises a confidence level corresponding to whether the audio data captures the particular word or phrase, and the additional scale relating to the acoustic event comprises an additional confidence level corresponding to whether the additional audio data captures the particular word or phrase.

4. The method of claim 3, wherein the step of determining that the acoustic event is the actual acoustic event comprises the step of determining, based on the confidence level and the additional confidence level, that the particular word or phrase is captured in both the audio data and the additional audio data.

5. The method according to claim 2, wherein the step of causing the action relating to the actual sound event to be performed comprises the step of activating one or more components of the automation assistant in the assistant device or the additional assistant device in response to the determination that the audio data and the additional audio data capture the particular word or phrase.

6. The method according to claim 1, wherein the acoustic event comprises a sound detection event, and the event detection model stored locally in the assistant device comprises a sound detection model trained to detect whether a particular sound is captured in the audio data and the additional audio data.

7. The method according to claim 6, wherein the scale related to the acoustic event comprises a confidence level corresponding to whether the audio data captures the particular sound, and the additional scale related to the acoustic event comprises an additional confidence level corresponding to whether the additional audio data captures the particular sound.

8. The method of claim 7, wherein the step of determining that the acoustic event is the actual acoustic event comprises the step of determining, based on the confidence level and the additional confidence level, that the particular sound is captured in both the audio data and the additional audio data.

9. The step of ensuring that the action related to the actual acoustic event is performed is, The steps include generating a notification indicating the occurrence of the aforementioned sound detection event, The method according to claim 6, further comprising the step of causing the notification to be presented to a user associated with the ecosystem via the user's computing device.

10. The method according to claim 6, wherein the specific sound comprises one or more of the following: glass breaking, a dog barking, a cat meowing, a doorbell ringing, a fire alarm sounding, a carbon monoxide detector sounding, an infant crying, or a door being knocked on.

11. The method according to claim 1, wherein the audio data corresponds temporally to the additional audio data.

12. The method of claim 1, wherein the step of processing both the scale and the additional scale to determine whether the acoustic event detected by both the assistant device and the additional assistant device is the actual acoustic event is determined in response to determining that the timestamp associated with the audio data corresponds in time to the additional timestamp associated with the additional audio data.

13. The method according to claim 1, wherein the assistant device is a first-party assistant device manufactured by a first party, and the additional assistant device is a third-party assistant device manufactured by a third party, wherein the third party is a separate party from the first party.

14. A method carried out by one or more processors, A step of receiving audio data that captures an acoustic event, wherein the audio data is generated via one or more microphones of an assistant device located in an ecosystem that includes multiple assistant devices. The steps include: processing the audio data that captures the acoustic event using an event detection model to generate a measure related to the acoustic event; A step of receiving additional audio data that similarly captures the aforementioned acoustic event, wherein the additional audio data is generated via one or more additional microphones of an additional assistant device located in the same position as the assistant device in the ecosystem, the additional assistant device is added to the assistant device, the additional assistant device is identified based on the fact that the assistant device and the additional assistant device belong to the same group of assistant devices in the device topology representation of the ecosystem, and the assistant device and the additional assistant device have previously detected temporally corresponding audio data capturing the same acoustic event; The steps include using the event detection model to process the additional audio data that captures the acoustic event and generate an additional measure related to the acoustic event, A step of processing both the aforementioned scale and the additional scale to determine whether the acoustic event detected by at least both the assistant device and the additional assistant device is an actual acoustic event, A method comprising the step of causing an action related to the actual acoustic event to be performed in response to the determination that the acoustic event is the actual acoustic event.

15. The method according to claim 14, wherein the assistant device is a first-party assistant device manufactured by a first party, and the additional assistant device is a third-party assistant device manufactured by a third party, wherein the third party is a separate party from the first party.

16. The method according to claim 14, wherein the one or more processors are in a remote system located away from the ecosystem, and the event detection model is located away from the ecosystem.

17. It is an assistant device, At least one processor, When executed, the memory stores instructions that cause at least one processor to perform the method according to any one of claims 1 to 13, An assistant device equipped with the following features.

18. It is a remote system, At least one processor, When executed, the memory stores instructions that cause at least one processor to perform the method according to any one of claims 14 to 16, A remote system equipped with [the following features].

19. A non-temporary computer-readable storage medium that, when executed, stores instructions causing at least one processor to perform the method according to any one of claims 1 to 16.

Images