Simultaneous acoustic event detection across multiple assistant devices

Abstract

The invention relates to simultaneous acoustic event detection across multiple assistant devices. Embodiments may detect, at a plurality of assistant devices in an ecosystem including the plurality of assistant devices, respective audio data that captures an acoustic event, locally process, at each of the plurality of assistant devices, the respective audio data to generate a respective metric associated with the acoustic event using a respective event detection model, the respective metrics are processed to determine whether the detected acoustic event is an actual acoustic event, and in response to determining that the detected acoustic event is an actual acoustic event, an action associated with the actual acoustic event is caused to be performed. In some implementations, based on a plurality of historical acoustic events detected at each of the plurality of assistant devices, a plurality of assistant devices that detected respective audio data are expected to detect respective audio data that captured actual acoustic events.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 202080100908.3, with a filing date of December 15, 2020. Background Art

[0003] A human may participate in a human-computer conversation with an interactive software application, which is referred to herein as an "automated assistant" (also referred to as a "chatbot," "interactive personal assistant," "intelligent personal assistant," "personal voice assistant," "conversational agent," etc.). For example, a human (who may be referred to as a "user" when interacting with an automated assistant) may provide input (e.g., commands, queries, and / or requests) to the automated assistant, which may cause the automated assistant to generate and provide responsive output to control one or more Internet of Things (IoT) devices, and / or to perform one or more other functions. The input provided by the user may be, for example, spoken natural language input (i.e., spoken utterances) that may in some cases be converted to text (or other semantic representations) and then further processed, and / or typed natural language input.

[0004] In some cases, an automated assistant may include an automated assistant client that is executed locally by an assistant device and directly engaged by a user, and a cloud-based counterpart that leverages virtually unlimited cloud resources to help the automated assistant client respond to the user's input. For example, the automated assistant client may provide an audio recording of the user's spoken utterances (or their textual conversions) to the cloud-based counterpart, and optionally data indicating the user's identity (e.g., credentials). The cloud-based counterpart may perform various processing on the query to return results to the automated assistant client, which may then provide corresponding output to the user.

[0005] Many users may employ an automated assistant using multiple assistant devices. For example, some users may have a coordinated “ecosystem” of assistant devices that may receive user input directed to the automated assistant and / or may be controlled by an automated assistant of assistant devices such as one or more smartphones, one or more tablets, one or more vehicle computing systems, one or more wearable computing devices, one or more smart TVs, one or more interactive standalone speakers, and / or one or more IoT devices. A user may use any of these assistant devices to participate in a human-machine conversation with an automated assistant (assuming that an automated assistant client is installed and the assistant device is capable of receiving input). In some cases, these assistant devices may be dispersed around a user's primary residence, secondary residence, workplace, and / or other structures. For example, mobile assistant devices such as smartphones, tablets, smart watches, and the like may be on the user and / or wherever the user last placed them. Other assistant devices such as traditional desktop computers, smart TVs, interactive standalone speakers, and IoT devices may be more stable, but may also be located in different locations (e.g., rooms) within a user's home or workplace.

[0006] Existing technology can alert a user (e.g., a single user, multiple users in a household, co-workers, co-residents, etc.) when an acoustic event is detected at a given assistant device in an ecosystem of assistant devices. However, such technology can help detect false positive events. In other words, a given assistant device may determine that an acoustic event has occurred when in fact the acoustic event did not actually occur. As a result, an automated assistant client for a given assistant device may cause one or more actions to be performed when the automated assistant client should not have performed any actions. Summary of the invention

[0007] The embodiments described herein relate to simultaneously detecting audio data capturing an acoustic event at multiple assistant devices in an ecosystem including multiple assistant devices. The corresponding audio data detected at each of the multiple assistant devices capturing the acoustic event can be processed using an event detection model to generate corresponding metrics associated with the acoustic event. Each of the corresponding metrics can be processed to determine whether the acoustic event is an actual acoustic event. In response to determining that the acoustic event is an actual acoustic event, an action associated with the acoustic event can be performed. The corresponding audio data can be processed locally at the assistant device using an event detection model, remotely at a remote system using an event detection model, and / or at a given assistant device in the ecosystem (e.g., which may or may not have detected audio data capturing an acoustic event) using an event detection model.

[0008] In some embodiments, an acoustic event may correspond to a hot word event. In these embodiments, an acoustic event detection model may correspond to a hot word detection model that is trained to detect a specific word or phrase that, when detected, causes one or more components and / or functionality of an automated assistant to be activated at one or more assistant devices in the ecosystem. For example, assume that a given assistant device with a microphone is located in the primary residence of a user associated with the ecosystem. It is further assumed that a given assistant detects audio data via a microphone that captures an acoustic event, and further assumes that the audio data is processed using a hot word detection model to generate a probability indicating whether the audio data includes a specific word or phrase. In addition, it is assumed that at least one additional assistant device with an additional microphone is close to the given assistant device in the ecosystem. It is further assumed that at least one additional assistant device detects additional audio data via an additional microphone that also captures an acoustic event, and further assumes that the additional audio data is processed using a hot word detection model to generate an additional probability that also indicates whether the additional audio data includes a specific word or phrase. In this example, the probability and the additional probability can be processed to determine whether the acoustic event actually corresponds to the occurrence of a specific word or phrase.

[0009] For example, assume that the probability is a value of 0.70 and the additional probability is a value of 0.65. In this example, if both the probability and the additional probability meet a threshold value (e.g., a value of 0.60), the hot word event can be verified as an actual hot word event, and even if both probabilities are not relatively high, the one or more components and / or functionalities of the automated assistant can be activated. Also, for example, assume that the probability is a value of 0.90 and the additional probability is a value of 0.55. In this example, the additional probability may not meet the threshold value, but the hot word event can still be verified as an actual hot word event, and the one or more components and / or functionalities of the automated assistant can be activated because the indicated probability is relatively high (and optionally, because the additional probability is within a threshold range of the threshold value (e.g., within a value of 0.10)).

[0010] In some additional or alternative embodiments, an acoustic event may correspond to a sound event. In these embodiments, an acoustic event detection model may correspond to a sound detection model trained to detect one or more specific sounds, which, when detected, results in a notification being presented visually and / or audibly at a client device of a user associated with the ecosystem. Specific sounds may include, for example, glass breaking, barking, cat meowing, doorbell ringing, smoke alarm ringing, carbon monoxide detector ringing, baby crying, knocking on the door, and / or any other sound that may occur in the ecosystem. In some versions of those embodiments, multiple sound detection models may be trained to detect corresponding specific sounds, while in other embodiments, a single sound detection model may be trained to detect multiple sounds. Audio data may be processed in the same or similar manner as described above with respect to hot word events, but additionally or alternatively includes processing audio data using a sound detection model to verify whether a sound event is an actual sound event.

[0011] In various embodiments, in response to a given assistant device detecting audio data capturing an acoustic event, at least one additional assistant device in the ecosystem can be identified. If the acoustic event detected at the given assistant device is in fact an actual acoustic event, the at least one additional assistant device identified in the ecosystem should have detected the temporally corresponding audio data that also captured the acoustic event. The audio data captured by the assistant device can be considered temporally corresponding audio data based on, for example, corresponding timestamps associated with the audio data captured at the assistant device. For example, the audio data can be considered temporally corresponding audio data when the timestamps match or are within a threshold duration of each other's time (e.g., within a few milliseconds, a few seconds, or any other suitable duration). In some versions of those embodiments, the at least one additional assistant device can be identified based on, for example, a given assistant device and the at least one additional assistant device historically detecting audio data capturing the same acoustic event. In some additional or alternative versions of those embodiments, the at least one additional assistant device can be identified based on, for example, a given assistant device and the at least one additional assistant device belonging to the same group of assistant devices in a device topology representation of the ecosystem.

[0012] By using the techniques described herein to detect and verify the occurrence of acoustic events using multiple assistant devices in the ecosystem, the amount of false positives of acoustic events can be reduced. As a result, both computing resources and network resources can be saved. For example, by using the techniques described herein to detect and verify the occurrence of hot word events in the ecosystem, the components and / or functionality of the automated assistant can be kept dormant, and when these techniques are missing, the components and / or functionality of the automated assistant may have been activated, even if they are not intended to be activated. As another example, by using the techniques described herein to detect and verify the occurrence of specific sounds in the ecosystem, notifications displayed to users associated with the ecosystem can be retained, and when these techniques are missing, even if the acoustic event does not actually occur, the notification may have been displayed. In addition, by using the techniques described herein to anticipate which assistant devices in the ecosystem should detect and capture the audio data corresponding to the time of the acoustic event, the amount of assistant devices that process audio data can be reduced. As a result, both computing resources and network resources can be saved. For example, by using the techniques described herein to anticipate assistant devices in an ecosystem that should have detected an acoustic event, other assistant devices that may have captured temporally corresponding audio data that may not correspond to the acoustic event may not be considered when determining whether the acoustic event is an actual acoustic event.

[0013] The above description is provided as an overview of only some embodiments of the present disclosure. Further descriptions of those embodiments and other embodiments are described in more detail herein. As a non-limiting example, various embodiments are described in more detail in the claims included herein.

[0014] In addition, some embodiments include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause the performance of any of the methods described herein. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the methods described herein.

[0015] It should be understood that all combinations of the aforementioned concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0017] Figure 2A and Figure 2B Depicted are some examples associated with simultaneous acoustic event detection of hotword events at multiple assistant devices in an ecosystem according to various implementations.

[0018] Figure 3 Depicted are examples associated with simultaneous acoustic event detection of a particular acoustic event at multiple assistant devices in an ecosystem, according to various implementations.

[0019] Figure 4 is a flow chart illustrating an example method of determining whether an acoustic event is an actual acoustic event based on audio data captured at multiple assistant devices in an ecosystem according to various implementations.

[0020] Figure 5 is a flow chart illustrating an example method of identifying which assistant devices should have detected an acoustic event in an ecosystem based on a given assistant device detecting the acoustic event, according to various implementations.

[0021] Figure 6 Depicted is an example architecture of a computing device in accordance with various implementations. DETAILED DESCRIPTION

[0022] There is a proliferation of smart multi-sensing network-connected devices (also referred to herein as assistant devices), such as smartphones, tablets, vehicle computing systems, wearable computing devices, smart televisions, interactive stand-alone speakers (e.g., with or without displays), speakers, home alarms, door locks, cameras, lighting systems, treadmills, thermostats, weighing stations, smart beds, irrigation systems, garage door openers, home appliances, baby monitors, fire alarms, moisture detectors, etc. Typically, multiple assistant devices are located within the confines of a structure, such as a home - or within multiple related structures, such as a user's primary residence and a user's secondary residence, a user's vehicle, and / or a user's place of work.

[0023] In addition, there is a proliferation of assistant devices that each include an automated assistant client, which can form a logical instance of an automated assistant (also referred to herein as an assistant input device). These assistant input devices can be solely dedicated to assistant functionality (e.g., an interactive stand-alone speaker and / or a stand-alone audio / visual device that includes only an assistant client and an associated interface and is solely dedicated to assistant functionality), or can perform assistant functionality in addition to other functions (e.g., a mobile phone or tablet that includes an assistant client as one of multiple applications). In addition, some IoT devices may also be assistant input devices. For example, some IoT devices may include an automated assistant client and (at least in part) at least a speaker and / or microphone that serves as a user interface output and / or input device for an assistant interface for an automated assistant client. Although some assistant devices may not implement an automated assistant client or have devices for interacting with a user (e.g., a speaker and / or microphone), they can still be controlled by an automated assistant (also referred to herein as an assistant non-input device). For example, a smart light bulb may not include an automated assistant client, speaker, and / or microphone, but commands and / or requests may be transmitted to the smart light bulb via the automated assistant to control the functionality of the smart light bulb (e.g., turn on / off, dim, change color, etc.).

[0024] Various techniques have been proposed for adding assistant devices (including both assistant input devices and assistant non-input devices) to an ecosystem of assistant devices and / or grouping assistant devices within an ecosystem. For example, when adding a new assistant device to an ecosystem, a user associated with the ecosystem can manually add the new assistant device to a group of assistant devices in the ecosystem via a software application in a device topology representation of the ecosystem (e.g., via an automated assistant application, a software application associated with the ecosystem, a software application associated with the new assistant device, etc.). In addition, if the assistant device moves within the ecosystem, the user can manually change the group to which the assistant device is assigned via the software application. Otherwise, the group to which the assistant device is assigned may not accurately reflect the location of the assistant device within the ecosystem. For example, if a smart speaker labeled "Living Room Speaker" is located in the living room of a user's primary home and is associated with the "Living Room" group of assistant devices, but the smart speaker is moved to the kitchen of the user's primary home, the smart speaker can still be labeled "Living Room Speaker" and included in the "Living Room" group of the assistant device, even though the label and group do not represent the location of the assistant device unless the user manually changes the label and group in the device topology representation of the ecosystem for the user's primary home.

[0025] The device topology representation may include a tag (or unique identifier) ​​associated with the corresponding assistant device. In addition, the device topology representation may specify a tag (or unique identifier) ​​associated with the corresponding assistant device. The device attributes for a given assistant device may indicate, for example, one or more input and / or output modes supported by the corresponding assistant device. For example, a device attribute for a standalone speaker-only assistant client device may indicate that it is capable of providing audible output but not visual output. Additionally or alternatively, the device attributes for a given assistant device may, for example, identify one or more states of a given assistant device that can be controlled; identify a party (e.g., first party (1P) or third party (3P)) that manufactures, distributes, and / or creates firmware for the assistant device; and / or identify a unique identifier for a given assistant device, such as a fixed identifier provided by 1P or 3P or a tag assigned to a given assistant device by a user. According to various embodiments disclosed herein, the device topology representation may optionally further specify: which smart devices can be locally controlled by which assistant devices; local addresses for assistant devices that can be locally controlled (or local addresses for hubs that can directly locally control those assistant devices); local signal strength and / or other preference indicators among the corresponding assistant devices. In addition, according to various embodiments disclosed herein, a device topology representation (or a variation thereof) can be stored locally at each of a plurality of assistant devices for use in locally controlling the assistant device and / or locally assigning tags to the assistant device. In addition, the device topology representation can specify groups associated with corresponding assistant devices that can be defined with various levels of granularity. For example, a plurality of smart lights in the living room of a user's primary house can be considered to belong to a "living room lights" group. Furthermore, if the living room of the primary house also includes a smart speaker, all assistant devices located in the living room can be considered to belong to a "living room assistant device" group.

[0026] The automated assistant can detect various events occurring in the ecosystem based on one or more signals generated by one or more assistant devices. For example, the automated assistant can use an event detection model or rule to process one or more signals to detect these events. In addition, the automated assistant can perform one or more actions based on the output generated based on one or more signals for events occurring in the ecosystem. In some embodiments, the detected event can be an acoustic event captured via the corresponding microphone of one or more assistant devices. The automated assistant can cause the capture of audio data of the acoustic event to be processed using the acoustic event model. The acoustic events detected by the acoustic event model can include, for example: using a hot word detection model to detect hot words for calling the automated assistant included in the spoken utterance; using an environmental noise detection model (and optionally, when voice reception is active at a given one of the assistant devices) to detect environmental noise in the ecosystem, using a sound detection model to detect specific sounds in the ecosystem (e.g., glass breaking, barking, cat meowing, doorbell ringing, smoke alarm ringing, carbon monoxide detector ringing, baby crying, knocking on the door and / or other acoustic events), and / or other sound-related events that can be detected using the corresponding acoustic event detection model. For example, assume that audio data is detected via a corresponding microphone of at least one of the assistant devices. In this example, the automated assistant may cause the audio data to be processed by a hot word detection model of the at least one assistant device to determine whether the audio data captures a hot word that invokes the automated assistant. Furthermore, additionally or alternatively, the automated assistant may cause the audio data to be processed by an ambient noise detection model of the at least one assistant device to classify any ambient (or background) noise captured in the audio data into one or more disparate semantic categories of ambient noise (e.g., movie or television sounds, cooking sounds, and / or other disparate categories of sounds). Furthermore, additionally or alternatively, the automated assistant may cause the audio data to be processed by a sound detection model of the at least one assistant device to determine whether any particular sound is captured in the audio data.

[0027] Embodiments described herein relate to determining acoustic events that actually occurred in an ecosystem based on temporally corresponding audio data captured by respective microphones of a plurality of assistant devices. Those embodiments further relate to processing the audio data locally at the assistant device to generate a metric indicating whether the acoustic event actually occurred. Still further, those embodiments relate to identifying which assistant devices in the ecosystem should have detected audio data that temporally corresponds to audio data represented based on a topology of devices in the ecosystem, and when a plurality of assistant devices detect temporally corresponding audio data, causing the temporally corresponding audio data to be processed by respective event detection models.

[0028] Now go to Figure 1, illustrates an example environment in which the technology disclosed herein can be implemented. The example environment includes multiple assistant input devices 106 1-N (also referred to herein as “assistant input device 106 ”), one or more cloud-based automated assistant components 119 , one or more assistant non-input systems 180 , one or more assistant non-input devices 185 1-N (also referred to herein as “assistant non-input devices 185 ”), a device activity database 191 , a machine learning (“ML”) model database, and a device topology database 193 . Figure 1 The assistant input device 106 and the assistant non-input device 185 may also be collectively referred to as "assistant devices" in this article.

[0029] One or more (eg, all) of the assistant input devices 106 may execute a corresponding automated assistant client 118 1-N However, in some implementations, one or more assistant input devices 106 may optionally lack a corresponding automated assistant client 118. 1-N , and still includes engines and hardware components (e.g., microphones, speakers, speech recognition engines, natural language processing engines, speech synthesis engines, etc.) for receiving and processing user input directed to the automated assistant. Automated assistant client 118 1-N An example of may be an application separate from the operating system of the corresponding assistant input device 106 (e.g., installed "on top" of the operating system) - or may alternatively be implemented directly by the operating system of the corresponding assistant input device 106. As further described below, in response to a corresponding user interface component 107 being displayed by any of the corresponding assistant input devices 106, 1-N Provide various requests to the automated assistant client 118 1-N Each instance of can optionally interact with one or more cloud-based automated assistant components 119. In addition, and as also described below, other engines of the assistant input device 106 can optionally interact with one or more cloud-based automated assistant components 119.

[0030] One or more cloud-based automated assistant components 119 may be implemented on one or more computing systems (e.g., servers collectively referred to as “cloud” or “remote” computing systems) that are communicatively coupled to respective assistant input devices 106 via one or more local area networks (“LANs”, including Wi-Fi LANs, Bluetooth networks, near field communication networks, mesh networks, etc.) and / or wide area networks (“WANs”, including the Internet, etc.). The communicative coupling of the cloud-based automated assistant component 119 to the assistant input device 106 is generally accomplished by Figure 1 110 out of 110 1Furthermore, in some embodiments, the assistant input devices 106 can be communicatively coupled to each other via one or more networks (eg, LANs and / or WANs), which are generally comprised of Figure 1 110 out of 110 2 express.

[0031] The one or more cloud-based automated assistant components 119 may also be communicatively coupled to the one or more assistant non-input systems 180 via one or more networks (e.g., LANs and / or WANs). The communicatively coupling of the cloud-based automated assistant components 119 to the assistant non-input systems 180 is generally accomplished by: Figure 1 110 out of 110 3 Indication. In addition, the assistant non-input systems 180 can each be communicatively coupled to one or more (e.g., groups) of assistant non-input devices 185 via one or more networks (e.g., LANs and / or WANs). For example, a first assistant non-input system 180 can be communicatively coupled to and receive data from a first group of one or more assistant non-input devices 185, a second assistant non-input system 180 can be communicatively coupled to and receive data from a second group of one or more assistant non-input devices 185, and so on. The communicatively coupling of the assistant non-input systems 180 with the assistant non-input devices 185 is generally determined by Figure 1 110 out of 110 4 express.

[0032] An instance of automated assistant client 118, through its interaction with one or more cloud-based automated assistant components 119, can form a logical instance that appears to be an automated assistant 120 from the user's perspective, and the user can participate in a human-computer dialogue with it. Figure 1 1 and 2. Two examples of such automated assistants 120 are depicted in FIG. The first automated assistant 120A, surrounded by a dashed line, includes an assistant input device 106. 1 Automated Assistant Client 118 1 and one or more cloud-based automated assistant components 119. A second automated assistant 120B enclosed by a dash-dash-dotted line includes an assistant input device 106 N Automated Assistant Client 118 Nand one or more cloud-based automated assistant components 119. Thus, it should be understood that each user who employs an automated assistant client 118 executing on one or more assistant input devices 106 can actually employ his or her own logical instance of an automated assistant 120 (or a logical instance of an automated assistant 120 shared among a family or other group of users). For the sake of brevity and simplicity, the term "automated assistant" as used herein will refer to the combination of an automated assistant client 118 executing on a respective one of the assistant input devices 106 and one or more cloud-based automated assistant components 119 (which can be shared among multiple automated assistant clients 118). Although Figure 1 Only a plurality of assistant input devices 106 are illustrated in FIG, but it should be understood that the cloud-based automated assistant component 119 can additionally serve many additional groups of assistant input devices.

[0033] Assistant input device 106 may include, for example, one or more of a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), an interactive stand-alone speaker (e.g., with or without a display), a smart appliance such as a smart TV, a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device), and / or any IoT device capable of receiving user input directed to automated assistant 120. Additional and / or alternative assistant input devices may be provided. Assistant non-input devices 185 may include many of the same devices as assistant input devices 106, but are not capable of receiving user input directed to automated assistant 120 (e.g., do not include a user interface input component). Although assistant non-input devices 185 do not receive user input directed to automated assistant 120, assistant non-input devices 185 may still be controlled by automated assistant 120.

[0034] In some embodiments, multiple assistant input devices 106 and assistant non-input devices 185 can be associated with each other in various ways to facilitate the performance of the techniques described herein. For example, in some embodiments, multiple assistant input devices 106 and assistant non-input devices 185 can be associated with each other via one or more networks (e.g., via Figure 1The assistant input devices 106 and the assistant non-input devices 185 are communicatively coupled to each other in a network 110 in a network. This can be, for example, a situation where multiple assistant input devices 106 and assistant non-input devices 185 are deployed across a particular area or environment, such as a home, a building, or the like. Additionally or alternatively, in some embodiments, multiple assistant input devices 106 and assistant non-input devices 185 can be associated with each other by virtue of being members of a coordinated ecosystem that is at least selectively accessible to one or more users (e.g., an individual, a family, employees of an organization, other predetermined groups, etc.). In some of those embodiments, the ecosystem of multiple assistant input devices 106 and assistant non-input devices 185 can be manually and / or automatically associated with each other in a device topology representation of the ecosystem stored in a device topology database 193.

[0035] Assistant non-input systems 180 may include one or more first party (1P) systems and / or one or more third party (3P) systems. A 1P system refers to a system controlled by the same party as the party controlling the automated assistant 120 referenced herein. As used herein, a 3P system refers to a system controlled by a different party than the party controlling the automated assistant 120 referenced herein.

[0036] The assistant non-input system 180 can receive data from the assistant non-input device 185 and / or the one or more cloud-based automated assistant components 119 communicatively coupled thereto (e.g., via Figure 1 110) and selectively transmits data (e.g., state, state change, and / or other data) to the assistant non-input device 185 and / or the one or more cloud-based automated assistant components 119. For example, assuming that the assistant non-input device 185 1 is a smart doorbell IoT device. In response to an individual pressing a button on the doorbell IoT device, the doorbell IoT device may transmit corresponding data to one of the assistant non-input systems 180 (e.g., one of the assistant non-input systems managed by the doorbell manufacturer, which may be a 1P system or a 3P system). The one assistant non-input system 180 may determine a change in the state of the doorbell IoT device based on such data. For example, the one assistant non-input system 180 may determine a change in the doorbell from an inactive state (e.g., no button was recently pressed) to an active state (a button was recently pressed), and the change in the doorbell state may be (e.g., via Figure 1 The user input is transmitted to the one or more cloud-based automated assistant components 119 and / or one or more assistant input devices 106. Note that although the user input is transmitted to the assistant non-input device 185 1185 is received from an automated assistant 120 (e.g., pressing a button on a doorbell), but the user input is not directed to the automated assistant 120 (hence the term "assistant non-input device"). As another example, assume that the assistant non-input device 185 1 1 is a smart thermostat IoT device with a microphone, but the smart thermostat does not include an automated assistant client 118. The individual can interact with the smart thermostat (e.g., using touch input or verbal input) to change the temperature, set a specific value as a setpoint for controlling the HVAC system via the smart thermostat, etc. However, the individual cannot communicate directly with the automated assistant 120 via the smart thermostat unless the smart thermostat includes the automated assistant client 118.

[0037] In various embodiments, the one or more cloud-based automated assistant components 119 may further include various engines. Figure 1 As shown, the one or more cloud-based automated assistant components 119 may further include an event detection engine 130, a device identification engine 140, an event processing engine 150, a semantic tagging engine 160, and a query / command processing engine 170. Although these various engines are Figure 1 106 and / or the assistant non-input device 185 may include one or more of these various engines. As another example, these various engines may be distributed across the assistant input device 106, and the assistant non-input device 185 may include one or more of these various engines and / or the one or more cloud-based automated assistant components 119.

[0038] In some embodiments, the event detection engine 130 can detect various events that occur in the ecosystem. In some versions of those embodiments, the event detection engine 130 can determine when a given one of the assistant input devices 106 and / or a given one of the assistant non-input devices 185 (e.g., a given one of the assistant devices) is newly added to the ecosystem or has moved within the ecosystem. For example, the event detection engine 130 can determine when a given one of the assistant devices is newly added to the ecosystem based on one or more wireless signals detected on the network 110 and via the device identification engine 140. For example, when the given one of the assistant devices is newly connected to one or more networks 110, the given one of the assistant devices can broadcast a signal indicating that it has been newly added to the network 110. As another example, the event detection engine 130 can determine when a given one of the assistant devices has moved within the ecosystem based on one or more wireless signals detected on the network 110. In these examples, the device identification engine 140 can process signals to determine that the given one assistant device is newly added to the network 110 and / or to determine that the given one assistant device has moved position within the ecosystem. The one or more wireless signals detected by the device identification engine 140 can be, for example, network signals and / or acoustic signals that are imperceptible to humans, and which can optionally include corresponding unique identifiers for the given one assistant device and / or other assistant devices that are located close to the given one assistant device. For example, when the given one assistant device is moved within the ecosystem, the device identification engine 140 can detect that one or more wireless signals are being transmitted by other assistant devices that are located close to the given one assistant device. These signals can be processed to determine that one or more other assistant devices that are located close to the given one assistant device are different from one or more assistant devices that were previously located close to the given one assistant device.

[0039] In some further versions of those embodiments, the automated assistant 120 may cause the given one assistant device that is newly added to the ecosystem or has moved position within the ecosystem to be assigned to an assistant device group (e.g., in the device topology representation of the ecosystem stored in the device topology database 193). For example, in an embodiment where the given one assistant device is newly added to the ecosystem, the given one assistant device may be added to an existing assistant device group, or a new assistant device group may be created that includes the given one assistant device. For example, if the given one assistant device is located close to multiple assistant devices belonging to the "kitchen" group (e.g., a smart oven, a smart coffee machine, an interactive stand-alone speaker associated with a unique identifier or tag indicating that it is located in the kitchen, and / or other assistant devices), the given one assistant device may be added to the "kitchen" group, or a new group may be created. As another example, in an embodiment where the given one assistant device has moved position within the ecosystem, the given one assistant device may be added to an existing assistant device group, or a new assistant device group may be created that includes the given one assistant device. For example, if the given assistant device is located close to multiple assistant devices belonging to the "kitchen" group mentioned above, but is now located close to multiple assistant devices belonging to the "garage" group (e.g., smart garage doors, smart door locks and / or other assistant devices), the given assistant device can be removed from the "kitchen" group and added to the "garage" group.

[0040] In some additional or alternative versions of those embodiments, the event detection engine 130 can detect the occurrence of an acoustic event. The occurrence of an acoustic event can be detected based on audio data received at one or more assistant input devices 106 and / or one or more assistant non-input devices 185 (e.g., one or more assistant devices). The audio data received at the one or more assistant devices can be processed by an event detection model stored in the ML model database 192. In these embodiments, each of the one or more assistant devices that detect the occurrence of an acoustic event includes a corresponding microphone.

[0041] In some further versions of those embodiments, the occurrence of an acoustic event may include ambient noise captured in audio data at one or more assistant devices (and optionally only includes the occurrence of ambient noise detected when voice reception is active at the one or more assistant devices). The ambient noise detected at each of the one or more assistant devices may be stored in the device activity database 191. In these embodiments, the event processing engine 150 may process the ambient noise detected at the one or more assistant devices using an ambient noise detection model, the ambient noise detection model being trained to classify the ambient noise into one or more semantic categories of multiple disparate semantic categories based on metrics generated when processing the ambient noise using the ambient noise detection model. The multiple disparate categories may include, for example, a movie or television sound category, a cooking sound category, a music sound category, a garage or workshop sound category, a courtyard sound category, and / or other semantically meaningful disparate categories of sound. For example, if the event processing engine 150 determines that the ambient noise processed using the ambient noise detection model includes sounds corresponding to microwave sounds, food sizzling on a frying pan, food processor processing food, etc., the event processing engine 150 may classify the ambient noise into a cooking sound category. As another example, if event processing engine 150 determines that ambient noise processed using the ambient noise detection model includes sounds corresponding to saw buzzing, hammers hitting, etc., event processing engine 150 can classify the ambient noise into a garage or workshop category. The classification of ambient noise detected at a particular device can also be used as a device-specific signal used to infer semantic tags for the assistant device (e.g., as described with respect to semantic tagging engine 160).

[0042] In some additional or alternative versions of those further embodiments, the occurrence of an acoustic event may include a hot word or a specific sound detected at one or more assistant devices. In these embodiments, the event processing engine 150 may process the audio data detected at the one or more assistant devices using a hot word detection model, the hot word detection model being trained to determine whether the audio data includes a specific word or phrase that invokes the automated assistant 120 based on a metric generated when processing the audio data using the hot word detection model. For example, the event processing engine 150 may process the audio data to determine whether the audio data captures a user's spoken utterance including "Assistant", "Hey assistant", "Okay, assistant", and / or any other word or phrase that invokes the automated assistant. In addition, the metric generated using the hot word detection model may include a corresponding confidence level or probability indicating whether the audio data includes a term or phrase that invokes the automated assistant 120. In some versions of these embodiments, if the metric meets a threshold, the event processing engine 150 may determine that the audio data captures the term or phrase. For example, if the event processing engine 150 generates a metric of 0.70 associated with audio data that captures a term or phrase that invokes the automated assistant 120, and the threshold is 0.65, the event processing engine 150 can determine that the audio data captures a term or phrase that invokes the automated assistant 120. The hot word detection model can be trained using, for example, supervised learning techniques. For example, multiple training instances can be obtained. Each training instance can include: a training instance input, the training instance input including audio data (or features of the audio data, such as mel-Cepstral frequency coefficients, audio waveforms, mel-bank features, and / or other acoustic features), and a corresponding training instance output, the training instance output including an indication of whether the training instance input includes a specific word or phrase that invokes the automated assistant 120. For example, if the training instance input includes audio data that captures a particular word or phrase, the corresponding training instance output can be assigned a label (e.g., "Yes") or a value (e.g., "1") indicating that the training instance input includes the particular word or phrase, and if the training instance input does not include the particular word or phrase, the corresponding training instance output can be assigned a different label (e.g., "No") or value (e.g., "0").

[0043] In these embodiments, additionally or alternatively, the event processing engine 150 may process the audio data detected at the one or more assistant devices using a sound detection model, the sound detection model being trained to determine whether the audio data includes a specific sound based on a metric generated when the audio data is processed using the sound detection model. Specific sounds may, for example, include glass breaking, dog barking, cat meowing, doorbell ringing, smoke alarm ringing, carbon monoxide detector ringing, baby crying, or knocking on the door. For example, the event processing engine 150 may process the audio data to determine whether the audio data captures any of these specific sounds. In this example, a single sound detection model may be trained to determine whether multiple specific sounds are captured in the audio data, or multiple sound detection models may be trained to determine whether a given specific sound is captured in the audio data. In addition, the metric generated using the sound detection model may include a corresponding confidence level or probability indicating whether the audio data includes a specific sound. In some versions of these embodiments, if the metric meets a threshold, the event processing engine 150 may determine that the audio data captures a specific sound. For example, if event processing engine 150 generates a metric of 0.70 associated with audio data capturing the sound of glass breaking, and the threshold is 0.65, event processing engine 150 may determine that the audio data captures the sound of glass breaking.

[0044] In various embodiments, the occurrence of an acoustic event can be captured by multiple assistant devices in the ecosystem. For example, multiple assistant devices in the environment can capture temporally corresponding audio data (e.g., temporally corresponding in that corresponding audio data is detected on multiple assistant devices at the same time or within a threshold duration). In these embodiments, and in response to a given assistant device in the ecosystem detecting audio data, the device identification engine 140 can identify one or more additional assistant devices that should have detected temporally corresponding audio data that also captured the acoustic event. For example, the device identification engine 140 can identify the one or more additional assistant devices that should have detected temporally corresponding audio data that also captured the acoustic event based on historically detecting one or more additional assistant devices that also captured temporally corresponding audio data for the acoustic event. In other words, the device identification engine 140 can expect that the one or more additional assistant devices should also capture audio data including the acoustic event because the given assistant device and the one or more additional assistant devices have historically captured temporally corresponding audio data including the same acoustic event (e.g., as described with respect to Figure 2A , Figure 2B and Figure 3 described).

[0045] In various embodiments, one or more device-specific signals generated or detected by a corresponding assistant device can be stored in a device activity database 191. In some embodiments, the device activity database 191 can correspond to a portion of a memory dedicated to device activity for that particular assistant device. In some additional or alternative embodiments, the device activity database 191 can be associated with a memory that is in communication with the assistant device (e.g., via Figure 1 The device activity may correspond to a memory of a remote system of network 110). The device activity may be used to generate candidate semantic tags for a given one of the assistant devices (e.g., as described with respect to semantic tagging engine 160). The device activity may include, for example, queries or requests received at the corresponding assistant device (and / or semantic categories associated with each of a plurality of queries or requests), commands executed at the corresponding assistant device (and / or semantic categories associated with each of a plurality of commands), ambient noise detected at the corresponding assistant device (and / or semantic categories associated with various instances of ambient noise), unique identifiers or tags of any assistant devices that are in location proximity to the given assistant device (e.g., identified via event detection engine 140), and / or any other data received, generated, and / or executed by the corresponding assistant device.

[0046] In some embodiments, the semantic tag engine 160 can process one or more device-specific signals to generate candidate semantic tags for a given one of the assistant devices (e.g., a given one of the assistant input devices 106 and / or a given one of the assistant non-input devices 185) based on the one or more device-specific signals. Candidate semantic tags can be generated using one or more rules (optionally heuristically defined) or machine learning models (e.g., stored in the ML model database 192). For example, one or more heuristically defined rules can indicate candidate semantic tags associated with each semantic category, where the one or more device-specific signals should be generated. For example, assume that the device-specific signals are classified into a "kitchen" category, a "cooking" category, a "bedroom" category, and a "living room" category. In this example, the candidate semantic tags may include a first candidate semantic tag "kitchen assistant device", a second candidate semantic tag "cooking assistant device", a third candidate semantic tag "bedroom assistant device", and a fourth semantic tag "living room assistant device". As another example, the one or more device-specific signals (or the one or more semantic categories corresponding thereto) can be processed using a machine learning model that is trained to generate candidate semantic labels. For example, the machine learning model can be trained based on multiple training instances. Each training instance can include a training instance input and a corresponding training instance output. The training instance input can include, for example, one or more device-specific signals and / or one or more semantic categories, and the corresponding training instance output can include, for example, a true value output of a semantic label that should be assigned based on the training instance input.

[0047] In some versions of those embodiments, in response to determining that a given assistant device has been newly added to the ecosystem and / or has moved location within the ecosystem, a given assistant device that generates a candidate semantic tag can be identified. In some additional or alternative versions of those embodiments, a given assistant device that generates a candidate semantic tag can be identified periodically (e.g., once a month, once every six months, once a year, etc.). In some additional or alternative versions of those embodiments, in response to determining that a portion of the ecosystem in which the given assistant device is located has been repurposed (e.g., a room in the main residence of the ecosystem has been repurposed from an activity room to a bedroom), a given assistant device that generates a candidate semantic tag can be identified. In these embodiments, the event detection engine 130 can be used to identify the given assistant device. About Figure 2A and Figure 2B Identifying a given assistant device in these and other ways is described.

[0048] In some embodiments, the semantic tag engine 160 can select a given semantic tag for a given assistant device from candidate semantic tags based on one or more device-specific signals. In embodiments where candidate semantic tags for a given assistant device are generated based on queries, requests, and / or commands stored in the device activity database 191 (or text corresponding thereto), a semantic classifier (e.g., stored in the ML model database 192) can be used to process the queries, requests, and / or commands to index device activities for a given assistant device into one or more different semantic categories corresponding to queries, requests, and / or commands of completely different types. Candidate semantic tags can be generated based on the semantic categories into which the queries, commands, and / or requests are classified, and a given semantic tag selected for a given assistant device can be selected based on the amount of multiple queries, requests, and / or commands classified in a given semantic category. For example, assume that a given assistant device has previously received nine queries related to obtaining cooking recipes and two commands related to controlling smart lights in an ecosystem. In this example, the candidate semantic tags may include, for example, a first semantic tag "kitchen equipment" and a second semantic tag "control smart light device". Furthermore, the semantic tagging engine 160 may select the first semantic tag “kitchen equipment” as the given semantic tag for the given assistant device because historical usage of the given assistant device indicates that it is mainly used for cooking-related activities.

[0049] In some embodiments, the semantic classifier stored in the ML model database 192 can be a natural language understanding engine (e.g., implemented by the NLP module 122 described below). Intents determined based on processing queries, commands, and / or requests previously received at the assistant device can be mapped to one or more semantic categories. Note that the multiple disparate semantic categories described herein can be defined using various levels of granularity. For example, a semantic category can be associated with a genus category of smart device commands and / or a species category of the genus, such as a category of smart lighting commands, a category of smart thermostat commands, and / or a category of smart camera commands. In other words, each category can have a unique set of intents associated with it determined by a semantic classifier, although some intents of a category can also be associated with additional categories. In some additional or alternative embodiments, a semantic classifier stored in the ML model database 192 can be used to generate text embeddings (e.g., low-dimensional representations, such as word2vec representations) corresponding to the text of the query, command, and / or request. These embeddings can be points within an embedding space where semantically similar words or phrases are associated with the same or similar portions of the embedding space. Furthermore, these portions of the embedding space may be associated with one or more of a plurality of disparate semantic categories, and a given embedding may be classified into a given semantic category if a distance measure between the given embedding and the one or more portions of the embedding space satisfies a distance threshold. For example, cooking-related words or phrases may be associated with a first portion of the embedding space associated with a "cooking" semantic label, weather-related words or phrases may be associated with a second portion of the embedding space associated with a "weather" semantic label, and so on.

[0050] In embodiments where the one or more device-specific signals additionally or alternatively include ambient noise activity, instances of ambient noise may be processed using an ambient noise detection model (e.g., stored in the ML model database 192) to index device activity for a given assistant device into one or more different semantic categories corresponding to disparate types of ambient noise. Candidate semantic tags may be generated based on the semantic categories into which the ambient noise instances are classified, and a given semantic tag selected for a given assistant device may be selected based on the number of ambient noise instances classified in a particular semantic category. For example, assume that the ambient noise detected at a given assistant device (and optionally only when voice recognition is active) primarily includes ambient noise classified as cooking sounds. In this example, the semantic tagging engine 160 may select a semantic tag of "kitchen equipment" as the given semantic tag for the given assistant device because the ambient noise captured in the audio data indicates that the device is located near cooking-related activities.

[0051] In some embodiments, the environmental noise detection model stored in the ML model database 192 can be trained to detect specific sounds, and it can be determined whether the instance of environmental noise includes a specific sound based on the output generated across the environmental noise detection model. The environmental noise detection model can be trained using, for example, supervised learning techniques. For example, multiple training instances can be obtained. Each training instance can include a training instance input containing environmental noise and a corresponding training instance output, which includes an indication of whether the training instance input includes a specific sound that the environmental noise detection model is being trained to detect. For example, if the environmental noise detection model is being trained to detect glass breaking sounds, a label (e.g., "Yes") or a value (e.g., "1") can be assigned to the training instance that includes the glass breaking sound, and a different label (e.g., "No") or a value (e.g., "0") can be assigned to the training instance that does not include the glass breaking sound. In some additional or alternative embodiments, the ambient noise detection model stored in the ML model database 192 can be used to generate audio embeddings (e.g., low-dimensional representations of ambient noise instances) based on ambient noise instances (or their acoustic features, such as mel-Cepstral frequency coefficients, original audio waveforms, and / or other acoustic features). These embeddings can be points within an embedding space where similar sounds (or acoustic features that capture sounds) are associated with the same or similar portions of the embedding space. In addition, these portions of the embedding space can be associated with one or more semantic categories of a plurality of disparate semantic categories, and if a distance measure between a given embedding and one or more portions of the embedding space satisfies a distance threshold, the given embedding can be classified into a given semantic category. For example, an instance of glass breaking can be associated with a first portion of the embedding space that is associated with a "glass breaking" sound, an instance of a doorbell ringing can be associated with a second portion of the embedding space that is associated with a "doorbell" sound, and so on.

[0052] In embodiments where the one or more device-specific signals additionally or alternatively include unique identifiers or tags of additional assistant devices that are located proximate to the given assistant device, candidate semantic tags can be generated based on those unique identifiers or tags, and a given semantic tag selected for the given assistant device can be selected based on one or more unique identifiers or tags of the additional assistant devices. For example, assume that a first tag "smart oven" is associated with a first assistant device that is located proximate to the given assistant device, and a second tag "smart coffee maker" is associated with a second assistant device that is located proximate to the given assistant device. In this example, the semantic tagging engine 160 can select a semantic tag of "kitchen equipment" as the given semantic tag for the given assistant device because the tags associated with the additional assistant devices that are located proximate to the given assistant device are cooking-related. The unique identifiers or tags are processed using a semantic classifier stored in the ML model database 192 in the same or similar manner as described above with respect to processing queries, commands, and / or requests.

[0053] In some embodiments, the semantic tag engine 160 may automatically assign a given semantic tag to a given assistant device in a device topology representation of an ecosystem (e.g., stored in the device topology database 193). In some additional or alternative embodiments, the semantic tag engine 160 may cause the automated assistant 120 to generate a prompt including candidate semantic tags. The prompt may request a user associated with the ecosystem to select one of the candidate tags as the given semantic tag. In addition, the prompt may be presented visually and / or audibly at a given assistant device (which may or may not be the given assistant device to which the given semantic tag is being assigned) and / or the user's client device (e.g., a mobile device). In response to receiving a selection of one of the candidate tags as the given semantic tag, the selected given semantic tag may be assigned to a given assistant device in a device topology representation of an ecosystem (e.g., stored in the device topology database 193). In some versions of these embodiments, the given semantic tag assigned to the given assistant device may be added to a semantic tag list of the given assistant device. In other words, multiple semantic tags may be associated with a given assistant device. In other versions of these embodiments, the given semantic tag assigned to the given assistant device may replace any other semantic tags of the given assistant device. In other words, only a single semantic tag can be associated with a given assistant device.

[0054] In some embodiments, the query / command processing engine 170 can process queries, requests, or commands directed to the automated assistant 120 and received via one or more assistant input devices 106. The query / command processing engine 170 can process the query, request, or command to select one or more assistant devices to satisfy the query or command. Note that the one or more assistant devices selected to satisfy the query or command may be different from the one or more assistant input devices 106 that received the query or command. The query / command processing engine 170 can select the one or more assistant devices to satisfy the spoken utterance based on one or more criteria. The one or more criteria may include, for example, the proximity of one or more devices to the user providing the spoken utterance (e.g., determined using the presence sensor 105 described below), the device capabilities of one or more devices in the ecosystem, semantic tags assigned to the one or more assistant devices, and / or other criteria for selecting assistant devices to satisfy the spoken utterance.

[0055] For example, suppose a display device is required to satisfy a spoken utterance. In this example, the candidate assistant devices considered in selecting a given assistant device to satisfy a spoken utterance can be limited to those assistant devices that include a display device. If multiple assistant devices in the ecosystem include a display device, a given assistant device that includes the display device and is closest to the user can be selected to satisfy the utterance. Conversely, in embodiments where only a speaker is required to satisfy a spoken utterance (e.g., a display device is not required to satisfy a spoken utterance), the candidate assistant devices considered in selecting a given assistant device to satisfy a spoken utterance can include those assistant devices that have a speaker, regardless of whether they include a display device.

[0056] As another example, assume that a spoken utterance includes semantic attributes that match a semantic tag assigned to a given assistant device. The query / command processing engine 170 may determine that the semantic attributes of the spoken utterance match the semantic tag assigned to the given assistant device by generating a first embedding corresponding to one or more terms of the spoken utterance (or text corresponding thereto), and a second embedding corresponding to one or more terms of the semantic tag assigned to the given assistant device, and comparing the embeddings to determine whether a distance measure between the embeddings satisfies a distance threshold indicating that the embeddings match (e.g., whether it is an exact match or a soft match). In this example, the query / command processing engine 170 may select a given assistant device to satisfy the spoken utterance based on the spoken utterance matching the semantic tag (and optionally in addition to or in lieu of the proximity of the user providing the spoken utterance to the given assistant device). In this way, the selection of the satisfying spoken utterance for the assistant device may be biased toward the semantic tag assigned to the assistant device as described herein.

[0057] In various embodiments, one or more assistant input devices 106 may include one or more corresponding presence sensors 105 1-N (also referred to herein as simply "presence sensors 105"), which can be configured to provide signals indicating detected presence, particularly human presence, with approval of the corresponding user. In some embodiments of those embodiments, the automated assistant 120 can identify one or more assistant input devices 106 to satisfy a user's spoken utterance associated with the ecosystem based at least in part on the user's presence at the one or more assistant input devices 106. The spoken utterance can be satisfied by presenting responsive content (e.g., audibly and / or visually) at the one or more assistant input devices 106, by causing one or more assistant input devices 106 to be controlled based on the spoken utterance, and / or by causing one or more assistant input devices 106 to perform any other action to satisfy the spoken utterance. As described herein, the automated assistant 120 can leverage data determined based on the corresponding presence sensors 105 to determine those assistant input devices 106 based on where the user is near or most recently near, and provide corresponding commands only to those assistant input devices 106. In some additional or alternative embodiments, the automated assistant 120 can leverage data determined based on corresponding presence sensors 105 to determine whether any user (any user or a specific user) is currently in proximity to any assistant input device 106, and can optionally suppress the provision of commands based on determining that no user (any user or a specific user) is in proximity to any assistant input device 106.

[0058] The corresponding presence sensors 105 can appear in various forms. Some assistant input devices 106 can be equipped with one or more digital cameras configured to capture and provide signals indicative of movement detected within their field of view. Additionally or alternatively, some assistant input devices 106 can be equipped with other types of light-based presence sensors 105, such as passive infrared ("PIR") sensors that measure infrared ("IR") light radiated from objects within their field of view. Additionally or alternatively, some assistant input devices 106 can be equipped with presence sensors 105 that detect acoustic (or pressure) waves, such as one or more microphones. In addition, in addition to the assistant input devices 106, one or more assistant non-input devices 185 can also additionally or alternatively include the corresponding presence sensors 105 described herein, and the automated assistant 120 can additionally utilize signals from such sensors to determine whether and / or how to satisfy the spoken utterance in accordance with the embodiments described herein.

[0059] Additionally or alternatively, in some embodiments, presence sensor 105 can be configured to detect other phenomena associated with human presence or device presence in the ecosystem. For example, in some embodiments, a given assistant device can be equipped with a presence sensor 105 that detects various types of wireless signals (e.g., such as radio, ultrasonic, electromagnetic, etc. waves) emitted by, for example, other assistant devices carried / operated by a particular user (e.g., mobile devices, wearable computing devices, etc.) and / or other assistant devices in the ecosystem (e.g., as described with respect to event detection engine 130). For example, some assistant devices can be configured to emit waves that are not perceptible to humans, such as ultrasonic or infrared waves, which can be detected by one or more assistant input devices 106 (e.g., via ultrasonic / infrared receivers, such as microphones with ultrasonic capabilities).

[0060] Additionally or alternatively, various assistant devices may emit other types of waves that are imperceptible to humans, such as radio waves (e.g., Wi-Fi, Bluetooth, cellular, etc.), which can be detected by other assistant devices (e.g., mobile devices, wearable computing devices, etc.) carried / operated by a particular user and used to determine the specific location of the operating user. In some embodiments, Wi-Fi triangulation can be used to detect the location of a person, for example, based on the Wi-Fi signal of the assistant device. In other embodiments, other wireless signal characteristics (such as flight time, signal strength, etc.) can be used by various assistant devices alone or in combination to determine the location of a particular person based on signals emitted by other assistant devices carried / operated by a particular user.

[0061] Additionally or alternatively, in some embodiments, one or more assistant input devices 106 may perform speech recognition to identify a user from their speech. For example, some instances of the automated assistant 120 may be configured to match speech to a profile of a user, e.g., for the purpose of providing / restricting access to various resources. In some embodiments, movement of the speaker may then be determined, e.g., by the presence sensor 105 of the assistant device. In some embodiments, based on such detected movement, the location of the user may be predicted, and based at least in part on the proximity of the assistant devices to the user's location, that location may be assumed to be the location of the user when causing any content to be presented at those assistant devices. In some embodiments, the user may simply be assumed to be in the last location where he or she hired the automated assistant 120, particularly where not much time has passed since the last hire.

[0062] Each assistant input device 106 further includes a corresponding user interface component 107 1-N (also referred to herein as “user interface component 107”), the user interface component 1071-N Each may include one or more user interface input devices (e.g., microphone, touch screen, keyboard) and / or one or more user interface output devices (e.g., display, speaker, projector). As an example, assistant input device 106 1 User interface components 107 1 may include only a speaker and a microphone, while the assistant input device 106 N User interface components 107 N A speaker, a touch screen, and a microphone may be included. Additionally or alternatively, in some embodiments, assistant non-input device 185 may include one or more user interface input devices and / or one or more user interface output devices of user interface component 107, but the user input device of assistant non-input device 185 (if present) may not allow the user to interact directly with automated assistant 120.

[0063] Each assistant input device 106 and / or any other computing device operating one or more cloud-based automated assistant components 119 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components to facilitate communication over a network. The operations performed by one or more assistant input devices 106 and / or by automated assistant devices 120 may be distributed across multiple computer systems. Automated assistant 120 may be implemented, for example, as a computer running on one or more computers in one or more locations and communicated over a network (e.g., Figure 1 Any computer program coupled to each other in the network 110).

[0064] As described above, in various implementations, each assistant input device 106 may operate a corresponding automated assistant client 118. In various embodiments, each automated assistant client 118 may include a corresponding speech capture / text to speech (TTS) / speech to text (STT) module 114. 1-N (Also referred to herein simply as “voice capture / TTS / STT module 114 ”.) In other implementations, one or more aspects of a corresponding voice capture / TTS / STT module 114 may be implemented separately from a corresponding automated assistant client 118 .

[0065] Each respective voice capture / TTS / STT module 114 may be configured to perform one or more functions including, for example: capturing the user's voice (voice capture, e.g., via a respective microphone (which in some cases may include presence sensor 105)); converting the captured audio into text and / or other representations or embeddings (STT) using a speech recognition model stored in ML model database 192; and / or converting text into speech (TTS) using a speech synthesis model stored in ML model database 192. Instances of these models may be stored locally at each respective assistant input device 106 and / or accessible by the assistant input device (e.g., via Figure 1 In some embodiments, because one or more assistant input devices 106 may be relatively constrained in terms of computing resources (e.g., processor cycles, memory, battery, etc.), a corresponding voice capture / TTS / STT module 114 local to each assistant input device 106 may be configured to convert a limited number of different spoken phrases into text (or into other forms, such as lower dimensional embeddings) using a speech recognition model. Other voice input may be sent to one or more cloud-based automated assistant components 119, which may include a cloud-based TTS module 116 and / or a cloud-based STT module 117.

[0066] The cloud-based STT module 117 can be configured to fully utilize the virtually unlimited cloud resources to convert the audio data captured by the speech capture / TTS / STT module 114 into text (which can then be provided to the natural language processor 122) using the speech recognition model stored in the ML model database 192. The cloud-based TTS module 116 can be configured to fully utilize the virtually unlimited cloud resources to convert text data (e.g., text formulated by the automated assistant 120) into computer-generated speech output using the speech synthesis model stored in the ML model database 192. In some embodiments, the cloud-based TTS module 116 can provide the computer-generated speech output to one or more assistant devices for direct output, for example, using the corresponding speakers of the corresponding assistant devices. In other embodiments, the text data generated by the automated assistant 120 using the cloud-based TTS module 116 (e.g., client device notifications included in commands) can be provided to the speech capture / TTS / STT module 114 of the corresponding assistant device, which can then locally convert the text data into computer-generated speech using the speech synthesis model and cause the computer-generated speech to be presented via the local speakers of the corresponding assistant device.

[0067] The automated assistant 120 (and in particular, the one or more cloud-based automated assistant components 119) may include a natural language processing (NLP) module 122, the aforementioned cloud-based TTS module 116, the aforementioned cloud-based STT module 117, and other components, some of which are described in more detail below. In some embodiments, one or more engines and / or modules of the automated assistant 120 may be omitted, combined, and / or implemented in a component separate from the automated assistant 120. An instance of the NLP module 122 may additionally or alternatively be implemented locally at the assistant input device 106.

[0068] In some implementations, the automated assistant 120 generates responsive content in response to various inputs generated by a user of one of the assistant input devices 106 during a human-machine conversation session with the automated assistant 120. The automated assistant 120 may provide responsive content (e.g., when separate from the assistant device, in Figure 1 100) for presentation to the user as part of a conversation session via assistant input devices 106 and / or assistant non-input devices 185. For example, automated assistant 120 may generate responsive content in response to free-form natural language input provided via one of assistant input devices 106. As used herein, free-form input is input formulated by a user and is not constrained by a set of options presented for selection by the user.

[0069] NLP module 122 of automated assistant 120 processes natural language input generated by a user via assistant input device 106, and can generate annotated output for use by automated assistant 120, assistant input device 106, and / or one or more other components of assistant non-input devices 185. For example, NLP module 122 can process natural language free-form input generated by a user via one or more corresponding user interface input devices of assistant input devices 106. The annotated output generated based on processing the natural language free-form input can include one or more annotations of the natural language input, and optionally one or more (e.g., all) terms of the natural language input.

[0070] In some embodiments, the NLP module 122 is configured to recognize and annotate various types of grammatical information in natural language input. For example, the NLP module 122 may include a part of speech tagger configured to annotate terms using the grammatical roles of the terms. In some embodiments, the NLP module 122 may additionally and / or alternatively include an entity tagger (not depicted) that is configured to annotate entity references in one or more segments, such as references to people (including, for example, literary characters, celebrities, public figures, etc.), organizations, locations (real and fictional), etc. In some embodiments, data about entities may be stored in one or more databases, such as in a knowledge graph (not depicted). In some embodiments, a knowledge graph may include nodes representing known entities (and in some cases, entity attributes), and edges connecting nodes and representing relationships between entities.

[0071] The entity tagger of the NLP module 122 may annotate references to entities at a high level of granularity (e.g., enabling identification of all references to a class of entities such as people), and / or annotate references to entities at a lower level of granularity (e.g., enabling identification of all references to a specific entity such as a specific person). The entity tagger may rely on the content of the natural language input to resolve specific entities and / or may optionally communicate with a knowledge graph or other entity database to resolve specific entities.

[0072] In some implementations, the NLP module 122 may additionally and / or alternatively include a coreference resolver (not depicted) that is configured to group or "cluster" references to the same entity based on one or more contextual clues. For example, based on a mention of a "front door lock" in a client device notification presented immediately before receiving the natural language input "lock it," a coreference resolver may be utilized to resolve the term "it" in the natural language input "lock it" into "front door lock."

[0073] In some embodiments, one or more components of the NLP module 122 can rely on annotations from one or more other components of the NLP module 122. For example, in some embodiments, a named entity tagger can rely on annotations from a coreference resolver and / or a dependency parser to annotate all mentions of a particular entity. Also, for example, in some embodiments, a coreference resolver can rely on annotations from a dependency parser in clustering references to the same entity. In some embodiments, when processing a particular natural language input, one or more components of the NLP module 122 can determine one or more annotations using relevant data external to the particular natural language input—such as an assistant input device notification presented immediately prior to receiving the natural language input on which the assistant input device notification is based.

[0074] although Figure 1 Depicted as having a particular configuration of components implemented by an assistant device and / or server and depicted as having an assistant device and / or server communicating over a particular network, it should be understood that this is for illustrative purposes only and is not meant to be limiting. For example, assistant input device 106 and assistant non-input devices may be directly communicatively coupled to each other via one or more networks (not depicted). As another example, operations of the one or more cloud-based automated assistant components 119 may be implemented locally at one or more assistant input devices 106 and / or one or more assistant non-input devices. As yet another example, instances of various ML models stored in ML model database 192 may be stored locally at the assistant device, and / or instances of device topology representations of the ecosystem stored in device topology database 193 may be stored locally at the assistant input device. In addition, in the process of communicating via Figure 1 In embodiments where any of one or more networks 110 transmits data (e.g., device activity, audio data or recognized text corresponding thereto, device topology representations, and / or any other data described herein), the data may be encrypted, filtered, or otherwise protected in any manner to ensure the privacy of the user.

[0075] By using the technology described herein to detect and verify the occurrence of acoustic events using multiple assistant devices in the ecosystem, the amount of false positives of acoustic events can be reduced. Therefore, both computing resources and network resources can be saved. For example, by using the technology described herein to detect and verify the occurrence of hot word events in the ecosystem, the components and / or functionality of the automated assistant can be kept dormant, and when these technologies are missing, the components and / or functionality of the automated assistant may have been activated, even if they are not intended to be activated. As another example, by using the technology described herein to detect and verify the occurrence of specific sounds in the ecosystem, the notification displayed to the user associated with the ecosystem can be retained, and when these technologies are missing, even if the acoustic event does not actually occur, the notification may also have been displayed. In addition, by using the technology described herein to anticipate which assistant devices in the ecosystem should detect and capture the audio data corresponding to the time of the acoustic event, the amount of assistant devices processing audio data can be reduced. As a result, both computing resources and network resources can be saved. For example, by using the techniques described herein to anticipate assistant devices in an ecosystem that should have detected an acoustic event, other assistant devices that may have captured temporally corresponding audio data that may not correspond to the acoustic event may not be considered when determining whether the acoustic event is an actual acoustic event.

[0076] Reference now Figure 2A , Figure 2B and Figure 3 supply Figure 1Additional descriptions of the various components. Figure 2A , Figure 2B and Figure 3 A home floor plan is depicted in FIG. The depicted floor plan includes a plurality of rooms 250 to 262. A plurality of assistant input devices 106 1-5 Each assistant input device 106 is disposed throughout at least some of the rooms. 1-5 An instance of an automated assistant client 118 configured with selected aspects of the present disclosure may be implemented and may include one or more input devices, such as a microphone capable of capturing speech spoken by a nearby person. For example, a first assistant input device 106 in the form of an interactive stand-alone speaker and a display device (e.g., a display screen, a projector, etc.) 1 Deployed in room 250, which in this example is a kitchen. A second assistant input device 106 in the form of a so-called "smart" TV (e.g., a networked TV having one or more processors implementing a corresponding instance of the automated assistant client 118) 2 is deployed in room 252, which in this example is an activity room. A third assistant input device 106 in the form of an interactive stand-alone speaker without a display 3 is deployed in room 254, which in this example is a bedroom. A fourth assistant input device 106 in the form of another interactive independent speaker 4 is deployed in room 256, which in this example is the living room. A fifth assistant input device 106 also in the form of a smart TV 5 is deployed in a room 250, which in this example is a kitchen.

[0077] Although in Figure 2A , Figure 2B and Figure 3 Not depicted, but multiple assistant input devices 106 1-4 The communication may be via one or more wired or wireless WANs and / or LANs (e.g., Figure 1 In addition, other assistant input devices—particularly mobile devices such as smartphones, tablets, laptops, wearable devices, etc.—may also be present, for example, carried by one or more people in the home, and may or may not also be connected to the same WANs and / or LANs. It should be understood that in Figure 2A , Figure 2B and Figure 3The configuration of assistant input devices depicted in is only one example; more or fewer and / or different assistant input devices 106 may be deployed in any number of other rooms and / or areas of a home, and / or in locations outside of a home (e.g., businesses, hotels, public places, airports, vehicles, and / or other locations or spaces).

[0078] exist Figure 2A , Figure 2B and Figure 3 Further depicted in the figure are multiple assistant non-input devices 185 1-5 For example, a first assistant non-input device in the form of a smart doorbell 185 1 Deployed on the exterior of the home near the front door of the home. Second assistant non-input device in the form of a smart lock 185 2 Deployed on the exterior of the home on the front door of the home. A third assistant non-input device in the form of a smart washing machine 185 3 Deployed in room 262, which in this example is a laundry room. A fourth assistant non-input device 185 in the form of a door open / close sensor 4 Deployed near the back door of room 262 and detects whether the back door is open or closed. A fifth assistant non-input device 185 in the form of a smart thermostat 5 Deployed in room 252, which in this example is an activity room.

[0079] Each assistant non-input device 185 can be connected to a corresponding assistant non-input system 180 (such as Figure 1 as shown) communication (e.g., via Figure 1 The assistant non-input device 185 may be connected to a network 110 in the system 100 to provide data to the corresponding assistant non-input system 180 and may be controlled based on commands provided by the corresponding assistant non-input system 180. The one or more assistant non-input devices 185 may additionally or alternatively communicate directly with the one or more assistant input devices 106 (e.g., via a Figure 1 100) to provide data to one or more assistant input devices 106, and optionally be controlled based on commands provided by one or more assistant input devices 106. It should be understood that Figure 2A , Figure 2B and Figure 3 The configuration of assistant non-input device 185 depicted in is only one example; more or fewer and / or different assistant non-input devices 185 may be deployed in any number of other rooms and / or areas of a home, and / or locations outside of a home (e.g., businesses, hotels, public places, airports, vehicles, and / or other locations or spaces).

[0080] In various embodiments, acoustic events may be detected at multiple assistant devices (e.g., any assistant input device 106 and / or assistant non-input device 185) co-located in an ecosystem. For example, an acoustic event may be captured in audio data detected via a microphone of a given assistant device in the ecosystem, and may also be captured in corresponding audio data detected via a corresponding microphone of at least one additional assistant in the ecosystem. The audio data captured by the assistant device may be processed using an event detection model stored locally on the corresponding assistant device and / or remotely stored at a server and accessible by the corresponding assistant device to generate corresponding metrics associated with the acoustic event. In addition, each of these metrics may be processed to determine whether the acoustic event corresponds to an actual acoustic event that occurred in the ecosystem. If the acoustic event is an actual acoustic event, an action may be performed based on the occurrence of the actual acoustic event in the ecosystem. Otherwise, the audio data will be discarded.

[0081] In some embodiments, one or more assistant devices (e.g., a given assistant device and the at least one additional assistant device) can locally process corresponding audio data at the corresponding assistant device and use corresponding event detection models to generate corresponding metrics indicating whether the corresponding audio data detected at each of these assistant devices captures an acoustic event. In some additional or alternative embodiments, one or more assistant devices can each transmit audio data to a given assistant device in the ecosystem (e.g., one assistant device that detects audio data or a different assistant device that does not detect any audio data) to locally process the corresponding audio data at the given assistant device and use the event detection model to generate corresponding metrics indicating whether the corresponding audio data detected at each of these assistant devices captures an acoustic event. In some additional or alternative embodiments, one or more assistant devices can each transmit audio data to a remote system different from the ecosystem to remotely process the corresponding audio data at the remote system and use the event detection model to generate corresponding metrics indicating whether the corresponding audio data detected at each of these assistant devices captures an acoustic event.

[0082] In some embodiments, in response to a given assistant device detecting audio data capturing an acoustic event, the at least one additional assistant device in the ecosystem can be identified. If the acoustic event detected at the given assistant device is in fact an actual acoustic event, the at least one additional assistant device identified in the ecosystem should have detected the temporally corresponding audio data that also captured the acoustic event. The audio data captured by the assistant device can be considered temporally corresponding audio data based on, for example, corresponding timestamps associated with the audio data captured at the assistant device. For example, the audio data can be considered temporally corresponding audio data when the timestamps match or are within a threshold duration of each other's time (e.g., within a few milliseconds, a few seconds, or any other suitable duration). In some versions of those embodiments, the at least one additional assistant device can be identified based on, for example, a given assistant device and the at least one additional assistant device that have historically detected audio data capturing the same acoustic event. In some additional or alternative versions of those embodiments, the at least one additional assistant device can be identified, for example, based on a given assistant device and the at least one additional assistant device belonging to the same group of assistant devices in a device topology representation of the ecosystem.

[0083] In some embodiments, the acoustic event is a hot word event, and the event detection model corresponds to the hot word detection model. A hot word event can correspond to detecting a particular word or phrase that, when detected, causes one or more components or functionalities of the automated assistant to be activated. For example, and with specific reference to Figure 2A and Figure 2B , assuming that Figure 2A and Figure 2B A user 101 associated with the ecosystem depicted in FIG. 1 provides a spoken utterance of “Hey Assistant…” which is detected by multiple assistant devices in the ecosystem. Figure 2A As shown, when the spoken utterance is provided, the user 101 may be located in a room 250, which in this example is a kitchen. In this example, it is assumed that the first assistant input device 106 1 Audio data corresponding to the spoken utterance “Hey Assistant…” is detected (e.g., from the first assistant input device 106). 1 ). It is further assumed that the fifth assistant input device 106 5 and the fifth assistant non-input device 185 5 (assuming the smart thermostat includes a microphone) is also expected to detect audio data capturing the hotword (e.g., also indicated by a dash). In this example, based on the first assistant input device 106 being included in the same device group (e.g., the "kitchen" group), the first assistant input device 106 is also expected to detect audio data capturing the hotword (e.g., also indicated by a dash).1 and a fifth assistant input device 106 5 and / or based on the first assistant input device 106 having previously detected temporally corresponding audio data capturing the same spoken utterance 1 and a fifth assistant input device 106 5 , it is expected that the fifth assistant input device 106 5 In addition, based on the first assistant input device 106 having previously detected the temporally corresponding audio data capturing the same spoken utterance, 1 and the fifth assistant non-input device 185 5 (For example, assuming that the first assistant input device 106 1 Belongs to the "Kitchen" group, and the fifth assistant is not an input device 185 5 Belongs to the "Activity Room" group), the fifth assistant non-input device 185 can be expected 5 Audio data detected.

[0084] In addition, it is assumed that based on the first assistant input device 106 1 The metric generated by the audio data detected at the fifth assistant input device 106 indicates that the audio data captures the hot word, and it is assumed that based on 5 The metric generated by the audio data detected at the location indicates that the audio data captures the hot word, but based on the fifth assistant non-input device 185 5 The metric generated by the audio data detected at the input device 185 indicates that the audio data does not capture the hot word. In this example, this can determine that the hot word detected in the ecosystem is actually a hot word, and one or more components or functionalities of the automated assistant can be activated even if the fifth assistant non-input device 185 5 Indicates that the audio data did not capture the hot word (or even if the fifth assistant is not an input device 185 5 No audio data is detected at all) because the other assistant devices indicate that the hotword is captured in the spoken utterance. Instead, it is assumed that based on the first assistant input device 106 1 The metric generated by the audio data detected at the fifth assistant input device 106 indicates that the audio data captures the hot word, but based on 5 and the fifth assistant non-input device 185 5 The metric generated by the audio data detected at indicates that the audio data does not capture the hot word. In this example, it can be determined that the hot word detected in the ecosystem is not actually a hot word, and one or more components or functionalities of the automated assistant can remain dormant even if the first assistant input device 106 1 Indicates that the audio data captures the hotword, as other assistant devices indicate that the hotword is not captured in the spoken utterance.

[0085] As another example, and if Figure 2BAs shown, when providing spoken utterances, user 101 may be located in room 252, which in this example is an activity room. In this example, it is assumed that in the first assistant input device 106 1 The first assistant input device 106 detects audio data corresponding to the spoken utterance “Hey Assistant…” 1 ). It is further assumed that the fifth assistant non-input device 185 5 (assuming the smart thermostat includes a microphone) audio data capturing the hotword is also expected to be detected (e.g., also indicated by a dash). In this example, based on a second assistant input device 106 included in the same group of devices (e.g., the "Activity Rooms" group), 2 and the fifth assistant non-input device 185 5 and / or based on a second assistant input device 106 that has previously detected temporally corresponding audio data capturing the same spoken utterance 5 and the fifth assistant non-input device 185 5 , it is expected that the fifth assistant non-input device 185 5 Audio data detected.

[0086] In addition, it is assumed that based on the second assistant input device 106 2 The metric generated by the audio data detected at the location indicates that the audio data captures the hot words, and assumes that based on the fifth assistant non-input device 185 5 The metric generated from the audio data detected at the second assistant input device 106 indicates that the audio data captures a hot word. In this example, it can be determined that the hot word detected in the ecosystem is actually a hot word, and one or more components or functionalities of the automated assistant can be activated. In contrast, assume that based on the second assistant input device 106 2 The metric generated by the audio data detected at the location indicates that the audio data captures the hot word, but based on the fifth assistant non-input device 185 5 The metric generated by the audio data detected at indicates that the audio data does not capture the hot word. In this example, the audio data can be generated based on the audio data detected at the fifth assistant non-input device 185. 5 The method of determining whether a hot word detected in the ecosystem is actually a hot word is to determine whether a metric generated based on the audio data detected at the location is within a threshold range indicating that the metric indicates a hot word. For example, assume that the metric associated with the hot word event is a probability. Further assume that based on the audio data generated by the second assistant input device 106 2 The audio data detected generates a metric of 0.7 based on the non-input device 185 of the fifth assistant 5 The metric generated by the detected audio data is 0.6, and the threshold for determining whether a hot word is an actual hot word is 0.65. As a result, the acoustic event may not be considered a hot word because the second assistant input device 1062 The associated metric indicates the occurrence of the hot word, but with the fifth assistant non-input device 106 2 The associated metric does not indicate the occurrence of a hot word. However, further assuming that if based on the non-input device 185 of the fifth assistant 5 If the metric generated by the detected audio data (e.g., 0.6) is within 0.1 of the threshold (e.g., 0.65), the event may not be considered a hot word. Therefore, even if neither device is highly confident that a hot word is detected in the ecosystem, the acoustic event may be considered to be an occurrence of a hot word. Instead, assume that the acoustic event is based on the detection of the hot word by the fifth assistant non-input device 185 5 The metric generated by the detected audio data is 0.1. In this example, even if the 2 The metric of 0.7 generated by the detected audio data is indicative of a hot word, and the acoustic event is not considered as the occurrence of a hot word because it is based on the non-input device 185 of the fifth assistant. 5 The metric generated by the detected audio data fails to meet a threshold value (e.g., 0.65) and is not within a threshold range of the threshold value. In these examples, additional or alternative metric processing can be used to determine whether a hot word event is an actual hot word event. For example, one or more of the average, median, maximum, minimum, or percentile of a metric generated based on the corresponding audio data detected at the assistant device can be used to determine whether a hot word event is an actual hot word event. When determining one or more of these statistics, the hot word event can be based on, for example, (for example, using Figure 1 One or more metrics may optionally be weighted based on proximity of the assistant device to the user 101 (as determined by presence sensor 105 ), component type of the assistant device (e.g., an assistant device with multiple microphones versus an assistant device with a single microphone), and / or based on other factors.

[0087] Although the hotword event for activating one more automated assistant component or functionality is an acoustic event, Figure 2A and Figure 2B, but it should be understood that this is for illustration only and is not meant to be limiting. Additionally or alternatively, no-hotword events may also be detected by multiple assistant devices in the ecosystem. No-hotword events may include, for example, detecting a gaze directed toward a given assistant device (and optionally coupled with mouth movement of user 101), which, when detected, causes one more automated assistant component or functionality to be activated. For example, corresponding image data may be captured by multiple assistant devices (e.g., including visual components described with respect to presence sensors), the corresponding image data may be processed using a no-hotword model (e.g., stored in ML model database 192) to generate corresponding metrics, and the corresponding metrics may be processed to determine whether the no-hotword event is an actual no-hotword event. These techniques may also be used to detect other visual events that may occur in the ecosystem.

[0088] In addition, although this article describes acoustic events as hot word events Figure 2A and Figure 2B , but it should be understood that this is for illustration only and is not meant to be limiting. As another non-limiting example, and with particular reference to Figure 3 , any audio data captured in the environment may additionally or alternatively be processed using a sound detection model that is trained to detect the occurrence of specific sounds in the ecosystem. For example, assume that a window in room 250 is broken (rather than user 101 providing a verbal utterance), and further assume that a window is broken by first assistant input device 106 using a corresponding sound detection model. 1 The captured audio data, the sound detection model is trained to generate metrics associated with detecting the occurrence of glass breaking (and optionally additional metrics associated with other sounds, such as doorbells ringing, smoke alarms ringing, and / or other sounds that may occur in the ecosystem). Figure 2A Similar to the description, additional assistant devices in the ecosystem (e.g., the fifth assistant input device 106 5 and the fifth assistant non-input device 185 5 ) can be identified as being expected to detect temporally corresponding audio data, processing the temporally corresponding audio data to generate additional metrics, and determining whether a sound corresponding to glass breaking was in fact captured in the ecosystem based on the metrics and the additional metrics. As a result, a notification indicating a glass breaking sound detected in the ecosystem can be presented at a client device of user 101 (e.g., a mobile device and / or one or more assistant devices in the ecosystem), and optionally includes an indication of the assistant device that detected the glass breaking and / or an indication of a room in the ecosystem in which the glass breaking sound was detected.

[0089] Although this article describes acoustic events (and visual events) Figure 2A , Figure 2B and Figure 3 , but it should be understood that this is for illustration only and is not meant to be limiting. The techniques described herein can be used to verify the occurrence of any event that can be detected by multiple assistant devices in an ecosystem. For example, assume that a first assistant device is a mobile device that includes an accelerometer and is being carried by user 101, and assume that a second assistant device is a smart watch that is worn by user 101 and also includes an accelerometer. In this example, the corresponding accelerometer data generated by the mobile device and the smart watch can be processed in the same or similar manner as described above to determine whether the detected accelerometer event is an actual accelerometer event (e.g., user 101 walking, jogging, running, and / or any other motion activity that can be detected by an accelerometer).

[0090] Now go to Figure 4 , depicts a flowchart illustrating an example method 400 for determining whether an acoustic event is an actual acoustic event based on audio data captured at multiple assistant devices in an ecosystem. For convenience, the operations of method 400 are described with reference to a system that performs the operations. The system of method 400 includes one or more processors and / or other components of a computing device. For example, the system of method 400 can be implemented by: Figure 1 , Figure 2A , Figure 2B or Figure 3 assistant input device 106, Figure 1 , Figure 2A , Figure 2B or Figure 3 Assistant non-input device 185, Figure 6 The computing device 610, one or more servers, other computing devices, and / or any combination thereof. In addition, although the operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0091] At block 452, the system detects audio data capturing an acoustic event via a microphone of an assistant device located in an ecosystem including multiple assistant devices. The detected acoustic event can be, for example, a specific word or phrase (e.g., a hot word) provided by a user and associated with the ecosystem, a specific sound in the ecosystem that does not correspond to human speech, and / or other acoustic events that can occur in the ecosystem.

[0092] At block 454, the system processes the audio data capturing the acoustic event using an event detection model stored locally at the assistant device to generate a metric associated with the acoustic event. In some embodiments, the audio data capturing the acoustic event can be processed using a hot word detection model stored locally at the assistant device to generate a metric indicating whether the audio data detected at the assistant device includes a specific word or phrase (e.g., to activate one or more components of the automated assistant). In some additional or alternative embodiments, the audio data capturing the acoustic event can be processed using a sound detection model stored locally at the assistant device to generate a metric indicating whether the audio data detected at the assistant device includes a specific sound (e.g., glass breaking, dog barking, cat meowing, doorbell ringing, smoke alarm ringing, carbon monoxide detector ringing, baby crying, knocking on a door, and / or any other sound that may occur in an ecosystem).

[0093] In box 456, the system detects additional audio data of the acoustic event that is also captured via an additional microphone of an additional assistant device located in the ecosystem. The additional audio data detected at the additional assistant device can correspond in time to the audio data detected at the assistant device. Based on corresponding timestamps corresponding to when the audio data and the additional audio data were received, respectively, the system can determine that the additional audio data corresponds in time to the audio data. In addition, if the timestamps match each other or are within a threshold duration (e.g., within a few milliseconds, a few seconds, or any other suitable duration), the additional audio data can be considered to correspond in time to the audio data.

[0094] In box 458, the system processes the additional audio data that captures the acoustic event using the additional event detection model stored locally at the additional assistant device to generate additional metrics associated with the acoustic event. In some embodiments, the additional audio data that captures the acoustic event can be processed using the additional hot word detection model stored locally at the additional assistant device to generate additional metrics indicating whether the additional audio data detected at the additional assistant device includes a specific word or phrase (e.g., to activate one or more components of the automated assistant). In some additional or alternative embodiments, the additional audio data that captures the acoustic event can be processed using the additional sound detection model stored locally at the additional assistant device to generate additional metrics indicating whether the additional audio data detected at the additional assistant device includes a specific sound (e.g., glass breaking, dog barking, cat meowing, doorbell ringing, smoke alarm ringing, carbon monoxide detector ringing, and / or any other sound that may occur in the ecosystem).

[0095] At block 460, the system processes both the metric and the additional metric to determine whether the acoustic event detected by at least the assistant device and the additional assistant device is an actual acoustic event. The metric and the additional metric can be, for example, a binary value indicating whether the audio data captured the acoustic event (e.g., "1" for yes and "0" for no), a probability associated with whether the acoustic event occurred (e.g., 0.7 indicates that the audio data captured the acoustic event, while 0.3 indicates that the audio data did not capture the acoustic event), and / or other metrics associated with the acoustic event.

[0096] In box 462, the system determines whether the acoustic event is an actual acoustic event based on the metrics and additional metrics processed in box 460. If in the iteration of box 462, the system determines that the acoustic event is not an actual acoustic event, the system proceeds to box 464 and discards the audio data and additional audio data without any further processing. The system then returns to box 452. For example, it is assumed that the metric and the additional metric correspond to the probability of the audio data and the additional audio data capturing the hot word, respectively. For example, it is assumed that the metric is a probability of 0.7 that the audio data captures the hot word, and it is assumed that the additional metric is a probability of 0.3 that the additional audio data captures the hot word. In this example, the system can determine that the acoustic event is an actual acoustic event (e.g., the user does provide a verbal utterance including the hot word), assuming that the probability of 0.7 meets the threshold probability level based only on the metric. However, assuming that the probability of 0.3 cannot meet the threshold probability level based on the additional metric, the system can determine that the acoustic event is not an actual acoustic event (e.g., the user does not provide a verbal utterance including the hot word). As a result, even if the metric indicates that the acoustic event is an actual acoustic event, the system can determine that the acoustic event is not an actual acoustic event because the additional metric should also indicate that the acoustic event is an actual acoustic event, and the audio data and the additional audio data can be discarded. Additionally or alternatively, the system can determine whether the acoustic event is an actual acoustic event based on statistics associated with the metric, such as one or more of the mean, median, maximum, minimum, or percentile of the metric generated based on the corresponding audio data detected at the assistant device can be used to determine whether the acoustic event is an actual acoustic event.

[0097] If, in the iteration of box 462, the system determines that the acoustic event is an actual acoustic event, the system proceeds to box 466. For example, it is assumed that the metric and the additional metric correspond to the probability that the audio data and the additional audio data capture the hot word, respectively. For example, it is assumed that the metric is a probability of 0.7 that the audio data captures the hot word, and it is assumed that the additional metric is a probability of 0.8 that the additional audio data captures the hot word. In this example, it is assumed that the probability of 0.7 and the probability of 0.8 (or one or more statistics based thereon) meet the threshold probability levels based on the metric and the additional metric, respectively, then the system can determine that the acoustic event is an actual acoustic event (e.g., the user did provide a spoken utterance including the hot word). As a result, the system can determine that the acoustic event is an actual acoustic event because both the metric and the additional metric indicate that the acoustic event is an actual acoustic event.

[0098] In box 466, the system causes the operation associated with the actual acoustic event to be performed. The system then returns to box 452. The action performed by the system can be based on the actual acoustic event captured in the audio data and the additional audio data. For example, in an embodiment where the actual acoustic event corresponds to the occurrence of a hot word, the system can activate one or more components or functions of the automated assistant (e.g., speech processing, natural language processing, and / or other components or functions of the automated assistant). As another example, in an embodiment where the actual acoustic event corresponds to the occurrence of a specific sound, the system can generate a notification indicating the specific sound detected in the ecosystem (and optionally identifying which assistant devices detected the specific sound), and can cause the notification to be presented at a client device of a user associated with the ecosystem (e.g., the user's mobile device, one or more assistant devices, and / or any other client device associated with the user).

[0099] Now go to Figure 5 , depicts a flow chart illustrating an example method 500 for identifying which assistant devices should have detected an acoustic event in an ecosystem based on a given assistant device detecting the acoustic event. For convenience, the operations of method 500 are described with reference to a system performing the operations. The system of method 500 includes one or more processors and / or other components of a computing device. For example, the system of method 500 can be implemented by: Figure 1 , Figure 2A , Figure 2B or Figure 3 assistant input device 106, Figure 1 , Figure 2A , Figure 2B or Figure 3 Assistant non-input device 185, Figure 6The computing device 610 of the method 500, one or more servers, other computing devices, and / or any combination thereof. In addition, although the operations of the method 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and / or added.

[0100] At block 552, the system detects audio data capturing an acoustic event via a microphone of an assistant device located in an ecosystem including multiple assistant devices. The detected acoustic event can be, for example, a specific word or phrase (e.g., a hot word) provided by a user and associated with the ecosystem, a specific sound in the ecosystem that does not correspond to human speech, and / or other acoustic events that can occur in the ecosystem.

[0101] At box 554, the system identifies at least one additional assistant device based on the location of the assistant device within the ecosystem, and the at least one additional assistant device should have detected additional audio data via a corresponding microphone at the at least one additional assistant device, and the additional audio data corresponds in time to the audio data detected at the assistant device. The system can identify the at least one additional assistant device that should have detected the additional audio data based on the assistant device and the at least one additional assistant device that historically detected the corresponding audio data in time. In addition, based on corresponding timestamps corresponding to when the audio data and the additional audio data were received, respectively, the system can determine that the additional audio data corresponds in time to the audio data. In addition, if the timestamps match or are within a threshold duration of time of each other (e.g., within a few milliseconds, a few seconds, or any other suitable duration), the additional audio data can be considered to correspond in time to the audio data.

[0102] In box 556, the system determines whether the additional assistant device has detected additional audio data that corresponds in time to the audio data detected at the assistant device. If, in an iteration of box 556, the system determines that there is no additional audio data that corresponds in time to the audio data detected at the assistant device, the system proceeds to box 566 and discards the audio data and the additional audio data without any further processing. If, in an iteration of box 556, the system determines that there is additional audio data that corresponds in time to the audio data detected at the assistant device, the system proceeds to box 558. In other words, if no other assistant devices that should have detected audio data that captured the acoustic event capture any corresponding audio data in time, the system can determine that the acoustic event is not an actual acoustic event without processing the audio data using any event detection model.

[0103] At box 558, the system processes the audio data capturing the acoustic event using an event detection model stored locally at the assistant device to generate a metric associated with the acoustic event. In some embodiments, the audio data capturing the acoustic event can be processed using a hot word detection model stored locally at the assistant device to generate a metric indicating whether the audio data detected at the assistant device includes a specific word or phrase (e.g., to activate one or more components of the automated assistant). In some additional or alternative embodiments, the audio data capturing the acoustic event can be processed using a sound detection model stored locally at the assistant device to generate a metric indicating whether the audio data detected at the assistant device includes a specific sound (e.g., glass breaking, dog barking, cat meowing, doorbell ringing, smoke alarm ringing, carbon monoxide detector ringing, baby crying, knocking on the door, and / or any other sound that may occur in the ecosystem).

[0104] At box 560, the system processes the additional audio data capturing the acoustic event using the additional event detection model stored locally at the additional assistant device to generate additional metrics associated with the acoustic event. In some embodiments, the additional audio data capturing the acoustic event can be processed using the additional hot word detection model stored locally at the additional assistant device to generate additional metrics indicating whether the additional audio data detected at the additional assistant device includes a specific word or phrase (e.g., to activate one or more components of the automated assistant). In some additional or alternative embodiments, the additional audio data capturing the acoustic event can be processed using the additional sound detection model stored locally at the additional assistant device to generate additional metrics indicating whether the additional audio data detected at the additional assistant device includes a specific sound (e.g., glass breaking, dog barking, cat meowing, doorbell ringing, smoke alarm ringing, carbon monoxide detector ringing, and / or any other sound that may occur in the ecosystem).

[0105] At block 562, the system processes both the metric and the additional metric to determine whether the acoustic event detected by at least the assistant device and the additional assistant device is an actual acoustic event. The metric and the additional metric can be, for example, a binary value indicating whether the audio data captures an acoustic event (e.g., "1" for yes and "0" for no), a probability associated with whether the acoustic event occurred (e.g., 0.7 for the audio data to capture an acoustic event versus 0.3 for the audio data not to capture an acoustic event), and / or other metrics associated with an acoustic event.

[0106] In box 564, the system determines whether the acoustic event is an actual acoustic event based on the processing metric and the additional metric in box 562. If in the iteration of box 564, the system determines that the acoustic event is not an actual acoustic event, the system proceeds to box 566 and discards the audio data and the additional audio data without any further processing. The system then returns to box 552. For example, it is assumed that the metric and the additional metric correspond to the probability of the audio data and the additional audio data capturing the hot word, respectively. For example, it is assumed that the metric is a probability of 0.7 that the audio data captures the hot word, and it is assumed that the additional metric is a probability of 0.3 that the additional audio data captures the hot word. In this example, assuming that the probability of 0.7 meets the threshold probability level based only on the metric, the system can determine that the acoustic event is an actual acoustic event (e.g., the user does provide a verbal utterance including the hot word). However, assuming that the probability of 0.3 cannot meet the threshold probability level based on the additional metric, the system can determine that the acoustic event is not an actual acoustic event (e.g., the user does not provide a verbal utterance including the hot word). As a result, even if the metric indicates that the acoustic event is an actual acoustic event, the system can determine that the acoustic event is not an actual acoustic event because the additional metric should also indicate that the acoustic event is an actual acoustic event, and the audio data and the additional audio data can be discarded. Additionally or alternatively, the system can determine whether the acoustic event is an actual acoustic event based on statistics associated with the metric, such as one or more of the mean, median, maximum, minimum, or percentile of the metric generated based on the corresponding audio data detected at the assistant device can be used to determine whether the acoustic event is an actual acoustic event.

[0107] If, in the iteration of box 564, the system determines that the acoustic event is an actual acoustic event, the system proceeds to box 568. For example, it is assumed that the metric and the additional metric correspond to the probability that the audio data and the additional audio data capture the hot word, respectively. For example, it is assumed that the metric is a probability of 0.7 that the audio data captures the hot word, and it is assumed that the additional metric is a probability of 0.8 that the additional audio data captures the hot word. In this example, assuming that the probability of 0.7 and the probability of 0.8 (or one or more statistics based thereon) meet the threshold probability levels based on the metric and the additional metric, respectively, the system can determine that the acoustic event is an actual acoustic event (e.g., the user did provide a spoken utterance including the hot word). As a result, the system can determine that the acoustic event is an actual acoustic event because both the metric and the additional metric indicate that the acoustic event is an actual acoustic event.

[0108] In box 568, the system causes the action associated with the actual acoustic event to be performed. The system then returns to box 552. The action performed by the system can be based on the actual acoustic event captured in the audio data and the additional audio data. For example, in an embodiment where the actual acoustic event corresponds to the occurrence of a hot word, the system can activate one or more components or functions of the automated assistant (e.g., speech processing, natural language processing, and / or other components or functions of the automated assistant). As another example, in an embodiment where the actual acoustic event corresponds to the occurrence of a specific sound, the system can generate a notification indicating that a specific sound is detected in the ecosystem (and optionally identifying which assistant devices have detected the specific sound), and can cause the notification to be presented at a client device of a user associated with the ecosystem (e.g., the user's mobile device, one or more assistant devices, and / or any other client device associated with the user).

[0109] Figure 6 610 is a block diagram of an example computing device 610 that can optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more assistant input devices, one or more cloud-based automated assistant components, one or more assistant non-input systems, one or more assistant non-input devices, and / or other components can include one or more components of the example computing device 610.

[0110] The computing device 610 typically includes at least one processor 614 that communicates with a number of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624, which includes, for example, a memory subsystem 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow a user to interact with the computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0111] The user interface input device 622 may include: a keyboard, a pointing device such as a mouse, trackball, touchpad or graphic tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways to input information into the computing device 610 or over a communication network.

[0112] User interface output devices 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display, such as via an audio output device. In general, the use of the term "output device" is intended to include all possible types of devices and ways of outputting information from computing device 610 to a user or other machine or computing device.

[0113] The storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include logic for performing selected aspects of the methods described herein, as well as implementing Figure 1 The logic of the various components is depicted.

[0114] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory 625 used in the storage subsystem 624 may include a number of memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 for storing fixed instructions. The file storage subsystem 626 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive with associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain embodiments may be stored by the file storage subsystem 626 in the storage subsystem 624, or stored in other machines accessible by the processor 614.

[0115] The bus subsystem 612 provides a mechanism for the various components and subsystems of the computing device 610 to communicate with each other as intended. Although the bus subsystem 612 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0116] Computing device 610 may be of varying types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. As the nature of computers and networks continues to change, Figure 6 The description of computing device 610 depicted in FIG. 6 is intended only as a specific example for purposes of illustrating some embodiments. Many other configurations of computing device 610 may have more Figure 6 The computing devices depicted may have greater or fewer components.

[0117] Where certain embodiments discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about the user's social network, the user's location, the user's time, the user's biometric information, the user's activities and demographic information, the relationships between users, etc.), the user is provided with one or more opportunities to control whether the information is collected, whether the personal information is stored, whether the personal information is used, and how the information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information only after receiving explicit authorization from the relevant user.

[0118] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. One or more options are presented to each user for which personal information is collected to allow control over the collection of information associated with that user to provide permission or authorization as to whether and which portions of the information are collected. For example, one or more such control options may be provided to a user over a communications network. In addition, prior to storing or using certain data, the data may be processed in one or more ways to remove personally identifiable information. As an example, the identity of a user may be processed so that no personally identifiable information can be determined. As another example, a user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

[0119] In some embodiments, a method implemented by one or more processors is provided, and the method includes: detecting audio data capturing an acoustic event via one or more microphones of an assistant device located in an ecosystem including multiple assistant devices; processing the audio data capturing the acoustic event using an event detection model stored locally at the assistant device to generate a metric associated with the acoustic event; detecting additional audio data also capturing the acoustic event via one or more additional microphones of an additional assistant device located in the ecosystem, the additional assistant device being an addition to the assistant device, and the additional assistant device being co-located with the assistant device in the ecosystem; processing the additional audio data capturing the acoustic event using an additional event detection model stored locally at the additional assistant device to generate an additional metric associated with the acoustic event; processing both the metric and the additional metric to determine whether the acoustic event detected by at least the assistant device and the additional assistant device is an actual acoustic event; and in response to determining that the acoustic event is an actual acoustic event, performing an action associated with the actual acoustic event.

[0120] These and other implementations of the technology disclosed herein can include one or more of the following features.

[0121] In some embodiments, the acoustic event may include a hot word detection event, the event detection model stored locally at the assistant device may be a hot word detection model that is trained to detect whether a specific word or phrase is captured in the audio data, and the additional event detection model stored locally at the additional assistant device may be an additional hot word detection model that is trained to detect whether a specific word or phrase is captured in the additional audio data.

[0122] In some versions of those embodiments, the metric associated with the acoustic event can be a confidence level corresponding to whether the audio data captures a particular word or phrase, and the additional metric associated with the acoustic event can be an additional confidence level corresponding to whether the additional audio data captures a particular word or phrase. In some further versions of those embodiments, determining that the acoustic event is an actual acoustic event can include determining that the particular word or phrase is captured in both the audio data and the additional audio data based on the confidence level and the additional confidence level. In further versions of those embodiments, in response to determining that the acoustic event data indicates that the audio data or the additional audio data captures a particular word or phrase, causing an action associated with the actual acoustic event to be performed can include activating one or more components of an automated assistant.

[0123] In some additional or alternative versions of those embodiments, the hotword detection model stored locally at the assistant device may be a different hotword model than additional hotword detection models stored locally at additional assistant devices.

[0124] In some embodiments, the acoustic event can be a sound detection event, the event detection model stored locally at the assistant device can be a sound detection model that is trained to detect whether a specific sound is captured in the audio data, and the additional event detection model stored locally at the additional assistant device can be an additional sound detection model that is trained to detect whether a specific sound is captured in the additional audio data.

[0125] In some versions of those embodiments, the metric associated with the acoustic event may be a confidence level corresponding to whether the audio data captures a particular sound, and the additional metric associated with the acoustic event may be an additional confidence level corresponding to whether the additional audio data captures a particular sound. In some other versions of these embodiments, determining that the acoustic event is an actual acoustic event may include determining that a particular sound is captured in both the audio data and the additional audio data based on the confidence level and the additional confidence level. In further versions of those embodiments, causing an action associated with the actual acoustic event to be performed may include generating a notification indicating that the actual acoustic event occurred; and causing the notification to be displayed to the user via a computing device of the user associated with the ecosystem.

[0126] In some additional or alternative versions of those embodiments, the particular sound may include one or more of: glass breaking, a dog barking, a cat meowing, a doorbell ringing, a smoke alarm ringing, a carbon monoxide detector ringing, a baby crying, or a knock on a door.

[0127] In some embodiments, both the metric and the additional metric are processed to determine whether an acoustic event detected by both the assistant device and the additional assistant device is an actual acoustic event performed by a given assistant device, which may be one or more of the following: the assistant device, the additional assistant device, or a further additional assistant device co-located in an ecosystem with the assistant device and the additional assistant device.

[0128] In some embodiments, the method may further include: transmitting, by the assistant device, the audio data to the remote system; and transmitting, by the additional assistant device, the additional audio data to the remote system. In these embodiments, processing both the metric and the additional metric to determine whether the acoustic event detected by both the assistant device and the additional assistant device is an actual acoustic event may be performed by the remote system.

[0129] In some embodiments, the audio data corresponds in time to the additional audio data. In some versions of those embodiments, in response to determining that the timestamp associated with the audio data and the additional timestamp associated with the additional audio data correspond in time, both the metric and the additional metric can be processed to determine whether the acoustic event detected by both the assistant device and the additional assistant device is an actual acoustic event.

[0130] In some embodiments, in response to detecting audio data via the one or more microphones of the assistant device, the method may further include: anticipating detection of additional audio data via one or more additional microphones of the additional assistant device based on multiple historical acoustic events detected at both the assistant device and the additional assistant device.

[0131] In some embodiments, the method may further include: detecting subsequent audio data that captures a subsequent acoustic event via the one or more microphones of the assistant device; processing the subsequent audio data that captures the subsequent acoustic event using an event detection model to generate a subsequent metric associated with the subsequent acoustic event; detecting additional subsequent audio data that also captures a subsequent acoustic event via one or more further additional microphones of a further additional assistant device located in the ecosystem, the further additional assistant device being an addition to the assistant device, and the further additional assistant device being co-located with the assistant device in the ecosystem; processing the additional subsequent audio data that captures the acoustic event using a further additional event detection model locally stored at the further additional assistant device to generate an additional subsequent metric associated with the acoustic event; processing both the metric and the additional metric to determine whether the subsequent acoustic event detected by both the assistant device and the further additional assistant device is an actual subsequent acoustic event; and in response to determining that the subsequent acoustic event is an actual subsequent acoustic event, performing a subsequent action associated with the actual subsequent acoustic event.

[0132] In some versions of those embodiments, in response to detecting subsequent audio data via the one or more microphones of the assistant device, the method may further include: based on the additional multiple historical acoustic events detected at both the assistant device and the further additional assistant device, anticipating detection of additional subsequent audio data via the one or more further additional microphones of the further additional assistant device, while not anticipating detection of any audio data via the one or more additional microphones of the additional assistant device.

[0133] In some embodiments, a method implemented by one or more processors is provided, and the method includes: detecting audio data capturing an acoustic event via one or more microphones of an assistant device located in an ecosystem including multiple assistant devices; identifying at least one additional assistant device based on the position of the assistant device within the ecosystem, and the at least one additional assistant device should have detected additional audio data corresponding in time to the audio data via one or more corresponding microphones of the at least one additional assistant device in the ecosystem; in response to determining that the at least one additional assistant device has detected additional audio data corresponding in time to the audio data: processing the audio data capturing the acoustic event using an event detection model stored locally at the assistant device to generate a metric associated with the acoustic event, and processing the additional audio data capturing the acoustic event using a corresponding event detection model stored locally at the at least one additional assistant device to generate an additional metric associated with the acoustic event; determining whether the acoustic event detected by both the assistant device and the additional assistant device is an actual acoustic event based on both the metric and the additional metric; and in response to determining that the acoustic event is an actual acoustic event, performing an action associated with the actual acoustic event.

[0134] These and other implementations of the technology disclosed herein may include one or more of the following features.

[0135] In some implementations, the method may further include: in response to determining that the at least one additional assistant device does not detect any audio data that corresponds in time to the audio data, discarding the audio data.

[0136] In addition, some embodiments include one or more processors (e.g., central processing units (CPUs), graphics processing units (GPUs), and / or tensor processing units (TPUs)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to result in the performance of any of the aforementioned methods. Some embodiments also include one or more non-transitory computer-readable storage media storing computer instructions executable by one or more processors to perform any of the aforementioned methods. Some embodiments also include a computer program product comprising instructions executable by one or more processors to perform any of the aforementioned methods.

[0137] It should be understood that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered to be part of the subject matter disclosed herein.

Claims

1. A method implemented by one or more processors, the method comprising: detecting, via one or more microphones of an assistant device in an ecosystem including a plurality of assistant devices, audio data capturing an acoustic event; processing the audio data capturing the acoustic event using an event detection model stored locally at the assistant device to generate a metric associated with the acoustic event; receiving additional audio data that also captures the acoustic event from an additional assistant device co-located with the assistant device in the ecosystem, the additional assistant device being an addition to the assistant device, and the additional audio data being generated via one or more additional microphones of the additional assistant device; processing the additional audio data capturing the acoustic event using the event detection model stored locally at the assistant device to generate additional metrics associated with the acoustic event; processing both the metric and the additional metric to determine whether the acoustic event detected by at least both the assistant device and the additional assistant device is an actual acoustic event; as well as In response to determining that the acoustic event is the actual acoustic event, causing an action associated with the actual acoustic event to be performed.

2. The method according to claim 1, wherein: The acoustic events include hot word detection events, and wherein the event detection model stored locally at the assistant device includes a hot word detection model, which is trained to detect whether a specific word or phrase is captured in the audio data and the additional audio data.

3. The method according to claim 2, wherein: The metric associated with the acoustic event includes a confidence level corresponding to whether the audio data captures the particular word or phrase, and wherein the additional metric associated with the acoustic event includes an additional confidence level corresponding to whether the additional audio data captures the particular word or phrase.

4. The method according to claim 2, wherein: Determining that the acoustic event is the actual acoustic event includes determining, based on the confidence level and the additional confidence level, that the particular word or phrase is captured in both the audio data and the additional audio data.

5. The method according to claim 2, wherein: Causing the action associated with the actual acoustic event to be performed includes: at the assistant device or the additional assistant device, activating one or more components of the automated assistant in response to determining that the acoustic event data indicates that the audio data and the additional audio data capture the particular word or phrase.

6. The method according to claim 1, wherein: The acoustic event comprises a sound detection event, and wherein the event detection model stored locally at the assistant device comprises a sound detection model trained to detect whether a specific sound is captured in the audio data and the additional audio data.

7. The method according to claim 6, wherein: The metric associated with the acoustic event includes a confidence level corresponding to whether the audio data captures the particular sound, and wherein the additional metric associated with the acoustic event includes an additional confidence level corresponding to whether the additional audio data captures the particular sound.

8. The method according to claim 7, wherein: Determining that the acoustic event is the actual acoustic event includes determining, based on the confidence level and the additional confidence level, that the particular sound is captured in both the audio data and the additional audio data.

9. The method according to claim 6, wherein: Causing an action associated with the actual acoustic event to be performed includes: generating a notification indicating the occurrence of the sound detection event; and The notification is caused to be presented to a user associated with the ecosystem via a computing device of the user.

10. The method according to claim 6, wherein: The specific sounds include one or more of the following: breaking glass, barking dogs, meowing cats, doorbells, smoke alarms, carbon monoxide detectors, crying babies, and knocking doors.

11. The method according to claim 1, wherein: The audio data corresponds in time to the additional audio data.

12. The method according to claim 1, wherein: The assistant device and the additional assistant device historically detect corresponding audio data capturing the same acoustic event.

13. The method according to claim 12, wherein: In response to determining that the timestamp associated with the audio data and the additional timestamp associated with the additional audio data correspond in time, both the metric and the additional metric are processed to determine whether the acoustic event detected by both the assistant device and the additional assistant device is the actual acoustic event.

14. The method according to claim 1, wherein: The assistant device is a first-party assistant device manufactured by a first party, wherein the additional assistant device is a third-party assistant device manufactured by a third party, and wherein the third party is a different party from the first party.

15. A method implemented by one or more processors, the method comprising: receiving audio data capturing an acoustic event, the audio data generated via one or more microphones of an assistant device in an ecosystem including a plurality of assistant devices; processing the audio data capturing the acoustic event using an event detection model to generate a metric associated with the acoustic event; receiving additional audio data that also captures the acoustic event, the additional audio data being generated via one or more additional microphones of an additional assistant device co-located with the assistant device in the ecosystem and being an addition to the assistant device; processing the additional audio data capturing the acoustic event using the event detection model to generate additional metrics associated with the acoustic event; processing both the metric and the additional metric to determine whether the acoustic event detected by at least both the assistant device and the additional assistant device is an actual acoustic event; as well as In response to determining that the acoustic event is the actual acoustic event, causing an action associated with the actual acoustic event to be performed.

16. The method according to claim 15, wherein: The assistant device is a first-party assistant device manufactured by a first party, wherein the additional assistant device is a third-party assistant device manufactured by a third party, and wherein the third party is a different party from the first party.

17. The method according to claim 15, wherein: The one or more processors are processors of a remote system remote from the ecosystem, and wherein the event detection model is remote from the ecosystem.

18. A method implemented by one or more processors, the method comprising: detecting, via one or more microphones of an assistant device in an ecosystem including a plurality of assistant devices, audio data capturing an acoustic event, wherein the acoustic event comprises a hotword detection event; processing the audio data capturing the acoustic event using an event detection model stored locally at the assistant device to generate a metric associated with the acoustic event, wherein the event detection model stored locally at the assistant device comprises a hot word detection model trained to detect when a particular word or phrase is captured in the audio data; detecting additional audio data also capturing the acoustic event via one or more additional microphones of an additional assistant device located in the ecosystem, the additional assistant device being an addition to the assistant device and the additional assistant device being co-located with the assistant device in the ecosystem; processing the additional audio data capturing the acoustic event using an additional event detection model stored locally at the additional assistant device to generate an additional metric associated with the acoustic event, wherein the additional event detection model stored locally at the additional assistant device includes an additional hot word detection model trained to detect that the particular word or phrase is captured in the additional audio data; determining, based on both the metric and the additional metric, whether the acoustic event detected by at least the assistant device and the additional assistant device corresponds to an occurrence of an actual acoustic event; and In response to determining that the acoustic event corresponds to an occurrence of the actual acoustic event, one or more components of the automated assistant are caused to be activated at one or more of: the assistant device, the additional assistant device, and another additional assistant device.

19. The method according to claim 18, wherein: The metric associated with the acoustic event includes a corresponding confidence level corresponding to whether the audio data captures the particular word or phrase, and wherein the additional metric associated with the acoustic event includes a corresponding additional confidence level corresponding to whether the additional audio data captures the particular word or phrase.

20. The method according to claim 19, wherein: Determining that the acoustic event corresponds to an occurrence of the actual acoustic event includes determining that the particular word or phrase is captured in both the audio data and the additional audio data based on the corresponding confidence level and the corresponding additional confidence level.

21. The method according to claim 18, wherein: The hot word detection model stored locally at the assistant device is a different hot word model, and the different hot word model is different from the additional hot word detection model stored locally at the additional assistant device.

22. The method according to claim 18, wherein: The assistant device, the additional assistant device, and the another additional assistant device are co-located in an ecosystem of devices.

23. The method according to claim 18, wherein: The audio data corresponds in time to the additional audio data.

24. The method according to claim 23, wherein: In response to determining that the audio data corresponds in time to the additional audio data, determining whether the acoustic event detected by at least both the assistant device and the additional assistant device corresponds to an occurrence of an actual acoustic event based on both the metric and the additional metric.

25. The method according to claim 23, wherein: In response to determining that the timestamp associated with the audio data and the additional timestamp associated with the additional audio data correspond in time, determining whether the acoustic event detected by both the assistant device and the additional assistant device corresponds to an occurrence of the actual acoustic event.

26. The method of claim 18, further comprising: In response to detecting the audio data via the one or more microphones of the assistant device: Based on a plurality of historical acoustic events detected at both the assistant device and the additional assistant device, detection of the additional audio data via the one or more additional microphones of the additional assistant device is anticipated.

27. The method of claim 18, further comprising: transmitting the audio data by the assistant device to a remote system; as well as transmitting the additional audio data to the remote system by the additional assistant device, Wherein, determining whether the acoustic event detected by both the assistant device and the additional assistant device corresponds to the occurrence of the actual acoustic event is performed by the remote system.

28. The method of claim 18, wherein: Determining that the acoustic event corresponds to the occurrence of the actual acoustic event includes: determining that the metric satisfies a threshold confidence level; and A determination is made that the additional metric satisfies the threshold confidence level or an additional threshold confidence level.

29. The method according to claim 28, wherein: The additional threshold confidence level is within a threshold range of the threshold confidence level.

30. The method of claim 18, further comprising: In response to determining that the acoustic event does not correspond to an occurrence of the actual acoustic event, refraining from causing one or more components of the automated assistant to be activated at one or more of: the assistant device, the additional assistant device, and another additional assistant device.

31. The method according to claim 30, wherein: Determining that the acoustic event does not correspond to the occurrence of the actual acoustic event includes: determining that the metric does not satisfy the threshold confidence level; or It is determined that the additional metric does not satisfy the threshold confidence level and does not satisfy the additional threshold confidence level.

32. The method of claim 18, wherein: The one or more components of the automated assistant include one or more of the following: a speech-to-text component; and Natural language understanding component.

33. An assistant device, comprising: at least one processor; as well as A memory storing instructions which, when executed, cause the at least one processor to perform a method according to any one of claims 1 to 32.

34. A non-transitory computer-readable storage medium storing instructions that, when executed, cause at least one processor to perform the method of any one of claims 1 to 32.

Images